Data lake vector matching method and device, electronic equipment and storage medium

Abstract

The invention provides a data lake vector matching method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a first vector; projecting the first vector to a set vector by using locality sensitive hashing to obtain a first result; determining at least one second vector and at least one second result corresponding to the at least one second vector; comparing the first result with the at least one second result, determining a target result corresponding to the first result in the at least one second result, and determining the second vector corresponding to the target result as a target vector; and outputting the target vector as a matching result of the first vector.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a method, device, electronic device, and storage medium for vector matching on a data lake. Background Art

[0002] With the development and widespread use of big data and machine learning, a large number of recommendation systems, image retrieval, and text retrieval matching models and programs have emerged. Most of these systems, models, and programs convert data into vectors, perform similarity matching and calculations, and ultimately determine the target data for output. Consequently, vector similarity search and matching have become core requirements for scenarios such as recommendation systems, image retrieval, and text matching.

[0003] However, related technologies face challenges in calculating and matching similarities between vectors, including high computational complexity, slow response times, and low efficiency. Furthermore, some approximate indexing solutions require the use of standalone disks to store indexes, making them impractical for storage on data lakes.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of this, the present disclosure proposes a vector matching method, device, electronic device and storage medium on a data lake to solve or partially solve the above problems.

[0006] Based on the above objectives, the present disclosure provides a vector matching method on a data lake, comprising:

[0007] Get the first vector;

[0008] Projecting the first vector onto a set vector using locality sensitive hashing to obtain a first result;

[0009] determining at least one second vector and at least one second result corresponding to the at least one second vector;

[0010] comparing the first result with the at least one second result, determining a target result corresponding to the first result in the at least one second result, and determining the second vector corresponding to the target result as a target vector;

[0011] The target vector is output as a matching result of the first vector.

[0012] In some exemplary embodiments, the setting vector is a plurality of setting vectors;

[0013] The projecting the first vector onto a set vector using locality sensitive hashing to obtain a first result includes:

[0014] The first vector is copied according to the number of the multiple setting vectors, so that each setting vector corresponds to a copied first vector, and the first result corresponding to each setting vector is obtained by using the locality sensitive hash.

[0015] In some exemplary embodiments, the number of the second results corresponding to any second vector is the same as the number of the set vectors.

[0016] In some exemplary embodiments, determining at least one second vector and at least one second result corresponding to the at least one second vector includes:

[0017] Any second vector is projected onto the set vector according to the locality sensitive hash, so as to determine the second result corresponding to the any second vector.

[0018] In some exemplary embodiments, before determining at least one second vector and at least one second result corresponding to the at least one second vector, the method further includes:

[0019] obtaining the at least one second vector;

[0020] Projecting the at least one second vector onto the setting vector using the locality sensitive hash to obtain the at least one second result;

[0021] The at least one second result is stored in at least one database.

[0022] In some exemplary embodiments, comparing the first result with the at least one second result includes:

[0023] The first result is transmitted to the at least one database to complete the comparison in the at least one database.

[0024] In some exemplary embodiments, the number of the target vector is at least one;

[0025] After outputting the target vector as a matching result of the first vector, the method further includes:

[0026] In response to the number of target vectors being multiple, determining the Euclidean distance between the target vector and the first vector, sorting the target vectors according to the Euclidean distance, and selecting a set number of the target vectors as similarity results of the first vector according to the sorting result.

[0027] Based on the same concept, the present disclosure also provides a vector matching device on a data lake, comprising:

[0028] A first module is used to obtain a first vector;

[0029] A second module is configured to project the first vector onto a set vector using locality sensitive hashing to obtain a first result;

[0030] A third module is configured to determine at least one second vector and at least one second result corresponding to the at least one second vector;

[0031] a fourth module, configured to compare the first result with the at least one second result, determine a target result corresponding to the first result in the at least one second result, and determine the second vector corresponding to the target result as a target vector;

[0032] The fifth module is configured to output the target vector as a matching result of the first vector.

[0033] Based on the same concept, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above methods when executing the computer program.

[0034] Based on the same concept, the present disclosure further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to implement any of the above methods.

[0035] As can be seen from the above, the present disclosure provides a vector matching method, device, electronic device and storage medium on a data lake, the method comprising: obtaining a first vector; projecting the first vector onto a set vector using local sensitive hashing to obtain a first result; determining at least one second vector and at least one second result corresponding to the at least one second vector; comparing the first result with the at least one second result, determining a target result corresponding to the first result in the at least one second result, and determining the second vector corresponding to the target result as a target vector; outputting the target vector as a matching result of the first vector. The present disclosure uses local sensitive hashing to perform retrieval by hash value matching. For the first vector to be matched and the known second vector, both can be structured by projecting the local sensitive hashing onto the set vector. While facilitating storage, it also facilitates the comparison between vectors. At the same time, such structured data is also conducive to distributed storage. The ability to perform distributed storage and processing enables tasks that could only be completed on a single processor to be placed on more processors for joint processing, which is bound to improve the corresponding response speed and processing efficiency. At the same time, the use of local sensitive hashing can also make data calculations relatively simple, ultimately greatly improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of an exemplary method provided in accordance with an embodiment of the present disclosure.

[0038] Figure 2 A schematic diagram of the structure of an exemplary device provided in an embodiment of the present disclosure.

[0039] Figure 3 A schematic diagram of the electronic device structure provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of this specification more clear, this specification is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0041] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should have the usual meanings understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements, objects or method steps appearing before the word include the elements, objects or method steps listed after the word and their equivalents, without excluding other elements, objects or method steps. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0042] As mentioned in the background technology section, with the popularization of big data and machine learning applications, vector similarity search has become a core requirement in scenarios such as recommendation systems, image retrieval, and text matching. The traditional exact nearest neighbor (ENN) algorithm has the problems of high computational complexity and slow response speed on high-dimensional large-scale data sets. Although existing approximate nearest neighbor technology can improve efficiency, it has the following shortcomings: (1) Limited scalability: It is difficult to store billions of vector data on a data lake and efficiently process them in a distributed environment; (2) Data management separation: The separation of index and storage system leads to data redundancy and high maintenance costs; (3) Low batch retrieval efficiency: The single retrieval mode cannot fully utilize resources and has high I / O overhead.

[0043] Existing vector search solutions in the related art primarily perform approximate searches through indexing methods such as IVFlat and HNSW. However, these indexes require dedicated processes for updating, saving, and retrieving the index, requiring additional hardware resources and being difficult to scale horizontally, making them unsuitable for data lake storage scenarios.

[0044] In view of the above actual situation, the embodiment of the present disclosure provides a vector matching method on a data lake. The present disclosure uses local sensitive hashing to perform retrieval by hash value matching. For the first vector to be matched and the known second vector, both can be structured by projecting local sensitive hashing onto a set vector. While facilitating storage, it also facilitates the comparison between vectors. At the same time, such structured data is also conducive to distributed storage. The ability to perform distributed storage and processing enables matters that could only be completed on a single processor to be placed on more processors for joint processing, which is bound to improve the corresponding response speed and processing efficiency. At the same time, the use of local sensitive hashing can also make data calculations relatively simple, ultimately greatly improving the user experience.

[0045] Figure 1 A flowchart of an exemplary method provided by an embodiment of the present disclosure is shown.

[0046] like Figure 1 As shown, the vector matching method on the data lake proposed by an embodiment of the present disclosure exemplarily includes the following steps.

[0047] Step 102: Obtain a first vector.

[0048] In this step, the first vector is the vector to be matched. In specific scenarios, the first vector can be a vector generated from the image to be recognized in image recognition, a vector generated from the text to be recognized in text recognition, or a keyword vector generated by user input in a recommendation system. This first vector is generally used in scenarios where similarity or distance calculation is required.

[0049] Afterwards, the acquisition method can be provided by the user through the corresponding port when using the recommendation system, image recognition terminal, or text recognition terminal, or it can be imported in batches through tables or links.

[0050] In specific application scenarios, first vectors may be imported in batches, or a set of first vectors may be obtained all at once. This description uses only one first vector. If multiple first vectors are included, simply repeat the steps for one first vector. These can be processed sequentially or in parallel.

[0051] Step 104: Project the first vector onto a set vector using locality sensitive hashing to obtain a first result.

[0052] In this step, Locality Sensitive Hashing (LSH) is a technology widely used in high-dimensional data processing. Its core feature is that it can maintain the similarity of data to a certain extent, that is, similar data has a higher probability of being mapped to the same hash bucket after hashing, while dissimilar data is likely to be mapped to different hash buckets. In this embodiment, locality sensitive hashing is used to perform retrieval by hash value matching, without the need for additional indexes, and efficient index construction and distributed batch retrieval can be performed on the data lake. During the application process, locality sensitive hashing can be used to project the first vector x onto a randomly set vector v, and the projection results can be assigned to the hash buckets:

[0053]

[0054] Where h(x) is the ID of the hash bucket that maps the first vector x to the set vector v, and r is the custom bucket length. =((x) / ...

[0055] In some embodiments, to improve recognition and matching accuracy, multiple setting vectors v can be set. This allows a single first vector x to correspond to multiple first results. Taking two setting vectors v as an example, the aforementioned formula yields two h(x). For ease of distinction, these two results are represented as h1(x) and h2(x). Therefore, since there are two setting vectors, a single first vector x corresponds to two hash bucket IDs. Subsequently, if one wishes to map the first vector x to their respective buckets and process each bucket independently, this can be achieved by "expanding" the first vector x into multiple copies. The idea is to replicate the data for each bucket ID for each data point, using a mapping function for each bucket ID. In the example above, there are two bucket ID functions, so the data can be expanded into copies, one for each bucket generated by one of the bucket ID functions. The bucket ID column has two values: the first value records the bucket ID function number (i.e., the unit vector number), and the second value is the result of the bucket ID function. Using the aforementioned example, the two bucket IDs could be (0, h1(x)) and (1, h2(x)). Furthermore, in some embodiments, the first result may also be the aforementioned two bucket IDs (0, h1(x)) and (1, h2(x)). That is, in some embodiments, the setting vector is a plurality of setting vectors; and projecting the first vector onto the setting vector using locality-sensitive hashing to obtain the first result includes: replicating the first vector according to the number of the plurality of setting vectors, such that each setting vector corresponds to a replicated first vector, and obtaining the first result corresponding to each setting vector using the locality-sensitive hashing.

[0056] In some specific application scenarios, before projecting the first vector onto the set vector, the first vector may be normalized, that is, the first vector may be scaled to a set interval, such as the interval (0, 1), to facilitate subsequent calculations.

[0057] Step 106: Determine at least one second vector and at least one second result corresponding to the at least one second vector.

[0058] In this step, the second vector is a known vector. In a specific application scenario, the vector can be a vector corresponding to a known image, a vector corresponding to a known text, and so on. Since the second vector can generally be prepared in advance in different scenarios, the corresponding second vector can also be prepared in advance. Of course, in some specific application scenarios, it can also be obtained in real time from the Internet or by other means after the first vector is obtained. Afterwards, the method for obtaining the second result is the same as or similar to the method for obtaining the aforementioned first result, except that the first vector is replaced by the second vector. At the same time, since the second vector itself can be determined in advance, the setting vector (number or specific form, etc.) itself can also be set in advance according to the specific scenario, and then the second result itself can also complete the corresponding calculation in advance and finally determine the corresponding result.

[0059] Furthermore, in some embodiments, since the way in which the second vector obtains the second result is the same as or similar to the way in which the first vector obtains the first result, in some embodiments, determining at least one second vector and at least one second result corresponding to the at least one second vector includes: projecting any second vector onto the set vector according to the local sensitive hash, so as to determine the second result corresponding to the any second vector.

[0060] In some embodiments, since the setting vector itself can be multiple, the number of second results corresponding to each second vector is the same as the number of setting vectors. That is, in some embodiments, the number of second results corresponding to any second vector is the same as the number of setting vectors.

[0061] In some specific application scenarios, similar to the first vector, before projecting the second vector onto the set vector, the second vector can also be normalized, that is, the second vector can be scaled to a set interval, such as the (0, 1) interval, to facilitate subsequent calculations.

[0062] In some embodiments, since the hash bucket ID is obtained by projecting any second vector onto a set vector using locality-sensitive hashing, at least one hash bucket corresponding to each second vector, i.e., the second result, can be determined in advance. These hash buckets are essentially independent of each other. Therefore, storage of these hash buckets can be located in a single processor or multiple processors, enabling distributed processing to improve processing efficiency and response speed. In a more specific application scenario, the LakeSoul data lake can be used to store the data tables formed by the hash buckets. Within the LakeSoul data lake warehouse framework, a distributed computing framework can be used to efficiently and in parallel construct data indexes for these data tables. Once the index is constructed, searches only require calculating the hash value of the search point and then searching for candidate neighbors in the corresponding bucket, significantly reducing the number of data points required for comparison and thereby improving search efficiency. That is, in some embodiments, before determining at least one second vector and at least one second result corresponding to the at least one second vector, the method further includes: obtaining the at least one second vector; projecting the at least one second vector onto the set vector using locality-sensitive hashing to obtain the at least one second result; and storing the at least one second result in at least one database. The at least one database here can be different databases in the aforementioned LakeSoul data lake.

[0063] Step 108 : Compare the first result with the at least one second result, determine a target result corresponding to the first result in the at least one second result, and determine the second vector corresponding to the target result as a target vector.

[0064] In this step, after determining the first result and the second result, the first result and the second result can be compared by traversal or the like, and the second result identical to the first result can be determined, thereby determining the second vector corresponding to the first vector.

[0065] For example, in a specific application scenario, the two first results corresponding to the first vector x (assuming there are two set vectors) are (0,9) and (1,5). Then, assuming there are three second vectors y1, y2, and y3 corresponding to the second results are (0,4) and (1,6), (0,12) and (1,5), and (0,9) and (1,3), respectively. It can be seen that the first first result of the first vector x is the same as the first second result of the second vector y3, and the two first results of the first vector x are the same as the second second result of the second vector y2. Then it can be determined that the first vector x matches the second vector y3 and the second vector y2. It can be seen here that the target result is the second result in the second result that is the same as the first result or within the preset range. The target vector is the second vector corresponding to the target result. In the above example, the second vector y3 and the second vector y2 can be used as the target vector.

[0066] In specific application scenarios, an iterator can be set up to traverse all vectors in the same bucket. This allows you to determine which candidate pairs of first and second vectors are similar. Based on the aforementioned calculation, if different vectors have the same first or second result, they are considered to fall into the same hash bucket. The method for connecting candidate pairs here is similar or similar to the Join operation in SQL calculations, connecting rows with the same value in a specified column. If the same value appears multiple times, the join result will also appear multiple times.

[0067] In some embodiments, in conjunction with the embodiment in step 106 above, the second result may be stored in different databases, and the first result to be compared may be distributed to these databases, thereby enabling distributed data processing and improving processing speed. That is, in some embodiments, comparing the first result with the at least one second result includes: transmitting the first result to the at least one database to complete the comparison in the at least one database.

[0068] Step 110: Output the target vector as a matching result of the first vector.

[0069] In this step, after the target vector is determined, it can be output as the matching result of the first vector. For example, it can be displayed on a corresponding device to provide corresponding feedback to the operator. Of course, in other embodiments, the output method of the matching result is not limited to output display. It can also be used to store, display, use, or reprocess the matching result. The specific output method of the matching result can be flexibly selected according to different application scenarios and implementation needs.

[0070] Specifically, for example, in an application scenario where the method of this embodiment is executed on a single device, the matching results can be directly output in a displayed manner on the display component (display, projector, etc.) of the current device, so that the operator of the current device can directly see the content of the matching results from the display component.

[0071] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the matching results can be sent to other preset devices serving as receivers within the system, i.e., synchronization terminals, through any data communication method (wired connection, NFC, Bluetooth, wifi, cellular mobile network, etc.), so that the synchronization terminals can perform subsequent processing on them. Optionally, the synchronization terminal can be a preset server, which is generally set up in the cloud and serves as a data processing and storage center, capable of storing and distributing matching results; wherein the recipients of the distribution are terminal devices, and the holders or operators of these terminal devices can be managers, supervisors, maintenance personnel, relevant department personnel, etc. of image and text recognition systems and recommendation systems.

[0072] For another example, in an application scenario where the method of this embodiment is executed on a system composed of multiple devices, the matching results can be sent directly to a preset terminal device through any data communication method. The terminal device can be one or more of the devices listed in the preceding paragraphs.

[0073] In some embodiments, if there is only one target vector, it can be directly output as a similarity result to the first vector. If there are multiple target vectors, in some embodiments, they can also be directly output, while in other embodiments, only the most similar results (one or a few) may be needed, which may require further processing. In some embodiments, after determining the first vector and multiple target vectors, distance or similarity calculations can be performed by calculating the Euclidean distance, cosine similarity, or other methods between these target vectors and the first vector. These target vectors can then be sorted based on the calculated results, and the top or most similar target vectors can be selected and output as the vectors most similar to the first vector. Specifically, in specific application scenarios, the vector distances between these target vectors and the first vector are calculated to obtain the results of an approximate nearest neighbor search. That is, in some embodiments, there is at least one target vector; after outputting the target vector as a matching result to the first vector, the method further includes: responsive to the fact that there are multiple target vectors, determining the Euclidean distances between the target vectors and the first vector, sorting the target vectors based on the Euclidean distances, and selecting a set number of target vectors based on the sorted results as similarity results to the first vector.

[0074] As can be seen from the above embodiments, the embodiments of the present disclosure provide a vector matching method on a data lake, the method comprising: obtaining a first vector; projecting the first vector onto a set vector using local sensitive hashing to obtain a first result; determining at least one second vector and at least one second result corresponding to the at least one second vector; comparing the first result with the at least one second result, determining a target result corresponding to the first result in the at least one second result, and determining the second vector corresponding to the target result as a target vector; and outputting the target vector as a matching result of the first vector. The present disclosure uses local sensitive hashing to perform retrieval by hash value matching. For the first vector to be matched and the known second vector, both can be structured by projecting the local sensitive hashing onto the set vector. While facilitating storage, it also facilitates the comparison between vectors. At the same time, such structured data is also conducive to distributed storage. The ability to perform distributed storage and processing enables tasks that could only be completed on a single processor to be placed on more processors for joint processing, which is bound to improve the corresponding response speed and processing efficiency. At the same time, the use of local sensitive hashing can also make data calculations relatively simple, ultimately greatly improving the user experience.

[0075] In specific application scenarios, the combination of locality-sensitive hashing (LSH) and data lake technology enables high-scalability, low-latency large-scale vector similarity retrieval. Its core includes modules such as LSH index construction, data expansion optimization, and batch retrieval processing, which support efficient retrieval at the scale of billions of data and are suitable for scenarios such as recommendation systems and semantic search. Specifically, the data lake can be used to store vector data, LSH indexes, and other structured information in a unified table; hash bucket indexes are generated through the LSH algorithm to support multi-node parallel computing and adapt to the scale of billions of vectors; and pre-computation, data expansion (Explode), and distributed connection technologies are used to significantly reduce I / O overhead. It has the following features: (1) High scalability: Combining the LakeSoul partitioning mechanism with the Spark distributed framework, it supports horizontal expansion to billions of data; (2) Flexible distance measurement: Compatible with multiple measurement methods such as Euclidean distance and cosine similarity; (3) Resource reuse: LSH calculations and data structures are reused in batch retrieval mode to reduce duplication overhead; (4) Data-index integration: Indexes and original data are uniformly stored in LakeSoul, reducing operation and maintenance complexity. Ultimately, in the specific application process, retrieval efficiency was improved: compared with the traditional ENN algorithm, the retrieval speed was increased by more than 10 times; resource utilization was optimized: I / O overhead in batch retrieval scenarios was reduced by 40%, and computing resources were allocated more evenly; support for ultra-large-scale data: billions of vector data sets can be processed to meet enterprise-level application needs.

[0076] It should be noted that the methods of the embodiments of the present disclosure can be performed by a single device, such as a computer or server. The methods of the embodiments of the present disclosure can also be applied in a distributed scenario, performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the methods of the embodiments of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0077] It should be noted that the above description is of specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0078] Based on the same concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a vector matching device on a data lake.

[0079] refer to Figure 2 , the vector matching device on the data lake includes:

[0080] The first module 210 is configured to obtain a first vector.

[0081] The second module 220 is configured to project the first vector onto a set vector using locality sensitive hashing to obtain a first result.

[0082] The third module 230 is configured to determine at least one second vector and at least one second result corresponding to the at least one second vector.

[0083] The fourth module 240 is configured to compare the first result with the at least one second result, determine a target result corresponding to the first result in the at least one second result, and determine the second vector corresponding to the target result as a target vector.

[0084] The fifth module 250 is configured to output the target vector as a matching result of the first vector.

[0085] In some exemplary embodiments, the setting vector is a plurality of setting vectors;

[0086] The second module 220 is further configured to:

[0087] The first vector is copied according to the number of the multiple setting vectors, so that each setting vector corresponds to a copied first vector, and the first result corresponding to each setting vector is obtained by using the locality sensitive hash.

[0088] In some exemplary embodiments, the number of the second results corresponding to any second vector is the same as the number of the set vectors.

[0089] In some exemplary embodiments, the third module 230 is further configured to:

[0090] Any second vector is projected onto the set vector according to the locality sensitive hash, so as to determine the second result corresponding to the any second vector.

[0091] In some exemplary embodiments, the third module 230 is further configured to:

[0092] obtaining the at least one second vector;

[0093] Projecting the at least one second vector onto the setting vector using the locality sensitive hash to obtain the at least one second result;

[0094] The at least one second result is stored in at least one database.

[0095] In some exemplary embodiments, the fourth module 240 is further configured to:

[0096] The first result is transmitted to the at least one database to complete the comparison in the at least one database.

[0097] In some exemplary embodiments, the number of the target vector is at least one;

[0098] The fifth module 250 is further configured to:

[0099] In response to the number of target vectors being multiple, determining the Euclidean distance between the target vector and the first vector, sorting the target vectors according to the Euclidean distance, and selecting a set number of the target vectors as similarity results of the first vector according to the sorting result.

[0100] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the embodiments of the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0101] The apparatus of the above embodiment is used to implement the corresponding vector matching method on the data lake in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0102] Based on the same concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the vector matching method on the data lake as described in any of the above embodiments is implemented.

[0103] Figure 3 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0104] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0105] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0106] The input / output interface 1030 is used to connect an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0107] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0108] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0109] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0110] The electronic device of the above embodiment is used to implement the corresponding vector matching method on the data lake in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0111] Based on the same concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the vector matching method on the data lake as described in any of the above embodiments.

[0112] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0113] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the vector matching method on the data lake as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Based on the same concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a computer program product comprising computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the vector matching method on the data lake. Corresponding to the execution subject corresponding to each step in each embodiment of the vector matching method on the data lake, the processor executing the corresponding step can belong to the corresponding execution subject.

[0115] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the vector matching method on the data lake as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0116] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.

[0117] In addition, to simplify the description and discussion, and so as not to obscure the embodiments of the present disclosure, known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, devices may be shown in the form of block diagrams to avoid obscuring the embodiments of the present disclosure, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure are to be implemented (i.e., these details should be fully within the purview of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be implemented without these specific details or with variations in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0118] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0119] The embodiments of the present disclosure are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A vector matching method on a data lake, characterized in that: include: Get the first vector; Projecting the first vector onto a set vector using locality sensitive hashing to obtain a first result; determining at least one second vector and at least one second result corresponding to the at least one second vector; comparing the first result with the at least one second result, determining a target result corresponding to the first result in the at least one second result, and determining the second vector corresponding to the target result as a target vector; The target vector is output as a matching result of the first vector.

2. The method according to claim 1, characterized in that The setting vector is a plurality of setting vectors; The projecting the first vector onto a set vector using locality sensitive hashing to obtain a first result includes: The first vector is copied according to the number of the multiple setting vectors, so that each setting vector corresponds to a copied first vector, and the first result corresponding to each setting vector is obtained by using the locality sensitive hash.

3. The method according to claim 2, characterized in that The number of the second results corresponding to any second vector is the same as the number of the set vectors.

4. The method according to claim 1, wherein The determining of at least one second vector and at least one second result corresponding to the at least one second vector includes: Any second vector is projected onto the set vector according to the locality sensitive hash, so as to determine the second result corresponding to the any second vector.

5. The method according to claim 1, wherein Before determining at least one second vector and at least one second result corresponding to the at least one second vector, the method further includes: obtaining the at least one second vector; Projecting the at least one second vector onto the setting vector using the locality sensitive hash to obtain the at least one second result; The at least one second result is stored in at least one database.

6. The method according to claim 5, characterized in that The comparing the first result with the at least one second result comprises: The first result is transmitted to the at least one database to complete the comparison in the at least one database.

7. The method according to claim 1, characterized in that The number of the target vector is at least one; After outputting the target vector as a matching result of the first vector, the method further includes: In response to the number of target vectors being multiple, determining the Euclidean distance between the target vector and the first vector, sorting the target vectors according to the Euclidean distance, and selecting a set number of the target vectors as similarity results of the first vector according to the sorting result.

8. A vector matching device on a data lake, characterized in that: include: A first module is used to obtain a first vector; A second module is configured to project the first vector onto a set vector using locality sensitive hashing to obtain a first result; A third module is configured to determine at least one second vector and at least one second result corresponding to the at least one second vector; a fourth module, configured to compare the first result with the at least one second result, determine a target result corresponding to the first result in the at least one second result, and determine the second vector corresponding to the target result as a target vector; The fifth module is configured to output the target vector as a matching result of the first vector.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to implement the method according to any one of claims 1 to 7.

Images