A similar data searching method, system and electronic device

Abstract

The specification discloses a similar data search method, system and electronic equipment, which can efficiently and accurately determine similar data, reduce computing cost and improve search efficiency. The method comprises: acquiring specified query data and a given data set, the given data set comprising a plurality of basic data; performing feature vector extraction on the specified query data and the plurality of basic data to generate corresponding multi-dimensional feature vectors, determining a query point corresponding to the specified query data and a plurality of data points corresponding to the plurality of basic data in a multi-dimensional vector space; generating corresponding vector approximate coding by interval division of a plurality of dimensions of the multi-dimensional vector space; constructing a nearest neighbor graph for the plurality of data points, performing nearest neighbor search in the plurality of data points based on the nearest neighbor graph and the vector approximate coding, determining the nearest neighbor data point of the query point, and thereby determining similar data.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, specifically to a similar data search method, system, and electronic device. Background Technology

[0002] In today's information society, big data technology is being used more and more widely. Its main component is semi-structured and unstructured data (such as text, images, audio, and video data), accounting for over 85% of the total collected data, and containing immense value. However, unstructured big data is characterized by its massive volume, heterogeneity, and complexity, posing unprecedented challenges to information storage, computing, and data processing technologies for various applications. To fully explore and utilize the value of unstructured big data, it is necessary to be able to efficiently process and analyze massive amounts of unstructured data. Similarity search, as a key fundamental problem in this process, has significant research importance.

[0003] Similarity search refers to the process of searching for data most similar to specified query data from a given dataset. Due to the structural complexity of unstructured data, it is difficult to directly calculate similarity. Some related techniques extract feature vectors to convert the data into data points in a vector space for nearest neighbor search. However, nearest neighbor search in high-dimensional space often suffers from the "curse of dimensionality," resulting in less than ideal performance. Other related techniques also often suffer from high computational costs and low search efficiency. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a similar data search method, system, and electronic device that can efficiently and accurately identify similar data, reduce computational costs, and improve search efficiency.

[0005] In a first aspect, embodiments of this specification provide a method for searching similar data, including:

[0006] Obtain specified query data and a given dataset, wherein the given dataset includes multiple basic data items, and both the specified query data and the multiple basic data items are unstructured data;

[0007] Feature vectors are extracted from the specified query data and multiple basic data to generate corresponding multidimensional feature vectors. Based on the multidimensional vectors, query points corresponding to the specified query data and multiple data points corresponding to the multiple basic data are determined in the multidimensional vector space.

[0008] The multidimensional vector space is divided into intervals according to multiple dimensions. Based on the interval assignment of the multidimensional components of the multidimensional feature vector in the corresponding dimension, the multidimensional feature vector is approximated and encoded to generate the corresponding vector approximation code.

[0009] A nearest neighbor graph is constructed for the multiple data points, and a nearest neighbor search is performed on the multiple data points based on the nearest neighbor graph and the vector approximation encoding to determine the nearest neighbor data point of the query point;

[0010] The basic data corresponding to the nearest neighbor data point is determined to be similar data corresponding to the given query data.

[0011] This specification also provides an embodiment of a similar data search system, including:

[0012] The data acquisition module is used to acquire specified query data and a given dataset, wherein the given dataset includes multiple basic data, and both the specified query data and the multiple basic data are unstructured data.

[0013] The feature vector extraction module is used to extract feature vectors from the specified query data and multiple basic data, generate corresponding multidimensional feature vectors, and determine the query point corresponding to the specified query data and multiple data points corresponding to the multiple basic data in the multidimensional vector space based on the multidimensional vectors.

[0014] An approximate encoding module is used to divide multiple dimensions of a multidimensional vector space into intervals, and to approximate encode the multidimensional feature vector according to the interval assignment of the multidimensional components of the multidimensional feature vector in the corresponding dimension, so as to generate a corresponding vector approximate code.

[0015] The nearest neighbor search module is configured to construct a nearest neighbor graph for multiple data points, and perform a nearest neighbor search among the multiple data points based on the nearest neighbor graph and the vector approximation encoding to determine the nearest neighbor data point of the query point; and

[0016] The similar data determination module is used to determine that the basic data corresponding to the nearest neighbor data point is similar data corresponding to the given query data.

[0017] This specification also provides an electronic device for searching similar data, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the similar data search method as described in the first aspect.

[0018] As can be seen from the above, the similar data search method, system, and electronic device provided in the embodiments of this specification have the following beneficial technical effects:

[0019] The described similar data search method determines multi-dimensional feature vectors of multiple basic data points in a given dataset and approximates these feature vectors to generate corresponding vector approximations. Based on these approximate vector codes, a nearest neighbor search is performed. Due to the good distance preservation capability and high computational efficiency of the approximate codes, the nearest neighbor data points can be determined more quickly, thereby identifying similar data corresponding to the specified query data. This approach can efficiently and accurately determine similar data, reducing computational costs and improving search efficiency. Attached Figure Description

[0020] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0021] Figure 1 This specification illustrates a schematic diagram of a similar data search method provided by one or more optional embodiments;

[0022] Figure 2 This illustration shows a method for approximate encoding in a similar data search method provided by one or more optional embodiments of this specification;

[0023] Figure 3 This diagram illustrates a method for constructing a nearest neighbor graph in one or more optional embodiments of a similar data search method provided in this specification;

[0024] Figure 4 This diagram illustrates a method for performing nearest neighbor search in a similar data search method provided by one or more optional embodiments of this specification;

[0025] Figure 5 This specification shows a schematic diagram of the structure of a similar data search system provided by one or more optional embodiments;

[0026] Figure 6 This specification illustrates a schematic diagram of the structure of a similar data search electronic device provided by one or more optional embodiments. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] In today's information society, big data technology is being used more and more widely. Its main component is semi-structured and unstructured data (such as text, images, audio, and video data), accounting for over 85% of the total collected data, and containing immense value. However, unstructured big data is characterized by its massive volume, heterogeneity, and complexity, posing unprecedented challenges to information storage, computing, and data processing technologies for various applications. To fully explore and utilize the value of unstructured big data, it is necessary to be able to efficiently process and analyze massive amounts of unstructured data. Similarity search, as a key fundamental problem in this process, has significant research importance.

[0029] Similarity search refers to the process of searching for the most similar data to a specified query from a given dataset. Due to the structural complexity of unstructured data, it is difficult to directly calculate similarity. Some related techniques extract feature vectors to convert the data into data points in a vector space for nearest neighbor search. However, nearest neighbor search in high-dimensional space often suffers from the "curse of dimensionality," resulting in less than ideal performance. Other related techniques for nearest neighbor search in high-dimensional space also often suffer from excessive computational load, high computational cost, and low search efficiency.

[0030] To address the aforementioned issues, the purpose of the embodiments in this specification is to propose a similar data search method, system, and electronic device. This method performs near-neighbor retrieval after approximate encoding of specified query data and data in a given dataset. Utilizing the good distance preservation capability and fast computation speed of approximate encoding, it effectively reduces the computational load of the scheme, improves retrieval efficiency, and enables faster and more accurate retrieval and identification of similar data from a given dataset.

[0031] For the purposes described above, this specification provides a similar data search method.

[0032] like Figure 1 As shown, one or more optional embodiments of this specification provide a similar data retrieval method, including:

[0033] S1: Obtain the specified query data and the given dataset, wherein the given dataset includes multiple basic data, and both the specified query data and the multiple basic data are unstructured data.

[0034] Similarity search aims to find the data items in a given dataset that are most similar to specified query data. The specified query data and the given dataset can be obtained through a data input interface, or the given dataset can be directly obtained from a database. The specified query data and several of the basic data types are unstructured data, such as image data, video data, and text data.

[0035] S2: Extract feature vectors from the specified query data and multiple basic data to generate corresponding multidimensional feature vectors, and determine the query point corresponding to the specified query data and multiple data points corresponding to the multiple basic data in the multidimensional vector space based on the multidimensional vectors.

[0036] Feature vectors are extracted from the specified query data and multiple basic data, transforming the originally unstructured specified query data and basic data into multi-dimensional feature vectors for representation and description.

[0037] For different types of unstructured data, corresponding feature vector extraction methods can be employed. For example, for image and video data, there are SIFT and HOG (Histogram of Oriented Gradients) feature extraction methods; for audio data, there are MFCC (Mellow Frequency Cepstral Coefficient) and PLP (Perceptual Linear Prediction Coefficients); for text data, there are TF-IDF and word2vec algorithms for feature extraction. Furthermore, deep learning models can be built for various types of unstructured data, and the corresponding high-dimensional vectors extracted by these models can be used as the high-dimensional feature vectors corresponding to the unstructured data.

[0038] By extracting feature vectors from unstructured data to generate corresponding multidimensional feature vectors, the specified query data and multiple basic data can be represented by a point in a multidimensional vector space. In the multidimensional vector space, the distance between points can be used to measure the similarity between corresponding unstructured data.

[0039] S3: Divide the multidimensional vector space into intervals according to the interval assignment of the multidimensional components of the multidimensional feature vector in the corresponding dimension, and approximate the multidimensional feature vector to generate the corresponding vector approximate code.

[0040] By dividing the multidimensional vector space into intervals along multiple dimensions, the interval affiliation of the multidimensional feature vector in each dimension of the vector space can be determined. Based on the interval affiliation, the multidimensional feature vector is approximated and encoded. The resulting approximate vector encoding actually uses the corresponding affiliation interval to represent the specific vector component. Therefore, the approximate vector encoding can, to a certain extent, represent the data characteristics of the unstructured data corresponding to the multidimensional feature vector. For the query point and the data point in the multidimensional vector space, the corresponding approximate vector encoding has a good distance preservation ability, and therefore can also be used to measure the similarity between data.

[0041] S4: Construct a nearest neighbor graph for the multiple data points, and perform a nearest neighbor search on the multiple data points based on the nearest neighbor graph and the vector approximation encoding to determine the nearest neighbor data point of the query point.

[0042] By constructing a nearest neighbor graph for multiple data points, multiple neighboring points can be determined for each data point. Then, a nearest neighbor search method can be used to iteratively search for data points closer to the query point from among the neighbors of the reference point, continuously moving closer to the nearest neighbor until the nearest neighbor is finally determined. During the iterative process of the nearest neighbor search, the approximate distance between points can be determined based on the query point and the approximate vector encodings corresponding to the multiple data points, thus significantly improving computational efficiency and reducing computational costs.

[0043] S5: Determine that the basic data corresponding to the nearest neighbor data point is similar data corresponding to the given query data.

[0044] The described similar data search method determines multi-dimensional feature vectors of multiple basic data points in a given dataset and approximates these feature vectors to generate corresponding vector approximations. Based on these approximate vector codes, a nearest neighbor search is performed. Due to the good distance preservation capability and high computational efficiency of the approximate codes, the nearest neighbor data points can be determined more quickly, thereby identifying similar data corresponding to the specified query data. This approach can efficiently and accurately determine similar data, reducing computational costs and improving search efficiency.

[0045] like Figure 2 As shown, in one or more optional embodiments of this specification, a similar data search method is provided, in which multiple dimensions of a multidimensional vector space are divided into intervals, and the multidimensional feature vector is approximated and encoded according to the interval assignment of the multidimensional components of the multidimensional feature vector in the corresponding dimensions to generate a corresponding vector approximation code, including:

[0046] S201: Calculate the range of values ​​of the multiple multidimensional feature vectors corresponding to the given dataset in multiple dimensions.

[0047] In a multidimensional vector space, only the portion containing specific data has partitioning significance. Therefore, we can first statistically determine the value range of multiple multidimensional feature vectors corresponding to multiple basic data in multiple dimensions. This value range can be denoted as [L]. i U i ] represents the range of values ​​in the i-th dimension.

[0048] S202: Divide the range of values ​​into multiple intervals on an average basis, and number each interval.

[0049] An optional implementation is to divide the value range into 2... b Each interval is divided into 2^b intervals and numbered sequentially. Each interval's number can be represented by a bit string of length b. For example, if b = 2, then each dimension is divided into 2^b intervals. 2 = There are 4 intervals, and each interval can be represented by a 2-bit string with the number: 00, 01, 10, 11. It can be understood that the specific number of intervals can be flexibly set according to the actual situation. The more intervals, the closer the generated vector approximates the multidimensional feature vector. The fewer intervals, the lower the computational complexity and the higher the computational efficiency.

[0050] S203: Determine the interval to which each dimension component of the multidimensional feature vector belongs, and sequentially merge the interval numbers corresponding to each dimension component as the approximate code of the vector.

[0051] For the multidimensional feature vector, determine the code corresponding to the interval it falls into in each dimension, and concatenate the codes of all dimension intervals to form the corresponding approximate code of the vector:

[0052]

[0053] Among them, C x This represents the approximate vector encoding corresponding to the basic data X in the given dataset. These represent the interval codes corresponding to multiple dimensions, and d represents the total number of dimensions of the multidimensional feature vector.

[0054] like Figure 3 As shown, in one or more optional embodiments of this specification, a similar data search method is provided, which constructs a nearest neighbor graph for multiple data points, including:

[0055] S301: Select multiple data points as vertices in sequence.

[0056] S302: Determine the distance between the vertex and the other multiple data points, and select the multiple data points with the smallest distance to the vertex as the vertex's neighbor points.

[0057] The distances between the vertex and multiple other data points can be sorted from smallest to largest, and the data points corresponding to the highest sorted distances can be selected as the vertex's neighbor points. The number of selected neighbor points can be flexibly set according to actual circumstances.

[0058] S303: Connect the vertex to the corresponding plurality of neighboring points using directed edges, wherein the directed edges point from the vertex to the neighboring points.

[0059] A nearest neighbor graph can be constructed for multiple data points corresponding to the given dataset using a linear scanning method. The nearest neighbor graph can more intuitively represent the distance relationship between multiple points. Using the nearest neighbor graph to assist the nearest neighbor search can greatly improve the search efficiency.

[0060] like Figure 4 As shown, in one or more optional embodiments of this specification, a similar data search method is provided, in which nearest neighbor search is performed on multiple data points based on the nearest neighbor graph and the vector approximation encoding to determine the nearest neighbor data points of the query point, including:

[0061] S401: Randomly select one of the multiple data points as the nearest neighbor candidate point o.

[0062] S402: Determine multiple neighbor points of the nearest neighbor candidate point based on the nearest neighbor graph.

[0063] S403: Determine the approximate distance between the query point and the multiple neighboring points based on the vector approximation encoding corresponding to the query point and the multiple neighboring points.

[0064] The approximate distance between the query point and the neighboring points:

[0065]

[0066] Where q and x represent the query point and the neighbor point, respectively, dist2(q,x) represents the approximate distance, d represents the number of dimensions of the multidimensional feature vector, and c q ,c x These represent the approximate vector codes corresponding to the query point and its neighboring points, respectively. These represent the interval numbers corresponding to the approximate vector codes of the query point and the neighbor point in the i-th dimension, respectively. These represent the midpoints of the intervals corresponding to the query point and the neighbor point in the i-th dimension, respectively.

[0067] In some alternative implementations, when dividing the multidimensional vector into intervals, the midpoints of multiple intervals and the distances between each pair of midpoints in each interval are also determined and recorded in a distance table.

[0068] When determining the approximate distance between the query point and the multiple neighboring points based on the vector approximation encoding corresponding to the query point and the multiple neighboring points, the midpoint of the interval corresponding to the query point and the neighboring points and the distance between the midpoints are determined by using a lookup table based on the distance table.

[0069] The lookup table method can quickly determine the midpoint of the interval corresponding to the query point and the neighbor points in the i-th dimension, as well as the distance between the midpoints. When calculating the approximate pluripotency, the already determined and recorded data can be used directly without recalculation, which can greatly improve the calculation efficiency.

[0070] S404: Select multiple neighboring points with the smallest approximate distance to the query point, and calculate the original distance between these multiple neighboring points and the query point.

[0071] When selecting multiple neighbor points with the smallest approximate distance to the query point q, the number of selected neighbor points is determined according to the number of neighbor points maintained for each data point in the nearest neighbor graph.

[0072] The number of neighbor points maintained for each data point in the nearest neighbor graph is denoted as g, and the number of neighbor points selected when choosing multiple neighbor points with the smallest approximate distance to the query point is τ.

[0073] τ = log₂g.

[0074] After selecting τ neighboring points that have the smallest approximate distance to the query point q, the original distances between the query point and the τ neighboring points are further calculated. These original distances refer to the Euclidean distances between the neighboring points and the query point. The Euclidean distances are determined based on the multidimensional feature vectors corresponding to the neighboring points and the query point q.

[0075] The Euclidean distance between the neighbor point and the query point can be denoted as ‖q,x‖.

[0076] S405: Select the point with the smallest original distance to the query point as the undetermined point, and compare the original distance between the undetermined point and the query point with the original distance between the nearest neighbor candidate point and the query point.

[0077] The original distance between the point to be determined and the query point is denoted as ||q,x||. * ||, x * The point to be determined is denoted as q,o. The original distance between the nearest neighbor candidate point and the query point is denoted as ‖q,o‖, where o represents the nearest neighbor candidate point.

[0078] S406: In response to the fact that the original distance corresponding to the undetermined point is less than the original distance corresponding to the nearest neighbor candidate point, the undetermined point is selected as a new nearest neighbor candidate point to continue the search.

[0079] If the original distance corresponding to the undetermined point is less than the original distance corresponding to the nearest neighbor candidate point, i.e., ||q,x|| *If ||<||q,o||, it means that the undetermined point is closer to the query point. The undetermined point can be used as a new nearest neighbor candidate point. Multiple neighbor points of the new nearest neighbor candidate point are determined, and the new undetermined point is re-searched and selected in the above manner for iterative search.

[0080] S407: In response to the fact that the original distance corresponding to the undetermined point is not less than the original distance corresponding to the nearest neighbor candidate point, determine the nearest neighbor candidate point as the nearest neighbor data point corresponding to the query point.

[0081] If the original distance ratio corresponding to the undetermined point is less than the original distance corresponding to the nearest neighbor candidate point, i.e., ||q,x|| * If the condition "<" is not met, it means that the current nearest neighbor candidate point is the point that is closest to the query point among the multiple data points, and it can be determined that the current nearest neighbor candidate point is the nearest neighbor data point.

[0082] It should be noted that the methods of one or more embodiments of this specification can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this specification, and the multiple devices will interact with each other to complete the method described.

[0083] It should be noted that the above description describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0084] Based on the same concept, corresponding to the methods of any of the above embodiments, this specification also provides a similar data search system.

[0085] refer to Figure 5 The similar data search system includes:

[0086] The data acquisition module is used to acquire specified query data and a given dataset, wherein the given dataset includes multiple basic data, and both the specified query data and the multiple basic data are unstructured data.

[0087] The feature vector extraction module is used to extract feature vectors from the specified query data and multiple basic data, generate corresponding multidimensional feature vectors, and determine the query point corresponding to the specified query data and multiple data points corresponding to the multiple basic data in the multidimensional vector space based on the multidimensional vectors.

[0088] An approximate encoding module is used to divide multiple dimensions of a multidimensional vector space into intervals, and to approximate encode the multidimensional feature vector according to the interval assignment of the multidimensional components of the multidimensional feature vector in the corresponding dimension, so as to generate a corresponding vector approximate code.

[0089] The nearest neighbor search module is configured to construct a nearest neighbor graph for multiple data points, and perform a nearest neighbor search among the multiple data points based on the nearest neighbor graph and the vector approximation encoding to determine the nearest neighbor data point of the query point; and

[0090] The similar data determination module is used to determine that the basic data corresponding to the nearest neighbor data point is similar data corresponding to the given query data.

[0091] In a similar data search system provided by one or more optional embodiments of this specification, the approximate encoding module is further configured to: statistically analyze the value range of multiple multidimensional feature vectors corresponding to the given dataset in multiple dimensions; divide the value range into multiple intervals on an average basis and number the multiple intervals respectively; determine the interval to which each dimension component of the multidimensional feature vector belongs, and sequentially merge the numbers of the intervals corresponding to each dimension component as the vector approximate encoding.

[0092] In a similar data search system provided by one or more optional embodiments of this specification, the nearest neighbor search module is further configured to sequentially select a plurality of data points as vertices; determine the distance between the vertex and the other plurality of data points, select the plurality of data points with the smallest distance to the vertex as the neighbor points of the vertex; and connect the vertex to the corresponding plurality of neighbor points using directed edges, wherein the directed edges point from the vertex to the neighbor points.

[0093] In a similar data search system provided by one or more optional embodiments of this specification, the nearest neighbor search module is further configured to: randomly select one of the multiple data points as a nearest neighbor candidate point; determine multiple neighbor points of the nearest neighbor candidate point based on the nearest neighbor graph; determine the approximate distance between the query point and the multiple neighbor points according to the vector approximation encoding corresponding to the query point and the multiple neighbor points; select multiple neighbor points with the smallest approximate distance to the query point, and calculate the original distance between these multiple neighbor points and the query point; select the point with the smallest original distance to the query point as a pending point, and compare the original distance between the pending point and the query point with the original distance between the nearest neighbor candidate point and the query point; if the original distance corresponding to the pending point is less than the original distance corresponding to the nearest neighbor candidate point, select the pending point as a new nearest neighbor candidate point to continue searching; if the original distance corresponding to the pending point is not less than the original distance corresponding to the nearest neighbor candidate point, determine the nearest neighbor candidate point as the nearest neighbor data point corresponding to the query point.

[0094] In a similar data search system provided by one or more optional embodiments of this specification, the nearest neighbor search module calculates the approximate distance using the following formula:

[0095] The approximate distance between the query point and the neighboring points:

[0096]

[0097] Where q and x represent the query point and the neighbor point, respectively, dist2(q,x) represents the approximate distance, d represents the number of dimensions of the multidimensional feature vector, and c q ,c x These represent the approximate vector codes corresponding to the query point and its neighboring points, respectively. These represent the interval numbers corresponding to the approximate vector codes of the query point and the neighbor point in the i-th dimension, respectively. These represent the midpoints of the intervals corresponding to the query point and the neighbor point in the i-th dimension, respectively.

[0098] In a similar data search system provided by one or more optional embodiments of this specification, the original distance refers to the Euclidean distance between the neighbor point and the query point; the Euclidean distance is calculated and determined based on the multidimensional feature vectors corresponding to the neighbor point and the query point.

[0099] In a similar data search system provided by one or more optional embodiments of this specification, when the nearest neighbor search module selects multiple neighbor points with the smallest approximate distance to the query point, the number of selected neighbor points is determined according to the number of neighbor points maintained for each data point in the nearest neighbor graph.

[0100] The number of neighbor points maintained for each data point in the nearest neighbor graph is denoted as g, and the number of neighbor points selected when choosing multiple neighbor points with the smallest approximate distance to the query point is τ.

[0101] τ = log₂g.

[0102] In a similar data search system provided by one or more optional embodiments of this specification, the approximate encoding module, when dividing the multidimensional vector into intervals, is also used to determine the midpoints of multiple intervals and record the distances between each pair of midpoints in a distance table.

[0103] When determining the approximate distance between the query point and the multiple neighbor points based on the vector approximation encoding corresponding to the query point and the multiple neighbor points, the nearest neighbor search module also uses a lookup table method based on the distance table to determine the midpoint of the interval corresponding to the query point and the neighbor points in the i-th dimension, as well as the distance between the midpoints.

[0104] For ease of description, the above system is described by dividing it into various modules based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware.

[0105] The system described above is used to implement the corresponding methods in the foregoing embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0106] Figure 6 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0107] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0108] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0109] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0110] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0111] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0112] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0113] The electronic devices described above are used to implement the corresponding methods in the foregoing embodiments and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0114] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the similar data search method as described in any of the above embodiments.

[0115] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0116] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the similar data search method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0118] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0119] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0120] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0123] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0124] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure (including the claims) is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this specification as described above, which are not provided in detail for the sake of brevity.

[0125] Although this disclosure has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0126] One or more embodiments of this specification are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the scope of protection of this disclosure.

Claims

1. A method for searching similar data, characterized in that, The method includes: Obtain specified query data and a given dataset, wherein the given dataset includes multiple basic data items, and both the specified query data and the multiple basic data items are unstructured data; Feature vectors are extracted from the specified query data and multiple basic data to generate corresponding multidimensional feature vectors. Based on the multidimensional feature vectors, query points corresponding to the specified query data and multiple data points corresponding to the multiple basic data are determined in the multidimensional vector space. The range of values ​​for multiple multidimensional feature vectors corresponding to the given dataset in multiple dimensions is statistically analyzed; the range of values ​​is divided into multiple intervals on an average basis, and the intervals are numbered respectively; the interval to which each dimension component of the multidimensional feature vector belongs is determined, and the numbers of the intervals corresponding to each dimension component are merged in turn as the vector approximate code; A nearest neighbor graph is constructed for multiple data points; one of the multiple data points is randomly selected as a nearest neighbor candidate point; multiple neighbor points of the nearest neighbor candidate point are determined based on the nearest neighbor graph; the approximate distance between the query point and the multiple neighbor points is determined according to the vector approximation encoding corresponding to the query point and the multiple neighbor points; multiple neighbor points with the smallest approximate distance to the query point are selected, and the original distance between these multiple neighbor points and the query point is calculated; the point with the smallest original distance to the query point is selected as a pending point, and the original distance between the pending point and the query point is compared with the original distance between the nearest neighbor candidate point and the query point; in response to the original distance corresponding to the pending point being less than the original distance corresponding to the nearest neighbor candidate point, the pending point is selected as a new nearest neighbor candidate point to continue the search; in response to the original distance corresponding to the pending point being not less than the original distance corresponding to the nearest neighbor candidate point, the nearest neighbor candidate point is determined as the nearest neighbor data point corresponding to the query point. The approximate distance between the query point and the neighboring points: in, These represent the query point and the neighbor point, respectively. This represents the approximate distance. This represents the number of dimensions of the multidimensional feature vector. These represent the approximate vector codes corresponding to the query point and its neighboring points, respectively. The approximate vector codes representing the query point and the neighbor points respectively in the 1st... The interval number corresponding to each dimension; They represent the first time. The midpoint of the interval corresponding to the query point and the neighbor point in each dimension. The basic data corresponding to the nearest neighbor data point is determined to be similar data corresponding to the specified query data.

2. The method according to claim 1, characterized in that, Constructing a nearest neighbor graph for multiple data points includes: Multiple data points are selected sequentially as vertices; Determine the distance between the vertex and the other multiple data points, and select the multiple data points with the smallest distance to the vertex as the vertex's neighbor points; The vertex is connected to a plurality of its neighboring points using directed edges, with the directed edges pointing from the vertex to the neighboring points.

3. The method according to claim 1, characterized in that, The original distance refers to the Euclidean distance between the neighbor point and the query point; The Euclidean distance is calculated based on the multidimensional feature vectors corresponding to the neighbor points and the query point.

4. The method according to claim 1, characterized in that, When selecting multiple neighbor points with the smallest approximate distance to the query point, the number of selected neighbor points is determined according to the number of neighbor points maintained for each data point in the nearest neighbor graph; The number of neighbor points maintained for each data point in the nearest neighbor graph is denoted as . The number of neighboring points selected when choosing multiple neighboring points with the smallest approximate distance to the query point. :

5. The method according to claim 1, characterized in that, When dividing the multidimensional vector space into intervals, the midpoints of multiple intervals and the distances between each pair of midpoints in each interval are also determined and recorded in a distance table. When determining the approximate distance between the query point and the multiple neighboring points based on the vector approximation encoding corresponding to the query point and the multiple neighboring points, the distance is determined by using a lookup table method based on the distance table. The query point, the midpoint of the interval corresponding to the neighbor point, and the distance between the midpoints are all defined in each dimension.

6. A similar data search system, characterized in that, The system includes: The data acquisition module is used to acquire specified query data and a given dataset, wherein the given dataset includes multiple basic data, and both the specified query data and the multiple basic data are unstructured data. The feature vector extraction module is used to extract feature vectors from the specified query data and multiple basic data, generate corresponding multidimensional feature vectors, and determine the query point corresponding to the specified query data and multiple data points corresponding to the multiple basic data in the multidimensional vector space based on the multidimensional feature vectors. An approximate encoding module is used to statistically analyze the value range of multiple multidimensional feature vectors corresponding to the given dataset in multiple dimensions; divide the value range into multiple intervals on an average basis and number the multiple intervals respectively; determine the interval to which each dimension component of the multidimensional feature vector belongs, and merge the numbers of the intervals corresponding to each dimension component in turn as the vector approximate encoding; The nearest neighbor search module is used to construct a nearest neighbor graph for multiple data points; randomly select one of the multiple data points as a nearest neighbor candidate point; determine multiple neighbor points of the nearest neighbor candidate point based on the nearest neighbor graph; determine the approximate distance between the query point and the multiple neighbor points according to the vector approximation encoding corresponding to the query point and the multiple neighbor points; select multiple neighbor points with the smallest approximate distance to the query point, and calculate the original distance between these multiple neighbor points and the query point; select the point with the smallest original distance to the query point as a pending point, and compare the original distance between the pending point and the query point with the original distance between the nearest neighbor candidate point and the query point; in response to the original distance corresponding to the pending point being less than the original distance corresponding to the nearest neighbor candidate point, select the pending point as a new nearest neighbor candidate point to continue the search; in response to the original distance corresponding to the pending point being not less than the original distance corresponding to the nearest neighbor candidate point, determine the nearest neighbor candidate point as the nearest neighbor data point corresponding to the query point; the approximate distance between the query point and the neighbor points is: in, These represent the query point and the neighbor point, respectively. This represents the approximate distance. This represents the number of dimensions of the multidimensional feature vector. These represent the approximate vector codes corresponding to the query point and its neighboring points, respectively. The approximate vector codes representing the query point and the neighbor points respectively in the 1st... The interval number corresponding to each dimension; They represent the first time. The midpoint of the interval corresponding to the query point and the neighbor point in each dimension; The similar data determination module is used to determine that the basic data corresponding to the nearest neighbor data point is similar data corresponding to the specified query data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

Images