Data retrieval method, apparatus, device, and storage medium

Abstract

The application discloses a data retrieval method and device, equipment and a storage medium. The data retrieval method comprises the following steps: obtaining a to-be-retrieved feature vector; determining a plurality of centroid feature vectors in a preset retrieval data set; respectively calculating the similarity between the plurality of centroid feature vectors and the to-be-retrieved feature vector; when the similarity between any centroid feature vector and the to-be-retrieved feature vector meets a preset condition, determining, based on a preset centroid deployment mapping relationship, a shard server corresponding to the centroid feature vector whose similarity to the to-be-retrieved feature vector meets the preset condition; and sending a data retrieval instruction to the corresponding shard server, wherein the data retrieval instruction carries the to-be-retrieved feature vector, so that the corresponding shard server performs data retrieval based on the to-be-retrieved feature vector to obtain at least one retrieval result feature vector. The technical scheme provided by the application can improve the throughput of data retrieval, reduce resource waste, and improve the efficiency of big data retrieval.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, specifically to a data retrieval method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence and big data, data retrieval is now being used more and more widely in daily life. For example, it can be used for product retrieval on shopping websites and for tracking suspects. It can quickly retrieve similar or related objects based on a given object.

[0003] In practical retrieval systems, the datasets being retrieved often consist of massive amounts of data, making it difficult for a single server's CPU and memory to handle the computational load. Therefore, existing technologies reduce the retrieval pressure on individual servers by randomly dividing the entire dataset into subsets and deploying these subsets to different servers. When a retrieval is needed, the target data is broadcast to these servers for individual searching. However, this approach wastes computational resources and results in low throughput. Therefore, a more efficient solution is needed. Summary of the Invention

[0004] To address the problems of existing technologies, this application provides a data retrieval method, apparatus, device, and storage medium. The technical solution is as follows:

[0005] This application provides a data retrieval method, the method comprising:

[0006] Obtain the feature vector to be retrieved;

[0007] Determine multiple centroid feature vectors in the preset retrieval dataset;

[0008] Calculate the similarity between the plurality of centroid feature vectors and the feature vector to be retrieved;

[0009] When the similarity between any centroid feature vector and the feature vector to be retrieved meets a preset condition, the sharding server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset condition is determined based on the preset centroid deployment mapping relationship.

[0010] A data retrieval instruction is sent to the corresponding shard server. The data retrieval instruction carries the feature vector to be retrieved, so that the corresponding shard server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector.

[0011] This application also provides a data processing method, the method comprising:

[0012] Obtain the original dataset, which includes multiple sample feature vectors;

[0013] Clustering is performed on the multiple sample feature vectors to obtain a preset retrieval dataset. The preset retrieval dataset includes multiple first feature vector sets, and each first feature vector set includes a corresponding centroid feature vector.

[0014] The preset retrieval dataset is divided into multiple second feature vector sets, wherein the second feature vector set includes at least one centroid feature vector corresponding to the first feature vector set;

[0015] The second feature vector set is sent to the corresponding sharding server for storage;

[0016] Establish a mapping relationship between the centroid feature vector and the corresponding sharding server, and use the mapping relationship as the preset centroid deployment mapping relationship.

[0017] In another aspect, this application provides a data retrieval apparatus, the apparatus comprising:

[0018] The module for obtaining the feature vector to be retrieved is used to obtain the feature vector to be retrieved.

[0019] The centroid feature vector determination module is used to determine multiple centroid feature vectors in a preset retrieval dataset, wherein the preset retrieval dataset is generated in advance based on clustering processing of multiple sample feature vectors;

[0020] The sharding server determination module is used to determine the sharding server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset conditions, based on the preset centroid deployment mapping relationship, when any centroid feature vector meets the preset conditions.

[0021] The retrieval instruction sending module is used to send a data retrieval instruction to the corresponding shard server. The data retrieval instruction carries the feature vector to be retrieved, so that the corresponding shard server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector.

[0022] Another aspect of this application provides a data processing apparatus, the apparatus comprising:

[0023] The original dataset acquisition module is used to acquire the original dataset, which includes multiple sample feature vectors.

[0024] The clustering module is used to perform clustering processing on the multiple sample feature vectors to obtain a preset retrieval dataset. The preset retrieval dataset includes multiple first feature vector sets, and each first feature vector set includes a corresponding centroid feature vector.

[0025] The data sharding module is used to shard the preset retrieval dataset to obtain multiple sets of second feature vectors, wherein the second set of feature vectors includes at least one centroid feature vector corresponding to the first set of feature vectors.

[0026] The data sending and storage module is used to send the second feature vector set to the corresponding sharding server for storage;

[0027] The mapping relationship establishment module is used to establish the mapping relationship between the centroid feature vector and the corresponding sharding server, and to use the mapping relationship as the preset centroid deployment mapping relationship.

[0028] In another aspect, this application provides an apparatus comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the data retrieval method.

[0029] In another aspect, this application provides a computer-readable storage medium storing at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the data retrieval method.

[0030] The data retrieval method, apparatus, device, and storage medium provided in this application have the following technical advantages:

[0031] This application determines multiple centroid feature vectors in a preset retrieval dataset, which is generated in advance by clustering multiple sample feature vectors. The similarity between each of the multiple centroid feature vectors and the target feature vector is calculated. When the similarity between any centroid feature vector and the target feature vector meets a preset condition, a shard server corresponding to the centroid feature vector whose similarity to the target feature vector meets the preset condition is determined based on a preset centroid deployment mapping relationship. A data retrieval instruction is sent to the corresponding shard server, enabling it to perform data retrieval based on the target feature vector and obtain at least one retrieval result feature vector. This is equivalent to only needing to send a data retrieval instruction to a specific shard server for retrieval, which helps improve data retrieval throughput, reduce resource waste, and enhance the efficiency and reliability of big data retrieval. By acquiring an original dataset containing multiple sample feature vectors and performing clustering on these feature vectors, a pre-defined retrieval dataset is obtained. This pre-defined retrieval dataset includes multiple sets of first feature vectors, each set containing a corresponding centroid feature vector. This clustering of feature vectors with similar characteristics facilitates the rapid and accurate determination of the approximate location of the retrieval result feature vector corresponding to the feature vector to be retrieved during subsequent retrieval, reducing computational load and minimizing resource waste. Then, the pre-defined retrieval dataset is partitioned to obtain multiple sets of second feature vectors, wherein each second feature vector set includes at least one... The first set of feature vectors mentioned above corresponds to the centroid feature vectors. Next, the second set of feature vectors mentioned above is sent to the corresponding sharding server for storage. This helps to divide the feature vectors of samples with similar features into the same shard. Thus, during actual retrieval, only the feature vector to be retrieved needs to be sent to the corresponding sharding server for retrieval, which greatly improves the retrieval throughput. Finally, a mapping relationship between the centroid feature vectors mentioned above and the corresponding sharding servers is established, and this mapping relationship is used as the preset centroid deployment mapping relationship. Thus, during retrieval, the corresponding sharding server that needs to be retrieved can be quickly and accurately determined according to the preset centroid deployment mapping relationship, improving the efficiency and throughput of data retrieval.

[0032] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0033] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of an application environment provided in an embodiment of this application;

[0035] Figure 2 This is a flowchart of a data retrieval method provided in an embodiment of this application;

[0036] Figure 3 This is a flowchart of another data retrieval method provided in the embodiments of this application;

[0037] Figure 4 This is a flowchart of another data retrieval method provided in the embodiments of this application;

[0038] Figure 5 This is a flowchart of another data retrieval method provided in the embodiments of this application;

[0039] Figure 6 This is a flowchart of another data retrieval method provided in the embodiments of this application;

[0040] Figure 7 This is a schematic diagram of a set of multiple second feature vectors obtained after partitioning the preset retrieval dataset according to an embodiment of this application;

[0041] Figure 8 This is a flowchart of another data retrieval method provided in the embodiments of this application;

[0042] Figure 9 This is a schematic diagram illustrating how a client displays search results data, as provided in an embodiment of this application.

[0043] Figure 10 This is a flowchart of a data processing method provided in an embodiment of this application;

[0044] Figure 11 This is a schematic diagram of a data retrieval device provided in an embodiment of this application;

[0045] Figure 12 This is a schematic diagram of a data processing device provided in an embodiment of this application;

[0046] Figure 13 This is a hardware structure block diagram of a server used to implement the above-described data retrieval method or data processing method, provided in an embodiment of this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout.

[0048] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0049] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a collective term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies applied based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to backend systems for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0050] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0051] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application environment provided by one embodiment of this application, such as... Figure 1 As shown, the application environment may include a client 100, an execution server 200, and multiple sharding servers 300.

[0052] In the embodiments of this specification, the client 100 can be used to acquire target data to be retrieved and send the target data to be retrieved to the execution server 200. The client 100 can also be used to receive retrieval result data sent by the execution server 200. Specifically, the client 100 may include a processor and a memory, wherein the processor can be used to acquire data, parse data, and control the transmission of data, etc.; the memory can be used to store the target data to be retrieved and the retrieval result data, etc. In one embodiment, the client 100 may also include a physical device with image / text acquisition function, such as a camera and / or a scanner, for acquiring images / text and sending the acquired images / text as target data to be retrieved to the execution server 200, wherein the image may be a 2D image or a 3D image; in another embodiment, the client 100 may also include a display unit, which can be used to display information input by the user, acquired information, or information provided to the user, such as displaying the retrieval result data received from the execution server 200 based on the display unit. In specific embodiments, the client 100 may include, but is not limited to, physical devices such as smartphones, desktop computers, tablets, laptops, digital assistants, smart wearable devices (e.g., smartwatches), smart speakers, and portable gaming devices. It may also include software running on the physical device, such as applications (APPs), web pages, and WeChat mini-programs. The operating system running on the client in this embodiment may include, but is not limited to, Android, iOS, Linux, and Windows.

[0053] In the embodiments of this specification, the execution server 200 can be used to obtain the feature vector to be retrieved; determine multiple centroid feature vectors in a preset retrieval dataset; calculate the similarity between the multiple centroid feature vectors and the feature vector to be retrieved; when the similarity between any centroid feature vector and the feature vector to be retrieved meets a preset condition, determine the shard server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset condition based on a preset centroid deployment mapping relationship; send a data retrieval instruction to the corresponding shard server, and the data retrieval instruction carries the feature vector to be retrieved, so that the corresponding shard server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector. The execution server 200 can also be used to acquire an original dataset, which includes multiple sample feature vectors; perform clustering processing on the multiple sample feature vectors to obtain a preset retrieval dataset, which includes multiple first feature vector sets, and each first feature vector set includes at least one centroid feature vector; partition the preset retrieval dataset to obtain multiple second feature vector sets, wherein each second feature vector set includes at least one centroid feature vector from the first feature vector set; send the second feature vector sets to the corresponding partition servers for storage; establish a mapping relationship between the centroid feature vectors and the corresponding partition servers, and use the above mapping relationship as the preset centroid deployment mapping relationship. Specifically, the execution server 200 may include a processor and a memory, wherein the processor can be used to receive instructions, parse data, and perform pre-retrieval set clustering and partitioning; the memory can be used to store sample data and corresponding sample feature vectors, etc. In the embodiments of this specification, the execution server 200 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The embodiments of this application are not limited thereto.

[0054] It should be noted that the client 100 can connect to the execution server 200 directly or indirectly based on various wireless communication methods or wired communication methods.

[0055] In the embodiments of this specification, the sharding server 300 can be used to perform data retrieval based on the feature vector to be retrieved, obtain at least one retrieval result feature vector, determine the retrieval result data corresponding to the aforementioned retrieval result feature vector, and send it to the execution server 200, so that the execution server 200 sends the retrieval result data to the client 100. Specifically, the sharding server 300 may include a processor and a memory, wherein the processor can be used to receive data retrieval instructions, execute data, and send data, etc.; the memory can be used to store a second feature vector set including multiple sample feature vectors, etc. In practical applications, the sharding server 300 may include a network communication unit, a processor, and a memory, etc. In the embodiments of this specification, the sharding server 300 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, etc. The embodiments of this application are not limited thereto.

[0056] Please refer to Figure 1 In practical applications, the execution server 200 receives the target data to be retrieved sent by the client 100, performs feature extraction on the target data to be retrieved, and obtains at least one target feature vector. The execution server 200 uses the above-mentioned at least one target feature vector as the feature vector to be retrieved. The execution server 200 determines multiple centroid feature vectors in a preset retrieval dataset and calculates the similarity between each of the multiple centroid feature vectors and the feature vector to be retrieved. When the similarity between any centroid feature vector and the feature vector to be retrieved meets a preset condition, the execution server 200, based on a preset centroid deployment mapping relationship, determines the shard server 300 corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset condition. The execution server 200 sends a data retrieval instruction to the corresponding shard server 300, which carries the aforementioned feature vector to be retrieved. The corresponding shard server 300 performs a data retrieval based on the feature vector to be retrieved, obtaining at least one retrieval result feature vector. The corresponding shard server 300 determines the retrieval result data corresponding to the aforementioned retrieval result feature vector and sends it to the execution server 200. The execution server 200 then sends the aforementioned retrieval result data to the client 100.

[0057] In addition, it should be noted that, Figure 1 This is merely an example of an application environment provided by one embodiment of this application, and this application is not limited thereto. For example, in practical applications, the client 100 may include more than just a display unit.

[0058] Figure 2 This is a flowchart of a data retrieval method provided in an embodiment of this application. Please refer to it. Figure 2 The data retrieval method provided in this application includes the following steps:

[0059] S201: The server retrieves the feature vector to be searched.

[0060] In the embodiments described in this specification, such as Figure 3 As shown, the process of obtaining the feature vector to be retrieved by the execution server may include the following steps:

[0061] S301: The client responds to the retrieval object retrieval command and retrieves the target data to be retrieved.

[0062] Specifically, the aforementioned target data to be retrieved can represent a reference or exemplary data. By using this target data to be retrieved as a reference standard, the data retrieval method provided in this specification can be used to quickly determine the retrieval results corresponding to the target data to be retrieved from massive amounts of data, thus meeting the user's actual retrieval needs. In a specific embodiment, the aforementioned retrieval results may include, but are not limited to, data that is the same as, similar to, or related to the target data to be retrieved. In the embodiments of this specification, the aforementioned target data to be retrieved may include, but is not limited to, images (including, but not limited to, 2D or 3D images, such as pictures captured or scanned in real time using image acquisition devices such as cameras, three-dimensional image data, facial image data, etc.), articles, audio, and video.

[0063] In one specific embodiment, the client's acquisition of target data to be retrieved may include: acquiring images in real time using an image acquisition device, which may include, but is not limited to, a camera and a scanner. The images may include, but are not limited to, 2D pictures, 3D image data, facial image data, etc. For example, a user may use a mobile phone camera to scan a target entity to obtain image data of that entity, in order to identify products similar to that target entity in a virtual network / e-commerce marketplace. In practical applications, the client's acquisition of target data to be retrieved may also include acquiring images downloaded by the user, or acquiring images pre-stored by the user (e.g., images from the user's mobile phone photo album).

[0064] S303: The client sends the target data to be retrieved to the execution server.

[0065] In the embodiments described in this specification, the client can connect to the execution server via a network using various wireless communication methods or wired communication methods to transmit data. This network can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0066] S305: The execution server performs feature extraction on the target data to be retrieved, and obtains at least one target feature vector.

[0067] In the embodiments of this specification, the target feature vector mentioned above may include a high-dimensional vector obtained after feature extraction of the target data to be retrieved (for example, the dimension of the high-dimensional vector may include 1024 dimensions), and the target feature vector can be used to represent the essential features of the target data to be retrieved, which is beneficial to improving the reliability of retrieval.

[0068] S307: The execution server uses at least one of the target feature vectors mentioned above as the feature vectors to be retrieved.

[0069] S203: The execution server determines multiple centroid feature vectors in the preset retrieval dataset.

[0070] Specifically, the preset retrieval dataset is generated in advance based on clustering of multiple sample feature vectors. In the embodiments of this specification, when the target data to be retrieved includes images, the preset retrieval dataset may include multiple sample feature vectors obtained by feature extraction from multiple sample images, followed by clustering.

[0071] Please refer to Figure 4 In the embodiments of this specification, before obtaining the feature vector to be retrieved, the above method may further include:

[0072] S401: Execute the server to obtain the raw dataset.

[0073] Specifically, the original dataset mentioned above includes multiple sample feature vectors.

[0074] In the embodiments described in this specification, such as Figure 5 As shown, the execution server described above may obtain the raw dataset in the following ways:

[0075] S501: The execution server retrieves multiple sample data.

[0076] In practical applications, based on users' search needs, it is often necessary to determine the search results data corresponding to the target data from massive amounts of data. The sample data can be on the order of tens of millions to hundreds of millions, which are used to build datasets for retrieval, improve the accuracy of retrieval, and help meet users' search needs.

[0077] S503: The execution server performs feature extraction on the above sample data to obtain the feature vector of the above sample data.

[0078] In the embodiments of this specification, the feature vector of the sample data may include a high-dimensional vector obtained after feature extraction of the corresponding sample data, and the feature vector of the sample data can be used to represent the essential features of the sample data, thereby enabling the use of the feature vector of the sample data to distinguish different sample data.

[0079] S505: The execution server uses the feature vectors of the above-mentioned multiple sample data as the above-mentioned multiple sample feature vectors, and uses the above-mentioned multiple sample feature vectors as the original dataset.

[0080] S403: The server performs clustering processing on the above multiple sample feature vectors to obtain the preset retrieval dataset.

[0081] Specifically, the aforementioned preset retrieval dataset includes multiple sets of first feature vectors, and each set of first feature vectors includes a corresponding centroid feature vector.

[0082] In the embodiments described in this specification, the execution server can perform clustering processing on the above-mentioned multiple sample feature vectors based on the k-means clustering algorithm, which may specifically include the following steps:

[0083] (1) Randomly select a preset number of feature vectors as initial centroid feature vectors, and generate an initial first feature vector set corresponding to each initial centroid feature vector;

[0084] Specifically, the preset number can be determined based on actual application requirements, such as the required number of groups.

[0085] (2) Calculate the distance between each sample feature vector in the original dataset and each initial centroid feature vector;

[0086] In practical applications, the aforementioned distances can include Euclidean distance or cosine distance.

[0087] (3) Determine the initial centroid feature vector that is closest to the sample feature vector, and assign the sample feature vector to the initial first feature vector set corresponding to the initial centroid feature vector that is closest to it;

[0088] (4) After classifying all sample feature vectors, recalculate the centroid of each initial first feature vector set, and update the initial centroid feature vector based on the recalculated centroid of the initial first feature vector set;

[0089] (5) Repeat steps (2) to (4) until the preset termination condition is met.

[0090] In practical applications, the aforementioned preset termination conditions may include, but are not limited to: the number of repetitions reaching the preset number of iterations, the recalculated centroid no longer changing, or the sum of squared errors being less than a preset error threshold.

[0091] (6) The centroid calculated at this time is taken as the centroid eigenvector of the initial first eigenvector set, and the initial first eigenvector set including the centroid eigenvector at this time is taken as the first eigenvector set mentioned above.

[0092] By performing clustering processing on the aforementioned multiple sample feature vectors by the execution server, a preset retrieval dataset is obtained. This preset retrieval dataset includes multiple sets of first feature vectors, each set of first feature vectors including a corresponding centroid feature vector. This is equivalent to generating an inverted index, using the centroid feature vector as index information. During subsequent retrieval, the similarity between the feature vector to be retrieved and each centroid feature vector can be calculated first. This helps to quickly and accurately determine the approximate location of the retrieval result feature vector corresponding to the feature vector to be retrieved when performing retrieval based on the feature vector to be retrieved, reducing the amount of computation, reducing resource waste, and improving the performance of the retrieval system.

[0093] By performing clustering processing on the aforementioned multiple sample feature vectors using the execution server, a preset retrieval dataset is obtained, comprising multiple sets of first feature vectors. Each set of first feature vectors includes a corresponding centroid feature vector. However, this preset retrieval dataset may still contain data on the order of tens to hundreds of millions. Therefore, it is necessary to further divide this preset retrieval dataset. Please refer to [link / reference needed]. Figure 4 In the embodiments described in this specification, the above method further includes:

[0094] S405: The execution server divides the preset retrieval dataset into multiple sets of second feature vectors.

[0095] Specifically, the second set of feature vectors mentioned above includes at least one centroid feature vector from the first set of feature vectors mentioned above.

[0096] Please refer to Figure 6 In one specific embodiment, the execution server divides the preset retrieval dataset into multiple sets of second feature vectors, which may include:

[0097] S601: The execution server determines the dimension of the feature vector of the above sample.

[0098] In practical applications, the dimensions of the feature vector to be retrieved are the same as those of the sample feature vector. For example, if the dimension of the sample feature vector is k (k is a positive integer), then the dimension of the corresponding feature vector to be retrieved is also k.

[0099] S603: The server calculates the variance of the sample feature vectors in each dimension of the preset retrieval dataset.

[0100] S605: The execution server performs median hyperplane segmentation on the sample feature vectors of each dimension in descending order of variance, resulting in multiple sets of second feature vectors.

[0101] In this embodiment of the specification, for example, if the preset retrieval dataset contains x sample feature vectors of dimension m, then the variance of each of the x sample feature vectors in each of the m dimensions is calculated, and the median hyperplane segmentation is performed on the sample feature vectors in each dimension in descending order of variance. In this embodiment of the specification, median hyperplane segmentation refers to taking a hyperplane based on the average value of the sample feature vectors in the target linear space. The hyperplane is an n-dimensional linear space of dimension n. A subspace of size 1 can partition the target linear space into two disjoint parts. For example, first, taking the median hyperplane for the dimension with the largest variance divides the preset retrieval dataset into 2 parts; then, taking the median hyperplane for the second largest variance of these two parts, dividing the preset retrieval dataset into 4 parts; then, taking the median hyperplane for the third largest variance of these 4 parts, dividing the preset retrieval dataset into 8 parts... and so on, until all dimensions have been partitioned using the median hyperplane, resulting in a total partition of 1 / 2 * ... This part will Each part serves as a set of multiple second feature vectors as described above.

[0102] To facilitate understanding of the median hyperplane segmentation process, this specification provides a specific embodiment for illustration: For example, the preset retrieval dataset has 6 sample feature vectors with a dimension of 2, namely: A(-1,-100), B(-2, -50), C(2, -21), D(8, 99), E(-11, 102), F(21, -66). The sequence of these 6 sample feature vectors in the first dimension is: -1, -2, 2, 8, -11, 21, and in the second dimension, it is -100, -50, -21, 99, 102, -66. Calculations show that the variance of these 6 sample feature vectors is largest in the second dimension, followed by the first dimension. Therefore, the median hyperplane is first taken for the second dimension.

[0103]

[0104] The preset retrieval dataset is now divided into two parts: the lower part includes sample feature vectors A(-1,-100), B(-2, -50), and F(21, -66); the upper part includes sample feature vectors C(2, -21), D(8, 99), and E(-11,102). Then, the median hyperplane is taken for each of these two parts along the first dimension.

[0105]

[0106]

[0107] The lower part is then further divided into a left part (including A(-1,-100), B(-2, -50)) and a right part (including F(21, -66)), and the upper part is also divided into a left part (including E(-11, 102)) and a right part (C(2, -21), D(8,99)). These four parts can be used as different sets of second feature vectors for subsequent storage on the corresponding sharding server. Those skilled in the art will understand that in practical applications, the dimensionality may be high (e.g., the dimension may include 1024 dimensions), and the number of sample feature vectors in the preset retrieval dataset can reach tens of millions to hundreds of millions.

[0108] Figure 7 This is a schematic diagram illustrating multiple sets of second feature vectors obtained after partitioning the preset retrieval dataset according to an embodiment of this specification. Please refer to it. Figure 7 , Figure 7 Each irregular polygon in the diagram represents a set of first feature vectors. Large dots represent the centroid feature vectors of each set of first feature vectors, and multiple small dots represent multiple sample feature vectors. A thick horizontal line and two thick vertical lines divide the preset retrieval dataset into four sets of second feature vectors: upper left, lower left, upper right, and lower right.

[0109] Since variance can be used to measure the dispersion of data, by determining the dimension of the sample feature vectors by the execution server, calculating the variance of the sample feature vectors in the preset retrieval dataset in each dimension, and performing median hyperplane segmentation on the sample feature vectors in each dimension in descending order of variance, multiple sets of second feature vectors are obtained. This is beneficial for reasonably and evenly partitioning the preset retrieval dataset, improving the performance of the retrieval system. Compared with the existing technology that randomly partitions the sample feature vectors and then sends the feature vectors to be retrieved to each partition server for retrieval, the solution provided in this application is beneficial for partitioning sample feature vectors with similar features into the same partition. Thus, in actual retrieval, only the feature vectors to be retrieved need to be sent to the corresponding partition server for retrieval, greatly improving the retrieval throughput.

[0110] In another specific embodiment, the execution server described above divides the preset retrieval dataset into multiple sets of second feature vectors, which may include:

[0111] The execution server performs clustering on the preset retrieval dataset to obtain multiple sets of second feature vectors.

[0112] In the embodiments of this specification, the execution server can perform clustering processing on the preset retrieval dataset based on the k-means clustering algorithm. The specific steps are similar to the clustering process in S403, and you can refer to the relevant description in S403. However, for example, if the number of groups to be determined in the first clustering process includes 100 (a total of 100 first feature vector sets), then the number of groups to be determined in the second clustering can include 10, which is equivalent to performing secondary clustering on the original dataset. Since the preset retrieval dataset already includes the first feature vector set (a smaller vector set) at this time, it can be understood that several second feature vector sets (larger vector sets) are generated through secondary clustering. Each second feature vector set may include multiple first feature vector sets.

[0113] It should be noted that dividing the preset retrieval dataset into multiple sets of second feature vectors by using the median hyperplane or re-clustering is merely an example, and this application is not limited to this.

[0114] S407: The execution server sends the above-mentioned second feature vector set to the corresponding sharding server for storage.

[0115] In the embodiments of this specification, the aforementioned second feature vector set can correspond one-to-one with a sharding server. A corresponding second feature vector set is deployed on a sharding server. By executing the server to shard the preset retrieval dataset to obtain multiple second feature vector sets, the aforementioned second feature vector sets are sent to the corresponding sharding servers for storage. This helps to reduce the retrieval pressure on a single server, reduce the computational load on a single server, and thus improve the reliability and stability of data retrieval.

[0116] S409: The execution server establishes the mapping relationship between the centroid feature vector and the corresponding shard server, and uses this mapping relationship as the preset centroid deployment mapping relationship.

[0117] By establishing a mapping relationship between the centroid feature vectors and the corresponding shard servers through the execution server, and using this mapping relationship as the preset centroid deployment mapping relationship, it is beneficial to quickly determine the shard server corresponding to the centroid feature vector after calculating the similarity between multiple centroid feature vectors and the feature vector to be retrieved during the subsequent retrieval process. Only the feature vector to be retrieved needs to be sent to the corresponding shard server, which improves the efficiency and throughput of data retrieval and reduces resource waste.

[0118] S205: The execution server calculates the similarity between multiple centroid feature vectors and the feature vector to be retrieved.

[0119] In the embodiments of this specification, the execution server may calculate the similarity between multiple centroid feature vectors and the feature vector to be retrieved, including but not limited to:

[0120] The execution server calculates the Euclidean distance between multiple centroid feature vectors and the feature vector to be retrieved;

[0121] or;

[0122] The execution server calculates the cosine angle between multiple centroid feature vectors and the feature vector to be retrieved.

[0123] S207: When the similarity between any centroid feature vector and the feature vector to be retrieved meets the preset conditions, the execution server determines the shard server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset conditions based on the preset centroid deployment mapping relationship.

[0124] In the embodiments of this specification, when the similarity between any centroid feature vector and the feature vector to be retrieved meets a preset condition, the execution server determines the shard server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset condition based on the preset centroid deployment mapping relationship. This can include, but is not limited to:

[0125] The similarity scores are sorted in descending order between the centroid feature vector and the feature vector to be retrieved. The centroid feature vectors corresponding to the top preset number of similarity scores are then determined. Based on a preset centroid deployment mapping relationship, the sharding server corresponding to the centroid feature vectors whose similarity scores are among the top preset number of similarity scores to the feature vector to be retrieved is determined. Specifically, the top preset number can be determined based on actual retrieval requirements, for example, it can be the top 1, top 3, or top 10.

[0126] By sorting the similarity scores from highest to lowest, the centroid feature vectors corresponding to the top preset number of similarity scores are determined. This helps to select the centroid feature vectors most similar to the feature vector to be retrieved. Based on the preset centroid deployment mapping relationship, the corresponding sharding server is determined. The feature vector to be retrieved only needs to be sent to the corresponding sharding server, which improves the data retrieval throughput and retrieval efficiency, and helps to quickly determine the results that are similar to or related to the feature vector to be retrieved.

[0127] In the embodiments of this specification, when the similarity between any centroid feature vector and the feature vector to be retrieved meets a preset condition, the execution server determines the shard server corresponding to the centroid feature vector whose similarity meets the preset condition based on the preset centroid deployment mapping relationship. This can include, but is not limited to:

[0128] When it is determined that the similarity between any centroid feature vector and the feature vector to be retrieved is higher than a preset retrieval threshold, the execution server, based on a preset centroid deployment mapping relationship, determines the shard server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset condition. Specifically, the preset retrieval threshold can be determined in conjunction with actual retrieval needs.

[0129] When the similarity between any centroid feature vector and the feature vector to be retrieved is determined to be higher than a preset retrieval threshold, the execution server determines the shard server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset conditions based on the preset centroid deployment mapping relationship. This improves the data retrieval throughput while enhancing the reliability and accuracy of the retrieval, and helps to more comprehensively determine the results that are similar to or related to the feature vector to be retrieved.

[0130] S209: The execution server sends a data retrieval command to the corresponding shard server.

[0131] Specifically, the aforementioned data retrieval instruction carries the feature vector to be retrieved.

[0132] S211: The corresponding sharding server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector.

[0133] In the embodiments of this specification, the corresponding sharding server performs data retrieval based on the feature vector to be retrieved, and the resulting feature vector may include at least one retrieval result feature vector:

[0134] (1) The corresponding shard server calculates the similarity between the feature vector to be retrieved and the sample feature vector on the corresponding shard server.

[0135] (2) The corresponding sharding server determines the sample feature vector that meets the preset search result conditions and uses it as the above-mentioned search result feature vector.

[0136] In the embodiments of this specification, the corresponding sharding server calculates the similarity between the feature vector to be retrieved and the sample feature vector on the corresponding sharding server, and determines the sample feature vector that meets the preset retrieval result conditions and uses it as the above-mentioned retrieval result feature vector. This may also include sorting the similarity in descending order of similarity, determining the sample feature vectors that are among the first preset number of similarity rankings, and using them as the above-mentioned retrieval result feature vectors; or; when it is determined that the similarity between the sample feature vector and the feature vector to be retrieved is higher than a preset result threshold, it is used as the above-mentioned retrieval result feature vector, which is beneficial to improving the reliability of data retrieval.

[0137] like Figure 8 As shown in the embodiments of this specification, after the corresponding sharding server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector, the above method further includes:

[0138] S801: The corresponding sharding server determines the search result data corresponding to the above-mentioned search result feature vector.

[0139] In the embodiments of this specification, since the feature vector is extracted from the corresponding data, after the corresponding sharding server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector, the retrieval result data corresponding to the above-mentioned retrieval result feature vector can be further determined.

[0140] S803: The corresponding sharding server sends the retrieval result data to the execution server.

[0141] S805: The server executes the request and sends the search results data to the client.

[0142] In practical applications, the aforementioned client can display the search results data based on the display unit.

[0143] In one specific embodiment, a user can use the camera of a mobile client to scan a physical object (e.g., a bottle of beverage) to identify the product most similar to that object. Data retrieval can be performed by determining the similarity between product images in a sample database and images of physical objects collected by the mobile client. Please refer to [reference needed]. Figure 9 , Figure 9 This is a schematic diagram illustrating how a client displays search results data, as provided in an embodiment of this application. Figure 9The upper part shows the user's scan of a beverage bottle, and the lower part shows the product information returned by the server that is closest to the scanned image.

[0144] In the embodiments of this specification, when the target feature vector obtained after feature extraction of the target data to be retrieved includes multiple features (for example, multiple feature corner points of the target data to be retrieved are extracted, and these feature corner points are used as the target feature vectors mentioned above), using the target feature vectors as the target feature vectors to be retrieved and performing feature retrieval may include:

[0145] For each target feature vector, it is used as the feature vector to be retrieved, and data retrieval is performed according to steps S203~S211 to obtain the retrieval result feature vector corresponding to each target feature vector. The retrieval result data corresponding to the aforementioned retrieval result feature vector is determined. When the number of occurrences of the same retrieval result data exceeds a preset number (this preset number can be determined based on actual application requirements), the retrieval result data is output as the final retrieval result. For example, five feature vectors to be retrieved from an image are extracted. Data retrieval is performed on these five feature vectors to obtain the top 10 feature vectors with high similarity to each of the five feature vectors, and each feature vector corresponds to a retrieval result image. When the number of occurrences of the same retrieval result image exceeds a preset number (e.g., 3 times), meaning that at least 3 feature vectors in the image are similar to the feature vectors of the image to be retrieved, the two images are determined to be similar, and this image can be output as the final retrieval result. This approach improves the accuracy of data retrieval. When retrieving data for each target feature vector, the execution server first determines multiple centroid feature vectors in the preset retrieval dataset. Then, the execution server calculates the similarity between these centroid feature vectors and the target feature vector. When the similarity between any centroid feature vector and the target feature vector meets a preset condition, the execution server, based on the preset centroid deployment mapping relationship, determines the shard server corresponding to the centroid feature vector whose similarity to the target feature vector meets the preset condition. Next, the execution server sends a data retrieval instruction carrying the target feature vector to the corresponding shard server, enabling the shard server to perform data retrieval based on the target feature vector and obtain at least one retrieval result feature vector. By determining the corresponding shard server based on the similarity of the feature vectors when retrieving data for each target feature vector, the feature vector on this corresponding shard server has a higher similarity to the target feature vector compared to other shard servers. Each time, only a portion of the shard servers need to be sent for retrieval, which improves the efficiency and throughput of data retrieval and reduces resource waste.

[0146] In the embodiments of this specification, the execution server extracts features from the target data to be retrieved sent by the client to obtain at least one target feature vector. This target feature vector can be used to represent the essential features of the target data to be retrieved, which helps improve the reliability of the retrieval. The execution server determines multiple centroid feature vectors in a preset retrieval dataset, calculates the similarity between each of the multiple centroid feature vectors and the target feature vector, and when the similarity between any centroid feature vector and the target feature vector meets a preset condition, based on a preset centroid deployment mapping relationship, determines the shard server corresponding to the centroid feature vector whose similarity to the target feature vector meets the preset condition, and sends a data retrieval instruction to the corresponding shard server, so that the corresponding shard server performs data retrieval based on the target feature vector and obtains at least one retrieval result feature vector. This is equivalent to only needing to send a data retrieval instruction to a specific shard server for retrieval, which helps improve the throughput of data retrieval, reduce resource waste, improve the efficiency and reliability of big data retrieval, and enhance the comprehensiveness and accuracy of data retrieval. The execution server performs clustering processing on multiple sample feature vectors to obtain a preset retrieval dataset. Using centroid feature vectors as index information, subsequent retrievals can first calculate the similarity between the target feature vector and each centroid feature vector. This facilitates quickly and accurately determining the approximate location of the corresponding retrieval result feature vector when searching based on the target feature vector, reducing computational load and improving the performance of the retrieval system. The execution server further partitions the preset retrieval dataset into multiple sets of second feature vectors, and sends these sets to the corresponding partition servers for storage. This facilitates a reasonable and even partitioning of the preset retrieval dataset, improving the performance of the retrieval system and reducing the retrieval pressure and computational load on individual servers, thereby enhancing the reliability and stability of data retrieval.

[0147] By establishing a mapping relationship between centroid feature vectors and corresponding shard servers through the execution server, and using this mapping relationship as the aforementioned preset centroid deployment mapping relationship, it is beneficial to quickly determine the shard server corresponding to the centroid feature vector after calculating the similarity between multiple centroid feature vectors and the feature vector to be retrieved during the subsequent retrieval process. Only the feature vector to be retrieved needs to be sent to the corresponding shard server, which improves the efficiency and throughput of data retrieval and reduces resource waste.

[0148] This application also provides a data processing method, please refer to... Figure 10 The data processing methods described above may include:

[0149] S1001: Execute the server to obtain the original dataset.

[0150] Specifically, the original dataset mentioned above may include multiple sample feature vectors.

[0151] In the embodiments of this specification, the process of the execution server obtaining the original dataset in S1001 is similar to that in S401. It may include the execution server obtaining multiple sample data and performing feature extraction on the sample data to obtain the feature vector of the sample data. Then, the execution server uses the feature vector of the multiple sample data as the multiple sample feature vector and uses the multiple sample feature vector as the original dataset. For specific steps, please refer to the relevant description of S401, which will not be repeated here.

[0152] S1003: The server performs clustering processing on the above multiple sample feature vectors to obtain the preset retrieval dataset.

[0153] Specifically, the aforementioned preset retrieval dataset includes multiple sets of first feature vectors, and each set of first feature vectors includes a corresponding centroid feature vector.

[0154] In the embodiments of this specification, the process of the execution server performing clustering processing on the above-mentioned multiple sample feature vectors to obtain a preset retrieval dataset in S1003 is similar to that in S403. It may include the execution server performing clustering processing on the above-mentioned multiple sample feature vectors based on the k-means clustering algorithm. For specific steps, please refer to the relevant description in S403, which will not be repeated here.

[0155] By performing clustering processing on the aforementioned multiple sample feature vectors using the execution server, a pre-defined retrieval dataset is obtained. This facilitates reasonable data processing of the sample feature vectors, helps to group similar sample feature vectors together, and improves the orderliness of the data processing results. Furthermore, it is equivalent to generating an inverted index, using centroid feature vectors as index information. During subsequent retrieval, the similarity between the feature vector to be retrieved and each centroid feature vector can be calculated first. This helps to quickly and accurately determine the approximate location of the retrieval result feature vector corresponding to the feature vector to be retrieved when performing retrieval based on the feature vector to be retrieved, reducing computational load, minimizing resource waste, and improving the performance of the retrieval system.

[0156] S1005: The execution server divides the preset retrieval dataset into multiple sets of second feature vectors.

[0157] Specifically, the aforementioned second feature vector set includes at least one centroid feature vector corresponding to the aforementioned first feature vector set.

[0158] In the embodiments of this specification, the process of the execution server partitioning the preset retrieval dataset into multiple sets of second feature vectors in S1005 is similar to that in S405. In one embodiment, the partitioning of the preset retrieval dataset into multiple sets of second feature vectors may include: the execution server determining the dimensions of the sample feature vectors; the execution server calculating the variance of the sample feature vectors in the preset retrieval dataset in each dimension; and the execution server performing median hyperplane segmentation on the sample feature vectors in each dimension in descending order of variance to obtain multiple sets of second feature vectors, which is beneficial for reasonably and evenly partitioning the preset retrieval dataset. In another embodiment, the partitioning of the preset retrieval dataset into multiple sets of second feature vectors may include: the execution server performing clustering processing on the preset retrieval dataset to obtain multiple sets of second feature vectors. Specific steps can be found in the relevant description of S405, and will not be repeated here.

[0159] By dividing the preset retrieval dataset into multiple sets of second feature vectors by the execution server, it is beneficial to divide the feature vectors of samples with similar features into the same set, thereby improving the rationality and orderliness of data processing. In the actual retrieval, it is only necessary to send the feature vector to be retrieved to the corresponding set server for retrieval, which greatly improves the retrieval throughput.

[0160] S1007: The execution server sends the above-mentioned second feature vector set to the corresponding shard server for storage.

[0161] In the embodiments of this specification, the process of the execution server sending the above-mentioned second feature vector set to the corresponding sharding server for storage in S1007 is similar to that in S407. For details, please refer to the relevant description of S407, which will not be repeated here.

[0162] By dividing the preset retrieval dataset into multiple sets of second feature vectors using the execution server, and sending these sets of second feature vectors to the corresponding shard servers for storage, the retrieval pressure on individual servers can be reduced, the computational load on individual servers can be decreased, and the reliability and stability of data retrieval can be improved.

[0163] S1009: The execution server establishes the mapping relationship between the above centroid feature vector and the corresponding shard server, and uses the above mapping relationship as the preset centroid deployment mapping relationship.

[0164] In the embodiments of this specification, the process of S1009, in which the execution server establishes the mapping relationship between the centroid feature vector and the corresponding sharding server and uses the mapping relationship as the preset centroid deployment mapping relationship, is similar to that of S409. For details, please refer to the relevant description of S409, which will not be repeated here.

[0165] By establishing a mapping relationship between the centroid feature vectors and their corresponding shard servers through the execution server, and using this mapping relationship as a preset centroid deployment mapping relationship, it is equivalent to establishing a correspondence between each centroid feature vector and a shard server. This is beneficial for improving the orderliness of data processing, and in turn, it is beneficial for subsequent retrieval processes to calculate the similarity between multiple centroid feature vectors and the feature vector to be retrieved. Using the preset centroid deployment mapping relationship, after determining the centroid feature vector with a high similarity to the feature vector to be retrieved, the shard server corresponding to the centroid feature vector can be quickly determined. Only the feature vector to be retrieved needs to be sent to the corresponding shard server, improving the efficiency and throughput of data retrieval and reducing resource waste.

[0166] This application also provides a data retrieval device, such as... Figure 11 As shown, the device may include:

[0167] The feature vector acquisition module 1110 is used to acquire the feature vector to be retrieved.

[0168] The centroid feature vector determination module 1120 is used to determine multiple centroid feature vectors in a preset retrieval dataset;

[0169] The sharding server determination module 1130 is used to determine the sharding server corresponding to the centroid feature vector whose similarity to the feature vector to be retrieved meets the preset conditions, based on the preset centroid deployment mapping relationship, when any centroid feature vector and the feature vector to be retrieved meet the preset conditions.

[0170] The retrieval instruction sending module 1140 is used to send a data retrieval instruction to the corresponding shard server. The data retrieval instruction carries the feature vector to be retrieved, so that the corresponding shard server performs data retrieval based on the feature vector to be retrieved and obtains at least one retrieval result feature vector.

[0171] In one embodiment, the data retrieval device described above may further include:

[0172] The original dataset acquisition module is used to acquire the original dataset, which includes multiple sample feature vectors.

[0173] The clustering module is used to perform clustering processing on the multiple sample feature vectors to obtain a preset retrieval dataset. The preset retrieval dataset includes multiple first feature vector sets, and each first feature vector set includes a corresponding centroid feature vector.

[0174] The data sharding module is used to shard the preset retrieval dataset to obtain multiple second feature vector sets, wherein the second feature vector set includes at least one centroid feature vector of the first feature vector set;

[0175] The data sending and storage module is used to send the second feature vector set to the corresponding sharding server for storage;

[0176] The mapping relationship establishment module is used to establish the mapping relationship between the centroid feature vector and the corresponding sharding server, and to use the mapping relationship as the preset centroid deployment mapping relationship.

[0177] In one embodiment, the data sharding module described above may include:

[0178] A dimension determination unit is used to determine the dimension of the sample feature vector;

[0179] The variance calculation unit is used to calculate the variance of the sample feature vectors in the preset retrieval dataset in each dimension.

[0180] The median hyperplane segmentation unit is used to perform median hyperplane segmentation on the sample feature vectors of each dimension in descending order of variance, resulting in multiple sets of second feature vectors.

[0181] In another embodiment, the data sharding module described above may include:

[0182] Clustering units are used to perform clustering processing on the preset retrieval dataset to obtain multiple sets of second feature vectors.

[0183] In the embodiments described in this specification, the original dataset acquisition module described above may include:

[0184] The sample data acquisition unit is used to acquire multiple sample data.

[0185] A sample feature extraction unit is used to extract features from the sample data to obtain a feature vector of the sample data;

[0186] The original dataset determination unit is used to use the feature vectors of the plurality of sample data as the plurality of sample feature vectors, and the plurality of sample feature vectors as the original dataset.

[0187] In the embodiments described in this specification, the above-described apparatus may further include:

[0188] The retrieval result data receiving module is used to receive at least one retrieval result data sent by the corresponding shard server. The at least one retrieval result data is the retrieval result data corresponding to the retrieval result feature vector determined by the corresponding shard server after performing data retrieval based on the feature vector to be retrieved and obtaining at least one retrieval result feature vector.

[0189] The search result data forwarding module is used to send the search result data to the client.

[0190] This application also provides a data processing apparatus, such as... Figure 12 As shown, the device may include:

[0191] The original dataset acquisition module 1210 is used to acquire the original dataset, which includes multiple sample feature vectors;

[0192] Clustering module 1220 is used to perform clustering processing on the multiple sample feature vectors to obtain a preset retrieval dataset. The preset retrieval dataset includes multiple first feature vector sets, and each first feature vector set includes a corresponding centroid feature vector.

[0193] The data sharding module 1230 is used to shard the preset retrieval dataset to obtain multiple second feature vector sets, wherein the second feature vector set includes at least one centroid feature vector corresponding to the first feature vector set;

[0194] The data transmission and storage module 1240 is used to send the second feature vector set to the corresponding sharding server for storage.

[0195] The mapping relationship establishment module 1250 is used to establish the mapping relationship between the centroid feature vector and the corresponding sharding server, and to use the mapping relationship as the preset centroid deployment mapping relationship.

[0196] In one embodiment, the data sharding module 1230 described above may include:

[0197] A dimension determination unit is used to determine the dimension of the sample feature vector;

[0198] The variance calculation unit is used to calculate the variance of the sample feature vectors in the preset retrieval dataset in each dimension.

[0199] The median hyperplane segmentation unit is used to perform median hyperplane segmentation on the sample feature vectors of each dimension in descending order of variance, resulting in multiple sets of second feature vectors.

[0200] In another embodiment, the data sharding module 1230 described above may include:

[0201] Clustering units are used to perform clustering processing on the preset retrieval dataset to obtain multiple sets of second feature vectors.

[0202] In the embodiments described in this specification, the raw dataset acquisition module 1210 described above may include:

[0203] The sample data acquisition unit is used to acquire multiple sample data.

[0204] A sample feature extraction unit is used to extract features from the sample data to obtain a feature vector of the sample data;

[0205] The original dataset determination unit is used to use the feature vectors of the plurality of sample data as the plurality of sample feature vectors, and the plurality of sample feature vectors as the original dataset.

[0206] This application provides a computer device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the data retrieval method provided in the above method embodiments.

[0207] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the data processing method provided in the above method embodiments.

[0208] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0209] The methods and embodiments provided in this application can be executed in a mobile terminal, computer terminal, server, or similar computing device; that is, the aforementioned computer device may include a mobile terminal, computer terminal, server, or similar computing device. Taking running on a server as an example... Figure 13This is a hardware structure block diagram of a server used to implement the above-described data retrieval method or data processing method, provided in an embodiment of this application. Figure 13 As shown, the server 1300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1310 (CPUs 1310 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1330 for storing data, and one or more storage media 1320 (e.g., one or more mass storage devices) for storing application programs 1323 or data 1322. The memory 1330 and storage media 1320 may be temporary or persistent storage. The program stored in the storage media 1320 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 1310 may be configured to communicate with the storage media 1320 and execute the series of instruction operations stored in the storage media 1320 on the server 1300. Server 1300 may also include one or more power supplies 1360, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1340, and / or one or more operating systems 1321, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0210] The processor 1310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0211] The input / output interface 1340 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1300. In one example, the input / output interface 1340 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1340 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0212] The operating system 1321 may include system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks.

[0213] Those skilled in the art will understand that Figure 13 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1300 may also include... Figure 13 The more or fewer components shown, or having the same Figure 13 The different configurations shown.

[0214] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a data retrieval method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data retrieval method provided in the above method embodiments.

[0215] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a data processing method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the data processing method provided in the above method embodiments.

[0216] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0217] Embodiments of this application also provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0218] As can be seen from the embodiments of the data retrieval method, apparatus, computer equipment, storage medium, or computer program provided in this application, this application extracts features from the target data to be retrieved sent by the client through an execution server, obtaining at least one target feature vector. This target feature vector can represent the essential features of the target data to be retrieved, which is beneficial for improving the reliability of the retrieval. The execution server determines multiple centroid feature vectors in a preset retrieval dataset, calculates the similarity between these centroid feature vectors and the target feature vector, and when the similarity between any centroid feature vector and the target feature vector meets a preset condition, it determines the shard server corresponding to the centroid feature vector whose similarity to the target feature vector meets the preset condition based on a preset centroid deployment mapping relationship. This allows the corresponding shard server to perform data retrieval based on the target feature vector, obtaining at least one retrieval result feature vector. This is equivalent to only needing to send a data retrieval command to a specific shard server for retrieval, which is beneficial for improving the throughput of data retrieval, reducing resource waste, improving the efficiency and reliability of big data retrieval, and enhancing the comprehensiveness and accuracy of data retrieval. As can be seen from the embodiments of the data processing method, apparatus, computer equipment, storage medium, or computer program provided in this application, this application obtains a pre-defined retrieval dataset by acquiring an original dataset including multiple sample feature vectors and performing clustering processing on the multiple sample feature vectors. This pre-defined retrieval dataset includes multiple sets of first feature vectors, each set including a corresponding centroid feature vector. This facilitates grouping sample feature vectors with similar features together, improving the orderliness of the data processing results. Furthermore, it facilitates quickly and accurately determining the approximate location of the retrieval result feature vector corresponding to the feature vector to be retrieved during subsequent retrieval, reducing computational load and minimizing resource waste. Then, the pre-defined retrieval dataset is partitioned to obtain multiple second feature vectors. The first feature vector set includes at least one centroid feature vector corresponding to the first feature vector set. Next, the second feature vector set is sent to the corresponding sharding server for storage. This facilitates the partitioning of similar sample feature vectors into the same shard, improving the rationality and orderliness of data processing. Furthermore, during actual retrieval, only the feature vector to be retrieved needs to be sent to the corresponding sharding server, significantly improving retrieval throughput. Finally, a mapping relationship is established between the centroid feature vector and the corresponding sharding server, and this mapping relationship is used as a preset centroid deployment mapping relationship. This allows for quick and accurate determination of the corresponding sharding server during retrieval based on the preset centroid deployment mapping relationship, improving data retrieval efficiency and throughput.

[0219] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0220] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0221] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0222] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data retrieval method, characterized by, The method comprises: obtaining a to-be-retrieved feature vector; determining a plurality of centroid feature vectors in a preset retrieval data set; respectively calculating the similarity of the plurality of centroid feature vectors and the to-be-retrieved feature vector; when the similarity of any centroid feature vector and the to-be-retrieved feature vector meets a preset condition, determining, based on a preset centroid deployment mapping relationship, a shard server corresponding to the centroid feature vector whose similarity to the to-be-retrieved feature vector meets the preset condition; the determination method of the preset centroid deployment mapping relationship comprises: obtaining an original data set, the original data set comprising a plurality of sample feature vectors; performing clustering processing on the plurality of sample feature vectors to obtain a preset retrieval data set, the preset retrieval data set comprising a plurality of first feature vector sets, and each first feature vector set comprising a corresponding centroid feature vector; performing shard division on the preset retrieval data set to obtain a plurality of second feature vector sets, wherein the second feature vector set comprises at least one centroid feature vector corresponding to the first feature vector set; sending the second feature vector set to a corresponding shard server for storage; establishing a mapping relationship between the centroid feature vector and the corresponding shard server, and taking the mapping relationship as the preset centroid deployment mapping relationship; sending a data retrieval instruction to the corresponding shard server, the data retrieval instruction carrying the to-be-retrieved feature vector, so that the corresponding shard server performs data retrieval based on the to-be-retrieved feature vector to obtain at least one retrieval result feature vector.

2. The method of claim 1, wherein, The shard division of the preset retrieval data set to obtain a plurality of second feature vector sets comprises: determining the dimension of the sample feature vector; calculating the variance of the sample feature vector in the preset retrieval data set in each dimension; performing median hyperplane segmentation processing on the sample feature vectors in each dimension in descending order of variance size to obtain a plurality of second feature vector sets.

3. The method of claim 1, wherein, The shard division of the preset retrieval data set to obtain a plurality of second feature vector sets comprises: performing clustering processing on the preset retrieval data set to obtain a plurality of second feature vector sets.

4. The method of claim 1, wherein, The obtaining of the to-be-retrieved feature vector comprises: receiving target to-be-retrieved data sent by a client; performing feature extraction on the target to-be-retrieved data to obtain at least one target feature vector; taking the at least one target feature vector as the to-be-retrieved feature vector.

5. The method of claim 1, wherein, The obtaining of the original data set comprises: obtaining a plurality of sample data; performing feature extraction on the sample data to obtain feature vectors of the sample data; taking the feature vectors of the plurality of sample data as the plurality of sample feature vectors, and taking the plurality of sample feature vectors as the original data set.

6. The method of claim 1, wherein, After sending a data retrieval instruction to the corresponding shard server, the data retrieval instruction carrying the to-be-retrieved feature vector, so that the corresponding shard server performs data retrieval based on the to-be-retrieved feature vector to obtain at least one retrieval result feature vector, the method further comprises: receive at least one search result data sent by the corresponding shard server, the at least one search result data being determined by the corresponding shard server based on the to-be-searched feature vector for data searching to obtain at least one search result feature vector; send the search result data to the client.

7. A data retrieval apparatus characterized by comprising: The apparatus comprises: an original data set obtaining module configured to obtain an original data set, the original data set comprising a plurality of sample feature vectors; a clustering module configured to perform clustering processing on the plurality of sample feature vectors to obtain a preset search data set, the preset search data set comprising a plurality of first feature vector sets, and each first feature vector set comprising a corresponding centroid feature vector; a data sharding module configured to perform sharding division on the preset search data set to obtain a plurality of second feature vector sets, wherein the second feature vector set comprises a centroid feature vector of at least one first feature vector set; a data sending and storing module configured to send the second feature vector set to a corresponding shard server for storage; a mapping relationship establishing module configured to establish a mapping relationship between the centroid feature vector and the corresponding shard server, and take the mapping relationship as the preset centroid deployment mapping relationship; a to-be-searched feature vector obtaining module configured to obtain a to-be-searched feature vector; a centroid feature vector determining module configured to determine a plurality of centroid feature vectors in the preset search data set; a shard server determining module configured to, when a similarity between any centroid feature vector and the to-be-searched feature vector satisfies a preset condition, determine, based on the preset centroid deployment mapping relationship, a shard server corresponding to the centroid feature vector whose similarity to the to-be-searched feature vector satisfies the preset condition; a search instruction sending module configured to send a data search instruction to the corresponding shard server, the data search instruction carrying the to-be-searched feature vector, so that the corresponding shard server performs data searching based on the to-be-searched feature vector to obtain at least one search result feature vector.

8. The apparatus of claim 7, wherein, The data sharding module comprises: a dimension determining unit configured to determine a dimension of the sample feature vector; a variance calculating unit configured to calculate a variance of the sample feature vector in each dimension in the preset search data set; a median hyperplane segmentation unit configured to perform median hyperplane segmentation processing on the sample feature vector in each dimension in order of variance size from large to small to obtain a plurality of second feature vector sets.

9. The apparatus of claim 7, wherein, The data sharding module comprises: a clustering unit configured to perform clustering processing on the preset search data set to obtain a plurality of second feature vector sets.

10. The apparatus of claim 7, wherein, The original data set obtaining module comprises: a sample data obtaining unit configured to obtain a plurality of sample data; a sample feature extracting unit configured to perform feature extraction on the sample data to obtain a feature vector of the sample data; an original data set determining unit configured to take the feature vector of the plurality of sample data as the plurality of sample feature vectors, and take the plurality of sample feature vectors as the original data set.

11. The apparatus of claim 7, wherein, The apparatus further comprises: The search result data receiving module is configured to receive at least one search result data sent by the corresponding shard server, wherein the at least one search result data is determined by the corresponding shard server based on the to-be-searched feature vector, and is obtained after data searching based on the to-be-searched feature vector to obtain at least one search result feature vector. The search result data forwarding module is configured to send the search result data to the client.

12. A data retrieval apparatus, characterized by comprising: The device comprises a processor and a memory, and the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data searching method according to any one of claims 1 to 6.

13. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the data searching method according to any one of claims 1 to 6.

14. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium; the processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the data searching method according to any one of claims 1 to 6.

Images