Vector data processing method, device, equipment and storage medium

Abstract

The embodiments of the present application disclose a vector data processing method, apparatus, device and storage medium. The method includes: receiving newly added vector data, and placing the newly added vector data in a cache; if the newly added vector data in the cache meets a preset number, extracting training samples from the newly added vector data, and training a classifier based on the training samples to obtain a target classifier; based on the target classifier, classifying the newly added vector data according to the similarity of the newly added vector features; establishing a feature classification index based on the classification result, so as to retrieve vector data based on the feature classification index. The above scheme can train the newly added vector data in a targeted manner, save training time when the vector data continues to grow, reduce training complexity, solve the problem of increased sampling and training time when the data set continues to expand, and improve processing efficiency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a vector data processing method, apparatus, device, and storage medium. Background Art

[0002] With the development and implementation of artificial intelligence technology, more and more previously "indescribable" data are being described and calculated in the form of feature vector data, such as text, sound, appearance, etc. When these massive amounts of feature data are gathered together, how to quickly and accurately find the same or similar data has become an urgent problem to be solved.

[0003] As the dataset continues to expand, if the classifier is not retrained, the classification accuracy of the dataset will decrease; if the classifier is retrained, the time for resampling and training of each computing node will become longer and longer. As the dataset becomes larger, the sampling and data loading time will increase nonlinearly, and the search time will also increase with the growth of the overall data size. Summary of the Invention

[0004] Embodiments of the present invention provide a vector data processing method, apparatus, device, and storage medium for quickly training a classifier and establishing an index when vector data increases.

[0005] In one embodiment, the present application provides a vector data processing method, the method comprising:

[0006] Receive newly added vector data and place the newly added vector data in a cache;

[0007] Receive newly added vector data and place the newly added vector data in a cache;

[0008] If the newly added vector data in the cache meets the preset quantity, a training sample is extracted from the newly added vector data, and a classifier is trained according to the training sample to obtain a target classifier;

[0009] Based on the target classifier, classify the newly added vector data according to the similarity of the newly added vector features;

[0010] A feature classification index is established according to the classification result, so as to perform vector data retrieval according to the feature classification index.

[0011] In another embodiment, the present application further provides a vector data processing device, the device comprising:

[0012] A receiving module, configured to receive newly added vector data and place the newly added vector data in a cache;

[0013] A training sample extraction module is configured to extract training samples from the newly added vector data if the newly added vector data in the cache meets a preset quantity, and train a classifier based on the training samples to obtain a target classifier;

[0014] A classification module, configured to classify the newly added vector data according to the similarity of the newly added vector features based on a target classifier;

[0015] The feature classification index establishment module is used to establish a feature classification index according to the classification result, so as to perform vector data retrieval according to the feature classification index.

[0016] In yet another embodiment, the present application also provides a vector data processing device, including: one or more processors;

[0017] a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the vector data processing method described in any one of the embodiments of the present application.

[0019] In another embodiment, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the vector data processing method as described in any one of the embodiments of the present application.

[0020] In an embodiment of the present application, by caching the received newly added vector data, samples are extracted for training when the received newly added data reaches a preset number, thereby meeting the quantity requirement of model training and improving the accuracy of model training. When the newly added vector data in the cache meets the preset number, training samples are extracted from the newly added vector data, and a classifier is trained based on the training samples to obtain a target classifier, and the target classifier is used to classify the newly added vector data; a feature classification index is established based on the classification result, so as to perform vector data retrieval based on the feature classification index, thereby improving the efficiency of vector data retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a vector data processing method provided by one embodiment of the present invention;

[0022] Figure 2 A flowchart of a specific implementation of a vector data processing method provided by an embodiment of the present invention;

[0023] Figure 3 A flowchart of a vector data processing method provided by another embodiment of the present invention;

[0024] Figure 4A structural diagram of a vector data processing module provided by an embodiment of the present invention;

[0025] Figure 5 A vector data processing signaling diagram provided for an embodiment of the present invention;

[0026] Figure 6 A secondary index structure diagram provided by another embodiment of the present invention;

[0027] Figure 7 A schematic structural diagram of a vector data processing device provided by an embodiment of the present invention;

[0028] Figure 8 A schematic structural diagram of a vector data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0030] Figure 1 This is a flowchart of a vector data processing method provided by an embodiment of the present invention. The vector data processing method provided by this embodiment is applicable to situations where vector data is processed. Typically, this method is applicable to situations where vector data is accessed, indexed, and retrieved. The method can be specifically executed by a vector data processing device, which can be implemented by software and / or hardware, and can be integrated into a vector data processing device. See Figure 1 , the method of the embodiment of the present application specifically includes:

[0031] S110: Receive newly added vector data, and place the newly added vector data in a cache.

[0032] Among them, the newly added vector data is the vector data in the newly connected computing node. It can be vector data generated and connected in real time, or it can be historical vector data that already exists in the system or database. Specifically, the data access layer receives the newly added vector data and sends it to each computing node based on the load balancing principle according to the number of computing nodes. After receiving the newly added vector, the computing node first caches it. The beneficial effect is that the number of newly added vector data is small when it is first connected, and it has not yet formed a certain scale, which cannot meet the requirements of high-accuracy model training. Therefore, the connected data is cached first, and when the cached amount reaches a certain amount, a certain amount of newly added vector data is used for model training.

[0033] S120: If the newly added vector data in the cache meets the preset quantity, extracting training samples from the newly added vector data, and training a classifier based on the training samples to obtain a target classifier;

[0034] The preset number can be set based on actual conditions. For example, it can be the number of training samples required to train an accurate model. When a computing node detects that the newly added vector data it has received has reached the preset number, it sends information about the number of newly added vector data to the master node. The master node calculates the average number of newly added vector data across all computing nodes based on the information sent by each computing node. If the average number reaches the preset average number, it extracts training samples from the newly added vectors to train the classifier and obtain the target classifier.

[0035] The beneficial effect of the above scheme is that when the cached new vectors meet the preset number, training samples are extracted for classifier training, so that there are enough training samples for classifier training to meet the sample quantity requirements for high-accuracy classifier training, thereby training a target classifier with high accuracy. In addition, the number of new vectors received by a single computing node may be accidental and cannot reflect the overall level of the number of new vectors. If the number of new vectors of a single computing node is used to determine whether to extract training samples for training, there may be a problem that the overall number of new vectors is still not high and a sufficient number of training samples cannot be extracted. In an embodiment of the present application, the master node calculates the average number of new vector data currently added by all computing nodes based on the quantity information sent by each computing node. If the average number reaches the preset average number, training samples are extracted from the new vectors to train the classifier to obtain the target classifier, thereby comprehensively considering the overall level of the number of new vectors and ensuring that the overall number of new vectors meets the requirements of high-precision classifier training.

[0036] S130 : Based on the target classifier, classify the newly added vector data according to the similarity of the newly added vector features.

[0037] Specifically, the target classifier classifies the newly added vectors based on their vector features, grouping new vectors with similar features into one category to form vector data clusters. Furthermore, for each vector data cluster, a representative vector is selected to associate with the vector data cluster and establish a feature classification index.

[0038] S140: Establish a feature classification index according to the classification result, and perform vector data retrieval according to the feature classification index.

[0039] Specifically, to facilitate retrieval of the vector data to be retrieved in the vector search request, a feature classification index is established for the newly added vector data. For example, a feature classification index is established based on the classified vector data clusters, and then vector data clusters related to the vector data to be retrieved are determined based on the feature classification indexes, and the target vector is retrieved from the related vector data clusters.

[0040] In an embodiment of the present application, if the newly added vector data is real-time vector data, then after receiving the newly added vector data and placing the newly added vector data in the cache, the method also includes: if the target classifier training is not completed, using a historical classifier to classify the real-time vector data according to the vector features to obtain a temporary classification result; and establishing a temporary feature classification index based on the temporary classification result.

[0041] Specifically, such as Figure 2 As shown, when the real-time vector data starts to be accessed, if the amount of real-time vector data has not reached the preset amount, the training of the classifier has not started, or the classifier has not been completed during the training process, a feature classification index has not been established for the real-time vector data. If a vector retrieval request is received at this time, the vector data to be retrieved can only be searched by traversing the real-time vector data, which affects the retrieval efficiency. In an embodiment of the present application, if the training of the target classifier is not completed and the feature classification index is not established, the real-time vector data is classified by a historical classifier to obtain a temporary classification result, and a temporary feature classification index is established based on the temporary classification result to meet the current vector retrieval service requirements. If the training of the target classifier is completed, the real-time vector data is classified by the target classifier to obtain a classification result, and a feature classification index is established based on the classification result. At this time, the feature classification index can replace the temporary feature classification index, and the historical classifier is deleted, and the current target classifier is used as the historical classifier for the next batch of newly added vectors.

[0042] The beneficial effect of the above scheme is that before the target classifier is trained, a temporary feature classification index can be established for the newly added vector data to meet the current retrieval needs. Due to the continuity of vector data, the historical classifier can be applied to the temporary classification of the newly added vector data.

[0043] In an embodiment of the present application, if the newly added vector data is historical vector data, then after receiving the newly added vector data and placing the newly added vector data in a cache, the method further includes: if a historical classifier exists, using the historical classifier to classify the historical vector data according to vector features; establishing a feature classification index based on the classification result, so as to retrieve vector data according to the feature classification index. If no historical classifier exists, triggering the execution of the step of extracting training samples and sending them to the training node if the newly added vector data in the cache meets a preset number, so that the training node trains the target classifier.

[0044] Specifically, such as Figure 2 As shown, if the newly added vector data is historical vector data, the classifier training may have been completed before based on the historical vector data, and the trained historical classifier is suitable for classifying the historical vector data. Therefore, if the newly added vector data received is historical vector data, and a historical classifier currently exists, the historical classifier is used to classify the historical vector data, and a feature classification index for the historical vector data is established based on the classification result. If there is no historical classifier, the execution is triggered to extract training samples and send them to the training node if the newly added vector data in the cache meets the preset number, so that the training node trains the target classifier, uses the trained target classifier to classify the historical vector data, and establishes a feature classification index for the historical vector data based on the classification result.

[0045] The beneficial effect of the above scheme is that, when a historical classifier exists, it can be used to accurately and quickly classify historical vector data and establish a feature classification index, thereby improving the efficiency of index establishment. In the absence of a historical classifier, the classifier is trained using historical vector data to obtain a target classifier and classify the historical vector data, thereby establishing a feature classification index to facilitate vector data indexing.

[0046] In an embodiment of the present application, by caching the received newly added vector data, samples are extracted for training when the received newly added data reaches a preset number, thereby meeting the quantity requirement of model training and improving the accuracy of model training. When the newly added vector data in the cache meets the preset number, training samples are extracted from the newly added vector data, and a classifier is trained based on the training samples to obtain a target classifier, and the target classifier is used to classify the newly added vector data; a feature classification index is established based on the classification result, so as to perform vector data retrieval based on the feature classification index, thereby improving the efficiency of vector data retrieval.

[0047] Figure 3This is a flow chart of a vector data processing method provided by another embodiment of the present invention. This embodiment of the present application optimizes the above embodiment based on the above embodiment. For details not described in detail in this embodiment, please refer to the above embodiment. Figure 3 , the vector data processing method provided in this embodiment may include:

[0048] S210: Receive newly added vector data, and place the newly added vector data in a cache.

[0049] S220: If the newly added vector data in the cache meets the preset quantity, extract training samples from the newly added vector data, and train a classifier according to the training samples to obtain a target classifier.

[0050] In an embodiment of the present application, the classification and retrieval process of the newly added vector data is performed by the computing node, and the training process of the classifier is performed by the training node.

[0051] like Figure 4 As shown in the figure, the compute nodes extract training samples from a preset number of newly added vector data and send them to the training nodes. The training nodes use the training samples to train the classifier and obtain the target classifier. Among them, the Master node is the main node, responsible for managing other nodes and providing external data services.

[0052] In the embodiments of this application, Figure 5 As shown, when the computing node detects that the newly added vector data it has accessed reaches a preset number, it sends the quantity information of the newly added vector data to the master node. The master node calculates the average number of newly added vector data of all computing nodes based on the quantity information sent by each computing node. If the average number reaches the preset average number, training samples are extracted from the newly added vectors and sent to the training node, which trains the classifier to obtain the target classifier.

[0053] In this embodiment of the present application, vector data access is deployed on the compute node, and training is deployed on the training node. This advantageously allows the vector data access and training processes to be performed simultaneously by the two nodes without interfering with each other. Furthermore, during classifier training, the compute node can also process vector search requests and provide search services without being affected by classifier training.

[0054] S230. Obtain target classifier parameters from the memory; wherein the target classifier parameters are stored in the memory after the training node completes the classifier training.

[0055] Specifically, after the target classifier is trained, the training node will persistently store the target classifier in the storage space. The stored target classifier parameters may include: target classifier identifier, classification identifier, vector data dimension, vector data model, number of classification categories, number of approximate vector data clusters to search, whether the vector data is encrypted, representative vectors in each vector data cluster, etc. Before establishing a feature classification index for the newly added vector data, the computing node obtains the target classifier parameters from the storage space and classifies the newly added vector data according to the target classifier parameters, thereby establishing the feature classification index.

[0056] S240: Construct a target classifier according to the target classifier parameters.

[0057] Specifically, after obtaining the target classifier parameters, the computing node constructs a target classifier according to the target classifier parameters to classify the newly added vector data according to the target classifier parameters, and then establishes a feature classification index.

[0058] S250 : Based on the target classifier, classify the newly added vector data according to the similarity of the newly added vector features.

[0059] S260. Select representative vector data from the classified vector data cluster.

[0060] For example, new vector data with more obvious features can be selected from the vector data cluster as representative vector data, or new vector data in the middle position after arranging the new vector data according to the features can be selected as representative vector data. The selection can be based on actual conditions and is not limited here.

[0061] S270: Establish a mapping relationship between the representative vector data and the vector data cluster to form a feature classification index, so as to perform vector data retrieval according to the feature classification index.

[0062] Exemplarily, a mapping relationship is established between the representative vector selected from each vector data cluster and the vector data cluster, forming a feature classification index. When the master node receives a vector search request, it distributes the request to each computing node. Based on the similarity between the vector to be searched and each representative vector, the computing node identifies a preset number of similar representative vector data. Based on the feature classification index, the computing node determines the vector data clusters associated with the similar representative vector data. The final search result is obtained by traversing each associated vector data cluster, and the final search result is returned to the vector search requester via the master node.

[0063] In an embodiment of the present application, the method further includes: establishing an attribute classification index of the newly added vector data based on the attributes of the newly added vector data; wherein the attributes include the reception time and / or storage space of the newly added vector data; accordingly, after establishing a feature classification index based on the classification results, the method further includes: using the attribute classification index as a primary index, adding the feature classification index to the attribute classification index as a secondary index, and constructing a target classification index to perform vector data retrieval based on the target classification index.

[0064] For example, Figure 6 As shown, a primary index is established based on the access time of the newly added vector data. The feature classification index established for the newly added vector data in each time period is added to the primary index to form a secondary index. If a vector search request is received, the time period in which the search is connected can be determined based on the access time attribute of the vector to be searched, and the target feature classification index and vector data cluster located in the time period can be determined. Then, the vector to be searched can be searched specifically based on the target feature classification index. Secondary indexing through spatiotemporal classification can improve the efficiency of vector search and accelerate the vector search process.

[0065] In an embodiment of the present application, after establishing a feature classification index according to the classification result, the method further includes: storing the feature classification index and the newly added data in a memory, and clearing the cache.

[0066] The above solution can clear out available cache space in time through cache cleaning to provide access to the newly added vector, thereby ensuring normal access of the newly added vector.

[0067] The technical solution of the embodiment of the present application obtains persistently stored target classifier parameters and constructs a target classifier to classify newly added vector data, thereby enabling simultaneous access to newly added vector data and classifier training. The process of obtaining the target classifier does not affect access to newly added vector data, thereby improving vector processing efficiency. By establishing a mapping relationship between representative vector data and vector data clusters, a feature classification index is established, facilitating the rapid and efficient processing of subsequent vector retrieval requests based on the feature classification index.

[0068] Figure 7 This is a schematic diagram of the structure of a vector data processing device provided by an embodiment of the present invention. The device is applicable to processing vector data. Typically, this method is applicable to accessing, indexing, and retrieving vector data. The device can be implemented in software and / or hardware, and can be integrated into a vector data processing device. Figure 7 , the device specifically includes:

[0069] A receiving module 310 is configured to receive newly added vector data and place the newly added vector data in a cache;

[0070] A training sample extraction module 320 is configured to extract training samples from the newly added vector data if the newly added vector data in the cache meets a preset quantity, and train a classifier based on the training samples to obtain a target classifier;

[0071] A classification module 330 is configured to classify the newly added vector data based on similarities of the newly added vector features based on a target classifier;

[0072] The feature classification index establishing module 340 is used to establish a feature classification index according to the classification result, so as to perform vector data retrieval according to the feature classification index.

[0073] In an embodiment of the present application, the device further includes:

[0074] A temporary classification module is used to classify real-time vector data based on vector features using a historical classifier to obtain a temporary classification result if the target classifier training is not completed;

[0075] The temporary index establishment module is used to establish a temporary feature classification index based on the temporary classification results.

[0076] In an embodiment of the present application, the device further includes:

[0077] A historical classification module, configured to establish a feature classification index based on the classification results, so as to perform vector data retrieval based on the feature classification index;

[0078] The training trigger module is used to trigger the execution of the step of extracting training samples and sending them to the training node if the new vector data in the cache meets the preset quantity if there is no historical classifier, so that the training node trains the target classifier.

[0079] In an embodiment of the present application, the device further includes:

[0080] A parameter acquisition module is used to obtain target classifier parameters from the memory; wherein the target classifier parameters are stored in the memory after the training node completes the classifier training;

[0081] A construction module is used to construct a target classifier according to the target classifier parameters.

[0082] In the embodiment of the present application, the feature classification index establishment module 340 includes:

[0083] A representative vector data selection unit, used for selecting representative vector data from the classified vector data cluster;

[0084] The mapping relationship establishing unit is used to establish a mapping relationship between the representative vector data and the vector data cluster to form a feature classification index.

[0085] In an embodiment of the present application, the device further includes:

[0086] An attribute classification module, configured to establish an attribute classification index for the newly added vector data based on the attributes of the newly added vector data; wherein the attributes include a receiving time and / or a storage space for the newly added vector data;

[0087] An adding module is used to establish a feature classification index according to the classification results, use the attribute classification index as a primary index, add the feature classification index to the attribute classification index as a secondary index, and construct a target classification index to perform vector data retrieval based on the target classification index.

[0088] In an embodiment of the present application, the device further includes:

[0089] The feature classification index and the newly added data are stored in the memory, and the cache is cleared.

[0090] In an embodiment of the present application, the classification and retrieval process of the newly added vector data is performed by the computing node, and the training process of the classifier is performed by the training node.

[0091] The vector data processing device provided in the embodiment of the present application can execute the vector data processing method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method.

[0092] Figure 8 A schematic structural diagram of a vector data processing device provided by an embodiment of the present invention. Figure 8 FIG. 4 is a block diagram of an exemplary vector data processing device 412 suitable for implementing embodiments of the present application. Figure 8 The vector data processing device 412 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present application.

[0093] like Figure 8 As shown, the vector data processing device 412 may include: one or more processors 416; a memory 428 for storing one or more programs. When the one or more programs are executed by the one or more processors 416, the one or more processors 416 implement the vector data processing method provided in the embodiment of the present application, including:

[0094] Receive newly added vector data and place the newly added vector data in a cache;

[0095] If the newly added vector data in the cache meets the preset quantity, a training sample is extracted from the newly added vector data, and a classifier is trained according to the training sample to obtain a target classifier;

[0096] Based on the target classifier, classify the newly added vector data according to the similarity of the newly added vector features;

[0097] A feature classification index is established according to the classification result, so as to perform vector data retrieval according to the feature classification index.

[0098] Components of the vector data processing device 412 may include, but are not limited to, one or more processors or a processor 416 , a memory 428 , and a bus 418 connecting various device components (including the memory 428 and the processor 416 ).

[0099] Bus 418 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0100] The vector data processing device 412 typically includes a variety of computer-readable storage media, which can be any available storage media that can be accessed by the vector data processing device 412, including volatile and non-volatile storage media, and removable and non-removable storage media.

[0101] The memory 428 may include computer-readable storage media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. The vector data processing device 412 may further include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, the storage system 434 may be configured to read and write non-removable, non-volatile magnetic storage media ( Figure 8 Not shown, often called a "hard drive"). Although Figure 8Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical storage medium) may be provided. In these cases, each drive may be connected to bus 418 via one or more data storage medium interfaces. Memory 428 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0102] A program / utility 440 having a set (at least one) of program modules 442 may be stored, for example, in memory 428. Such program modules 442 include, but are not limited to, operating devices, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 442 generally implement the functions and / or methods of the embodiments described herein.

[0103] The vector data processing device 412 may also communicate with one or more external devices 414 (e.g., a keyboard, a pointing device, a display 426, etc.), and may also communicate with one or more devices that enable a user to interact with the vector data processing device 412, and / or any device that enables the vector data processing device 412 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed through an input / output (I / O) interface 422. Furthermore, the vector data processing device 412 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 420. Figure 8 As shown, the network adapter 420 communicates with other modules of the vector data processing device 412 via the bus 418. Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with the vector data processing device 412, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID devices, tape drives, and data backup storage devices.

[0104] The processor 416 executes various functional applications and data processing by running at least one of the other programs among the multiple programs stored in the memory 428, such as implementing a vector data processing method provided in an embodiment of the present application.

[0105] An embodiment of the present invention provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform a vector data processing method, including:

[0106] Receive newly added vector data and place the newly added vector data in a cache;

[0107] If the newly added vector data in the cache meets the preset quantity, a training sample is extracted from the newly added vector data, and a classifier is trained according to the training sample to obtain a target classifier;

[0108] Based on the target classifier, classify the newly added vector data according to the similarity of the newly added vector features;

[0109] A feature classification index is established according to the classification result, so as to perform vector data retrieval according to the feature classification index.

[0110] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable storage media. The computer-readable storage medium can be a computer-readable signal storage medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In an embodiment of the present application, a computer-readable storage medium can be any tangible storage medium containing or storing a program, which can be used by an instruction execution device, device or device or used in combination with it.

[0111] A computer-readable signal storage medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal storage medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution device, apparatus, or component.

[0112] The program code embodied on the computer-readable storage medium may be transmitted using any appropriate storage medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0113] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or device. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0114] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A vector data processing method, characterized in that: The method comprises: Receive newly added vector data and place the newly added vector data in a cache; If the newly added vector data in the cache meets the preset quantity, a training sample is extracted from the newly added vector data, and a classifier is trained according to the training sample to obtain a target classifier; Based on the target classifier, classify the newly added vector data according to the similarity of the newly added vector features; Establishing a feature classification index based on the classification result, so as to perform vector data retrieval based on the feature classification index; If the newly added vector data in the cache meets the preset quantity, a training sample is extracted from the newly added vector data, and a classifier is trained according to the training sample to obtain a target classifier, including: When a computing node detects that the number of newly added vector data it has accessed reaches a preset number, it sends the number of newly added vector data to the master node. The master node calculates the average number of newly added vector data of all computing nodes based on the number information sent by each computing node. If the average number reaches the preset average number, it extracts training samples from the newly added vectors to train the classifier and obtain the target classifier. If the newly added vector data is real-time vector data, after receiving the newly added vector data and placing the newly added vector data in a cache, the method further includes: If the target classifier training is not completed, the historical classifier is used to classify the real-time vector data according to the vector features to obtain a temporary classification result; A temporary feature classification index is established based on the temporary classification results.

2. The method according to claim 1, characterized in that If the newly added vector data is historical vector data, after receiving the newly added vector data and placing the newly added vector data in a cache, the method further includes: If a historical classifier exists, the historical classifier is used to classify the historical vector data according to the vector features; Establishing a feature classification index based on the classification result, so as to perform vector data retrieval based on the feature classification index; If there is no historical classifier, the step of extracting training samples and sending them to the training node is triggered to execute if the newly added vector data in the cache meets the preset quantity, so that the training node trains the target classifier.

3. The method according to claim 1, characterized in that Before classifying the newly added vector data using the target classifier, the method further includes: Obtaining target classifier parameters from the memory; wherein the target classifier parameters are stored in the memory after the training node completes the classifier training; A target classifier is constructed according to the target classifier parameters.

4. The method according to claim 1, wherein Establish a feature classification index based on the classification results, including: Select representative vector data from the classified vector data cluster; A mapping relationship between the representative vector data and the vector data cluster is established to form a feature classification index.

5. The method according to claim 1, wherein The method further comprises: Establishing an attribute classification index for the newly added vector data according to the attributes of the newly added vector data, wherein the attributes include a receiving time and / or a storage space for the newly added vector data; Accordingly, after establishing the feature classification index according to the classification result, the method further includes: The attribute classification index is used as a primary index, the feature classification index is added to the attribute classification index as a secondary index, and a target classification index is constructed to perform vector data retrieval based on the target classification index.

6. The method according to claim 1, characterized in that After establishing the feature classification index according to the classification results, the method further includes: The feature classification index and the newly added vector data are stored in a memory, and the cache is cleared.

7. The method according to any one of claims 1 to 6, characterized in that The classification and retrieval process of the newly added vector data is performed by the computing node, and the training process of the classifier is performed by the training node.

8. A vector data processing device, characterized in that: The device comprises: A receiving module, configured to receive newly added vector data and place the newly added vector data in a cache; A training sample extraction module is configured to extract training samples from the newly added vector data if the newly added vector data in the cache meets a preset quantity, and train a classifier based on the training samples to obtain a target classifier; A classification module, configured to classify the newly added vector data according to the similarity of the newly added vector features based on a target classifier; A feature classification index establishment module is used to establish a feature classification index according to the classification result, so as to perform vector data retrieval according to the feature classification index; The training sample extraction module extracts training samples from the newly added vector data if the newly added vector data in the cache meets the preset quantity, and trains the classifier according to the training samples to obtain the target classifier, including: When a computing node detects that the number of newly added vector data it has accessed reaches a preset number, it sends the number of newly added vector data to the master node. The master node calculates the average number of newly added vector data of all computing nodes based on the number information sent by each computing node. If the average number reaches the preset average number, it extracts training samples from the newly added vectors to train the classifier and obtain the target classifier. The device further comprises: A temporary classification module is used to classify real-time vector data based on vector features using a historical classifier to obtain a temporary classification result if the target classifier training is not completed; The temporary index establishment module is used to establish a temporary feature classification index based on the temporary classification results.

9. A vector data processing device, characterized in that: The vector data processing device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the vector data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the vector data processing method according to any one of claims 1 to 7 is implemented.

Images