Data Retrieval Method, Platform, Server, System, Device and Medium

Abstract

The present application discloses a data retrieval method, platform, server, system, device and medium. The method includes: creating operation objects and creating decorators corresponding to each operation object; wherein the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object; sending each decorator to a model training device so that the model training device trains a data retrieval model based on each decorator; and compiling and linking each operation object to a data retrieval service server so that the data retrieval service server loads the data retrieval model to perform data retrieval operations. The present application can support both model training in an offline state and data retrieval in an online state, without the need to separately develop an online retrieval service, and multiple model logics can be integrated in the offline training part.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a data retrieval method, platform, server, system, device and medium. Background Art

[0002] Currently, more and more recommendation services use HNSW (Hierarchical Navigable Small World, approximate nearest neighbor algorithm) for data recall. Its implementation process mainly includes two parts. One part is to create an index offline, and the other part is to perform approximate nearest neighbor search online. However, in the existing methods that support HNSW, the offline part uses the HNSW library to create and save the index, but it cannot integrate other model logics, resulting in the problem of a single supported logic model; there is currently no complete and available solution for the online part, and the business needs to implement it by itself. Summary of the Invention

[0003] In view of this, to solve the above technical problems, this application provides a data retrieval method, platform, server, system, device and medium, which can support model training in the offline state and data retrieval in the online state at the same time, without the need to develop the online retrieval service separately, and the offline training part can integrate multiple model logics.

[0004] In a first aspect, this application provides a data retrieval method, which is applied to an open-source machine learning platform and includes:

[0005] Create operation objects, and create decorators corresponding to each of the operation objects; wherein, the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object;

[0006] Send each of the decorators to a model training device, so that the model training device trains a data retrieval model based on each of the decorators;

[0007] Compile and link each of the operation objects to a data retrieval service server, so that the data retrieval service server loads the data retrieval model to perform data retrieval operations.

[0008] Optionally, the creating the operation objects includes:

[0009] Obtain registration information corresponding to each of the operation objects;

[0010] Create a kernel corresponding to each of the operation objects;

[0011] Bind the registration information and the kernel corresponding to the same operation object to generate the operation object.

[0012] Optionally, creating decorators corresponding to each of the operation objects includes:

[0013] For each of the operation objects, load the dynamic library of the operation object corresponding to the operation object;

[0014] Bind the operation object to the dynamic library of the operation object to generate a decorator corresponding to the operation object.

[0015] Optionally, compiling and linking each of the operation objects to the data retrieval service server includes:

[0016] For each of the operation objects, compile the operation object to generate a static library of the corresponding operation object;

[0017] Link the static libraries of each of the operation objects to the data retrieval service server.

[0018] In a second aspect, the present application provides a data retrieval method applied to a model training device, including:

[0019] Receive decorators corresponding to each operation object sent by an open-source machine learning platform; wherein, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object;

[0020] Train a data retrieval model using each of the decorators so that the data retrieval service server loads the data retrieval model to perform data retrieval operations.

[0021] Optionally, training the data retrieval model using each of the decorators includes:

[0022] Obtain a first feature sample of a retrieval element and a second feature sample of a recalled element;

[0023] Use each of the decorators to perform model training on the first feature sample and the second feature sample to obtain the data retrieval model.

[0024] In a third aspect, the present application provides a data retrieval method applied to a data retrieval service server, including:

[0025] When compiling and linking to each operation object created by an open-source machine learning platform, load and obtain a data retrieval model from a model training device; wherein, the data retrieval model is trained by the model training device based on the decorators corresponding to each of the operation objects, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object;

[0026] Execute a data retrieval operation using the data retrieval model.

[0027] In a fourth aspect, the present application provides a data retrieval method, which is applied to a data retrieval system. The data retrieval system includes an open-source machine learning platform, a model training device, and a data retrieval service server. The method includes:

[0028] Create an operation object and a decorator corresponding to each operation object through the open-source machine learning platform, send each decorator to the model training device, and compile and link each operation object to the data retrieval service server; wherein, the operation object includes an index generation operation object, an index saving operation object, and a data retrieval operation object.

[0029] Train a data retrieval model based on each decorator through the model training device.

[0030] Load the data retrieval model through the data retrieval service server to execute a data retrieval operation.

[0031] In a fifth aspect, the present application provides an open-source machine learning platform, including:

[0032] A creation module, configured to create an operation object and a decorator corresponding to each operation object; wherein, the operation object includes an index generation operation object, an index saving operation object, and a data retrieval operation object.

[0033] A sending module, configured to send each decorator to a model training device, so that the model training device trains a data retrieval model based on each decorator.

[0034] A compilation module, configured to compile and link each operation object to a data retrieval service server, so that the data retrieval service server loads the data retrieval model to execute a data retrieval operation.

[0035] In a sixth aspect, the present application provides a model training device, including:

[0036] A receiving module, configured to receive a decorator corresponding to each operation object sent by an open-source machine learning platform; wherein, each operation object and the decorator corresponding to each operation object are created by the open-source machine learning platform, and the operation object includes an index generation operation object, an index saving operation object, and a data retrieval operation object.

[0037] A training module, configured to train a data retrieval model using each decorator, so that a data retrieval service server loads the data retrieval model to execute a data retrieval operation.

[0038] In a seventh aspect, the present application provides a data retrieval service server, including:

[0039] A loading module, configured to load and obtain a data retrieval model from a model training device when compiling and linking to each operation object created by an open-source machine learning platform; wherein, the data retrieval model is obtained by the model training device based on decorators corresponding to each of the operation objects, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object;

[0040] A retrieval module, configured to perform a data retrieval operation by using the data retrieval model.

[0041] In an eighth aspect, the present application provides a data retrieval system, including:

[0042] An open-source machine learning platform, configured to create operation objects and decorators corresponding to each of the operation objects, send each of the decorators to a model training device, and compile and link each of the operation objects to a data retrieval service server; wherein, the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object;

[0043] The model training device, configured to train and obtain a data retrieval model based on each of the decorators;

[0044] The data retrieval service server, configured to load the data retrieval model and perform a data retrieval operation.

[0045] In a ninth aspect, the present application provides an electronic device, including:

[0046] A memory, configured to store a computer program;

[0047] A processor, configured to implement the steps of any of the data retrieval methods as described above when executing the computer program.

[0048] In a tenth aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any of the data retrieval methods as described above are implemented.

[0049] A data retrieval solution provided by this application includes creating operation objects and creating decorators corresponding to each operation object. Among them, the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object. Send each decorator to a model training device so that the model training device can train a data retrieval model based on each decorator. Compile and link each operation object to a data retrieval service server so that the data retrieval service server can load the data retrieval model to perform data retrieval operations. Applying the technical solution provided by this application, by creating operation objects to support operations such as index generation, index saving, and data retrieval related to HNSW, and creating decorators corresponding to the operations, and compiling and linking these operation objects to the data retrieval service server and sending the decorators to the model training device, the offline training model part can use these operation objects through the model training device to integrate other model logics and complete the training of multiple data retrieval models. At the same time, it supports the data retrieval service server to online load the data retrieval model for data retrieval, that is, online data retrieval can be realized without additional business development. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present invention, and are used together with the specification to explain the principles of the present invention.

[0051] To more clearly illustrate the technical solutions in the prior art and the embodiments of this application, the accompanying drawings required for describing the prior art and the embodiments of this application will be briefly introduced below. Of course, the following description of the accompanying drawings of the embodiments of this application only shows a part of the embodiments in this application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings, and the other drawings obtained also fall within the protection scope of this application.

[0052] Figure 1 It is a schematic structural diagram of a data retrieval system provided by an embodiment of this application;

[0053] Figure 2 It is a schematic flowchart of a data retrieval method provided by an embodiment of this application;

[0054] Figure 3 It is a schematic flowchart of another data retrieval method provided by an embodiment of this application;

[0055] Figure 4 It is a schematic flowchart of yet another data retrieval method provided by an embodiment of this application;

[0056] Figure 5 It is a schematic flowchart of still another data retrieval method provided by an embodiment of this application;

[0057] Figure 6 Schematic diagram of a data retrieval method provided by an embodiment of the present application;

[0058] Figure 7 Schematic diagram of the structure of an open-source machine learning platform provided by an embodiment of the present application;

[0059] Figure 8 Schematic diagram of the structure of a model training device provided by an embodiment of the present application;

[0060] Figure 9 Schematic diagram of the structure of a data retrieval service server provided by an embodiment of the present application;

[0061] Figure 10 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0063] An embodiment of the present application provides a data retrieval system. Refer to Figure 1 , the data retrieval system includes an open-source machine learning platform 100, a model training device 200, and a data retrieval service server 300. The data retrieval system is used to implement model training in an offline state and data retrieval in an online state. The open-source machine learning platform 100 may be TensorFlow (an end-to-end open-source machine learning platform). The open-source machine learning platform 100 is used to create operation objects and decorators corresponding to each operation object, send each decorator to the model training device, and compile and link each operation object to the data retrieval service server; among them, the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object; the model training device 200 is a device that can run the Python language for model training. The model training device 200 is used to train a data retrieval model based on each decorator; the data retrieval service server 300 may be TensorFlow Serving (a server for deploying machine learning models). The data retrieval service server 300 is used to load the data retrieval model to perform data retrieval operations.

[0064] An embodiment of the present application provides a data retrieval method.

[0065] Please refer toFigure 2 , Figure 2 It is a schematic flowchart of a data retrieval method provided by an embodiment of the present application. The data retrieval method is applied to an open-source machine learning platform in a data retrieval system, and the data retrieval method may include the following steps S101 to S103.

[0066] S101: Create operation objects and create decorators corresponding to each operation object; wherein, the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object.

[0067] This step aims to create operation objects and decorators corresponding to the operation objects. The operation object is OP (Operation). In the implementation process, first, TensorFlow OP can be created in the open-source machine learning platform TensorFlow. The TensorFlow OP is used to support operations related to HNSW (Hierarchical Navigable Small World, approximate nearest neighbor search for recall), such as creating indexes, saving indexes, and retrieving data (such as approximate nearest neighbor search), to obtain corresponding OPs, including an index generation OP, an index saving OP, and a data retrieval OP. Further, after creating each operation object, in order to implement model training in an offline state, decorators (Wrappers) corresponding to each operation object are created based on the open-source machine learning platform. Each Wrapper is used to implement model training in an offline state, and the decorators corresponding to each operation are sent to the model training device so that the model training device can perform training according to the Wrapper to obtain a corresponding model. Among them, in the specific implementation process of creating the decorator corresponding to the operation, first, for each OP, its corresponding Wrapper is created, and then each created Wrapper is sent to the model training device for model training. It can be understood that the Wrapper is essentially a Python function, and its role is to add additional functions to an existing function or object. It can be imagined that corresponding to the above index generation OP, index saving OP, and data retrieval OP, the Wrapper can include an index generation Wrapper, an index saving Wrapper, and a data retrieval Wrapper.

[0068] In a possible implementation manner, the above creation of operation objects may include the following steps:

[0069] Obtain the registration information corresponding to each operation object;

[0070] Create kernels corresponding to each operation object;

[0071] Bind the registration information kernels corresponding to the same operation to generate operation objects.

[0072] The embodiment of the present application provides an implementation method for creating an operation object. When creating an operation object in the open-source machine learning platform TensorFlow, first register each type of operation according to the TensorFlow specification, and create a kernel (Kernel) for each type of operation. Thus, the creation of the operation object is realized by binding the registration information and the Kernel belonging to the same operation. In the specific implementation process, for the generate index OP, first obtain the registration information of the generate index operation, implement the Kernel of the generate index operation, and then bind the registration information and the Kernel of the generate index operation to obtain the generate index OP; for the save index OP, first obtain the registration information of the save index operation, implement the Kernel of the save index operation, and then bind the registration information and the Kernel of the save index operation to obtain the save index OP; for the retrieve data OP, first obtain the registration information of the retrieve data operation, implement the Kernel of the retrieve data operation, and then bind the registration information and the Kernel of the retrieve data operation to obtain the retrieve data OP.

[0073] In a possible implementation manner, the above-mentioned creation of the decorator corresponding to each operation object may include the following steps:

[0074] For each operation object, load the operation object dynamic library corresponding to the operation object;

[0075] Bind the operation object to the operation object dynamic library to generate the decorator corresponding to the operation object.

[0076] The embodiment of the present application provides an implementation method for creating a decorator for an operation object. In the implementation process, first, for each OP, load the corresponding OP dynamic library of the OP to obtain the OP dynamic library corresponding to each OP, and then bind the OP and the OP dynamic library belonging to the same operation to obtain the corresponding Wrapper. In other words, bind the generate index OP to the OP dynamic library of the generate index operation to obtain the generate index Wrapper; bind the save index OP to the OP dynamic library of the save index operation to obtain the save index Wrapper; bind the retrieve data OP to the OP dynamic library of the retrieve data operation to obtain the retrieve data Wrapper.

[0077] S102: Send each decorator to the model training device so that the model training device trains a data retrieval model based on each decorator.

[0078] This step aims to implement the training of the data retrieval model. In this embodiment, the data retrieval model is used to implement data retrieval. For different types of retrieval elements, their corresponding data retrieval models are also different. Therefore, during the model training process, Wrapper can be used to train the model for the feature samples of different element pairs, so as to obtain the data retrieval models corresponding to different retrieval elements. The implementation process of the model training device using Wrapper to obtain the data retrieval model is described below, and this embodiment will not be elaborated here.

[0079] S103: Compile and link each operation object to the data retrieval service server so that the data retrieval service server loads the data retrieval model to perform data retrieval operations.

[0080] This step aims to implement data retrieval. When the created operation object is obtained according to step S101 and the trained data retrieval model is obtained according to step S102, the open-source machine learning platform compiles and links the operation object to the data retrieval service server, sends the data retrieval model to the data retrieval service server, and the data retrieval service can load the data retrieval model and perform data retrieval operations when receiving a data retrieval request.

[0081] In a possible implementation manner, the above-mentioned compiling and linking each operation object to the data retrieval service server may include the following steps:

[0082] For each operation object, compile the operation object to generate a corresponding operation object static library;

[0083] Link each operation object static library to the data retrieval service server.

[0084] The embodiment of the present application provides a method for implementing the compilation and linking of OP to the data retrieval service server. When the creation of OP is completed in the TensorFlow platform, each OP can be first compiled into a corresponding hnsw_ops static library, that is, the above-mentioned OP static library, and then each OP static library is linked to the data retrieval service server. Based on this, by adding TensorFlow OP dependencies in the Bazel target of Tensorflow_model_server in the data retrieval service server (TensorFlow Serving), the linking of each OP static library to the data retrieval service server can be realized.

[0085] It can be seen that a data retrieval method provided by this embodiment supports operations such as generating an index, saving an index, and retrieving data related to HNSW by creating operation objects, and creating corresponding decorators for the operations. These operation objects are compiled and linked to the data retrieval service server, and the decorators are sent to the model training device, enabling the offline training model part to use these operation objects through the model training device to integrate other model logics and complete the training of multiple data retrieval models. At the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional business development.

[0086] Another data retrieval method is provided in an embodiment of this application.

[0087] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of another data retrieval method provided in an embodiment of this application. This data retrieval method is applied to a model training device in a data retrieval system and may include the following steps S201 to S202.

[0088] S201: Receive the decorators corresponding to each operation object sent by an open-source machine learning platform; wherein, each operation object and the decorator corresponding to each operation object are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object.

[0089] This step aims to receive the decorators corresponding to each operation object to facilitate the model training device to perform model training based on the decorators. The implementation manner of the open-source machine learning platform to create each operation and the decorator corresponding to each operation object can refer to the description in step S101 and will not be elaborated in this embodiment.

[0090] S202: Use each decorator to train a data retrieval model so that the data retrieval service server can load the data retrieval model to perform data retrieval operations.

[0091] This step aims to implement the training of the model. When the model training device receives the decorators corresponding to each operation object, it can use each decorator to train the model to obtain the corresponding data retrieval model, so that the data retrieval service server can load the data retrieval model to perform data retrieval operations.

[0092] In a possible implementation manner, the above-mentioned training the data retrieval model using each decorator may include the following steps:

[0093] Obtain the first feature sample of the retrieval element and the second feature sample of the recall element;

[0094] Use each decorator to train the first feature sample and the second feature sample to obtain a data retrieval model.

[0095] An embodiment of this application provides an implementation method for training a data retrieval model. It can be understood that during the model training process, training samples need to be used. The training samples should be corresponding model input samples and model output samples. Corresponding to the data retrieval model, the model input sample is the feature information of the retrieval element (i.e., the target to be retrieved), that is, the above-mentioned first feature sample, and the model output sample is the feature information of the recalled element (i.e., the retrieval result corresponding to the target to be retrieved), that is, the above-mentioned second feature sample. Therefore, the first feature sample of the retrieval element and the second feature sample of the recalled element can be obtained first, and then each decorator (Wrapper) can be used to implement sample training to obtain a data retrieval model.

[0096] It can be seen that a data retrieval method provided in this embodiment supports operations such as generating indexes, saving indexes, and retrieving data related to HNSW by creating operation objects, and creating corresponding decorators for the operations. These operation objects are compiled and linked to the data retrieval service server, and the decorators are sent to the model training device, enabling the offline training model part to use these operation objects through the model training device to integrate other model logics and complete the training of multiple data retrieval models; at the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional business development.

[0097] An embodiment of this application provides another data retrieval method.

[0098] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of another data retrieval method provided by an embodiment of this application. This data retrieval method is applied to the data retrieval service server in the data retrieval system, and this data retrieval method may include the following steps S301 to S302.

[0099] S301: When compiling and linking to each operation object created by the open-source machine learning platform, load and obtain a data retrieval model from the model training device; among them, the data retrieval model is trained by the model training device based on the decorators corresponding to each operation object. Each operation object and the decorators corresponding to each operation object are created by the open-source machine learning platform. The operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object.

[0100] This step aims to realize the compilation and linking of operation objects. After the open source machine learning platform creates each OP and the model training device trains each data retrieval model, in order to realize data retrieval, the open source machine learning platform compiles and connects each operation object to the data retrieval service server. After the model training device obtains the data retrieval model, the data retrieval service server loads the data retrieval model. At this point, the data retrieval service server can perform data retrieval based on the data retrieval model and each operation object when receiving a user retrieval request.

[0101] S302: Execute a data retrieval operation using a data retrieval model.

[0102] This step is intended to achieve data retrieval. During the implementation process, when the data retrieval service server receives a data retrieval request, the data retrieval service server executes the data retrieval request to achieve data retrieval. For the data retrieval service server, since different retrieval elements correspond to different data retrieval models, after receiving the data retrieval request, it can load the corresponding target data retrieval model according to the instruction information in the data retrieval request, and then use the target data retrieval model to complete the data retrieval.

[0103] In one possible implementation, after receiving a data retrieval request, the data retrieval service server may load a corresponding target data retrieval model according to the indication information in the data retrieval request, and implement data retrieval using the target data retrieval model, which may include: after the data retrieval service server receives the data retrieval request, the data retrieval service server obtains the target retrieval element and the characteristic information of the target retrieval element according to the data retrieval request, and loads the target data retrieval model corresponding to the target retrieval element to process the characteristic information, and obtains the recall result of the target retrieval element to implement data retrieval.

[0104] The embodiment of the present application provides a method for realizing a data retrieval function based on a data retrieval service server. In a data retrieval request, indication information of a target retrieval element and its characteristic information may be carried. Therefore, after receiving the data retrieval request, the data retrieval service server may determine the target retrieval element and its characteristic information by parsing the data retrieval request, and then retrieve the target data retrieval model corresponding to the target retrieval element, and input the characteristic information of the target retrieval element into the target data retrieval model, so as to obtain the recall result of the target retrieval element, that is, the data retrieval result, thereby realizing data retrieval.

[0105] It can be seen that a data retrieval method provided in this embodiment supports operations such as generating indexes, saving indexes, and retrieving data related to HNSW by creating operation objects, creating decorators corresponding to the operations, compiling and linking these operation objects to a data retrieval service server, and sending the decorators to a model training device. This enables the offline training model part to use these operation objects through the model training device to integrate other model logics and complete the training of multiple data retrieval models. At the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional business development.

[0106] Another data retrieval method is provided in an embodiment of this application.

[0107] Please refer to Figure 5 , Figure 5 FIG. is a schematic flowchart of another data retrieval method provided in an embodiment of this application. This data retrieval method is applied to a data retrieval system, which is the data retrieval system described above, including an open-source machine learning platform, a model training device, and a data retrieval service server. This data retrieval method may include the following steps S401 to S403.

[0108] S401: Create operation objects and decorators corresponding to each operation object through an open-source machine learning platform, send each decorator to a model training device, and compile and link each operation object to a data retrieval service server; wherein, the operation objects include a generate-index operation object, a save-index operation object, and a retrieve-data operation object.

[0109] In this embodiment, the implementation manner of creating operation objects by the open-source machine learning platform and creating decorators corresponding to each operation object may refer to the description in step S101 above and will not be elaborated in this embodiment. After the open-source machine learning platform creates each operation object, compile and link each operation object to the data retrieval service server. After the open-source machine learning platform creates decorators corresponding to each operation object, send each decorator to the model training device.

[0110] S402: Train a data retrieval model based on each decorator through a model training device.

[0111] In this embodiment, the model training device receives each decorator sent by the open-source machine learning platform and trains a corresponding data retrieval model based on each decorator. The implementation manner of the model training device training a data retrieval model based on each decorator may refer to the description in step S202 above and will not be elaborated in this embodiment.

[0112] S403: Load the data retrieval model through a data retrieval service server to perform a data retrieval operation.

[0113] In this embodiment, after the model training device obtains each data retrieval model, the data retrieval service server loads each data retrieval model obtained by the model training device. After each created operation object is compiled and linked to the data retrieval service server on the open-source machine learning platform, when the data retrieval service server receives a data retrieval request, it can implement data retrieval based on each data retrieval model and each compiled and linked operation object. After receiving the data retrieval request, the implementation method of obtaining the data retrieval result according to the data retrieval request can refer to the steps of S302 described above, and will not be elaborated in this embodiment.

[0114] It can be seen that a data retrieval method provided in this embodiment supports operations related to HNSW such as index generation, index saving, and data retrieval by creating operation objects, and creates corresponding decorators for the operations. These operation objects are compiled and linked to the data retrieval service server, and the decorators are sent to the model training device, enabling the offline training model part to use these operation objects through the model training device to integrate other model logics and complete the training of multiple data retrieval models. At the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional business development.

[0115] Based on the above application embodiments, another data retrieval method is provided in this application embodiment taking U2I (User To Item) recall as an example. Please refer to Figure 6 , Figure 6 which is the schematic diagram of a data retrieval method provided in this application embodiment. Referring to this schematic diagram, the implementation process of performing U2I recall is as follows:

[0116] Step 1: The open-source machine learning platform creates TensorFlow OP according to the TensorFlow specification, including HNSWTable OP (index generation OP), HNSWSave OP (index saving OP), and HNSWFind OP (data retrieval OP). Among them, HNSWTable OP is used to implement the operation of generating the HNSW index, HNSWSave OP is used to implement the operation of saving the HNSW index, and HNSWFind OP is used to implement the operation of retrieving data, specifically approximate nearest neighbor search.

[0117] (1) Register HNSWTable OP according to the TensorFlow specification, implement HNSWTable Kernel, and bind HNSWTable OP and HNSWTable Kernel;

[0118] (2) Register the HNSWSave OP according to the TensorFlow specification, implement the HNSWSave Kernel, and bind the HNSWSaveOP to the HNSWSave Kernel;

[0119] (3) Register the HNSWFind OP according to the TensorFlow specification, implement the HNSWFind Kernel, and bind the HNSWFindOP to the HNSWFind Kernel.

[0120] Step 2: The open-source machine learning platform creates corresponding Python Wrappers for each TensorFlow OP, and can obtain hnsw_table (generate index Wrapper), hnsw_save (save index Wrapper), and hnsw_find (retrieve data Wrapper) respectively.

[0121] (1) Load the dynamic library corresponding to the HNSWTable OP, bind the dynamic library to the HNSWTable OP, and generate hnsw_table;

[0122] (2) Load the dynamic library corresponding to the HNSWSave OP, bind the dynamic library to the HNSWSave OP, and generate hnsw_save;

[0123] (3) Load the dynamic library corresponding to the HNSWFind OP, bind the dynamic library to the HNSWFind OP, and generate hnsw_find.

[0124] Step 3: The open-source machine learning platform compiles the TensorFlow OP into the hnsw_op static library and links the hnsw_ops static library to TensorFlow Serving.

[0125] (1) Compile the HNSWTable OP into the corresponding static library, and by adding the HNSWTable OP dependency to the Bazel target of TensorFlow Serving, implement linking the static library corresponding to the HNSWTable OP to TensorFlow Serving;

[0126] (2) Compile the HNSWSave OP into the corresponding static library, and by adding the HNSWSave OP dependency to the Bazel target of TensorFlow Serving, implement linking the static library corresponding to the HNSWSave OP to TensorFlow Serving;

[0127] (3) Compile the HNSWFind OP into the corresponding static library, and link the static library corresponding to the HNSWFind OP to TensorFlow Serving by adding the dependency of the HNSWFind OP in the Bazel target of TensorFlow Serving.

[0128] Step 4. Offline training: After the model training device receives the generated index Wrapper, saved index Wrapper, and retrieved data Wrapper sent by the open-source machine learning platform, it first loads the Embeddings (feature samples) of the Items and generates an index using the generated index Wrapper; further, calculates the Embedding (feature sample) of the User and retrieves data using the retrieved data Wrapper; finally, saves the index using the saved index Wrapper. Thus, a data retrieval model is generated.

[0129] Step 5. Online retrieval: When a data retrieval request is received, use the compiled TensorFlow Serving to load the data retrieval model for data retrieval. Among them, TensorFlow Serving supports GRPC (a kind of RPC, Remote Procedure Call, remote procedure call) service and HTTP (Hyper Text Transfer Protocol, hypertext transfer protocol) service.

[0130] It can be seen that the data retrieval method provided by the embodiments of this application, by creating operation objects to support operations such as generating indexes, saving indexes, and retrieving data related to HNSW, creating corresponding decorators for the operations, and compiling and linking these operation objects to the data retrieval business server and sending the decorators to the model training device, enables the offline training model part to integrate other model logics using these operation objects through the model training device and complete the training of multiple data retrieval models; at the same time, supports the data retrieval business server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional business development.

[0131] The embodiments of this application provide an open-source machine learning platform.

[0132] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of an open-source machine learning platform provided by the embodiments of this application. The open-source machine learning platform may include:

[0133] Creation module 101, which is used to create operation objects and create decorators corresponding to each operation object; among them, the operation objects include a generation index operation object, a save index operation object, and a retrieve data operation object.

[0134] Sending module 102, which is used to send each decorator to a model training device, so that the model training device can train a data retrieval model based on each decorator.

[0135] Compilation module 103, which is used to compile and link each operation object to a data retrieval service server, so that the data retrieval service server can load the data retrieval model to perform data retrieval operations.

[0136] It can be seen that for the open-source machine learning platform provided by the embodiments of the present application, by creating operation objects to support operations related to HNSW such as generating indexes, saving indexes, and retrieving data, and creating decorators corresponding to the operations, and compiling and linking these operation objects to a data retrieval service server and sending the decorators to a model training device, the offline training model part can use these operation objects through the model training device to integrate other model logics and complete the training of multiple data retrieval models; at the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional business development.

[0137] In an embodiment of the present application, the above-mentioned creation module 101 may specifically be used to obtain registration information corresponding to each operation object; create a kernel corresponding to each operation object; bind the registration information and the kernel corresponding to the same operation object to generate an operation object.

[0138] In an embodiment of the present application, the above-mentioned creation module 101 may specifically be used to, for each operation object, load the operation object dynamic library corresponding to the operation object; bind the operation object to the operation object dynamic library to generate a decorator corresponding to the operation object.

[0139] In an embodiment of the present application, the above-mentioned compilation module 103 may specifically be used to compile and link each operation object to a data retrieval service server, so that the data retrieval service server can load the data retrieval model to perform data retrieval operations.

[0140] For the introduction of the device provided by the embodiments of the present application, please refer to the above method embodiments, and the embodiments of the present application will not be elaborated here.

[0141] The embodiments of the present application provide a model training device.

[0142] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of a model training device provided by the embodiments of the present application. The model training device may include:

[0143] A receiving module 201, configured to receive decorators corresponding to each operation object sent by an open-source machine learning platform; wherein, each operation object and the decorator corresponding to each operation object are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object.

[0144] A training module 202, configured to use each decorator to train and obtain a data retrieval model, so that a data retrieval service server loads the data retrieval model to perform data retrieval operations.

[0145] It can be seen that the model training device provided by the embodiment of the present application supports operations related to HNSW such as generating indexes, saving indexes, and retrieving data by creating operation objects, and creates decorators corresponding to the operations, and compiles and links these operation objects to the data retrieval service server and sends the decorators to the model training device, so that the offline training model part can use these operation objects through the model training device to integrate other model logics and complete the training of various data retrieval models; at the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be realized without additional service development.

[0146] In an embodiment of the present application, the training module 202 may be specifically configured to obtain a first feature sample of a retrieval element and a second feature sample of a recalled element; use each decorator to perform model training on the first feature sample and the second feature sample to obtain a data retrieval model.

[0147] For the introduction of the device provided by the embodiment of the present application, please refer to the above method embodiment, and the embodiment of the present application will not be elaborated here.

[0148] The embodiment of the present application provides a data retrieval service server.

[0149] Please refer to Figure 9 , Figure 9 , which is a schematic structural diagram of a data retrieval service server provided by an embodiment of the present application. The data retrieval service server may include:

[0150] A loading module 301, configured to load and obtain a data retrieval model from a model training device when compiled and linked to each operation object created by an open-source machine learning platform; wherein, the data retrieval model is obtained by the model training device based on the decorators corresponding to each operation object, and each operation object and the decorator corresponding to each operation object are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object.

[0151] A retrieval module 302, configured to perform data retrieval operations using the data retrieval model.

[0152] It can be seen that the model training device provided by the embodiments of the present application supports operations related to HNSW such as generating indexes, saving indexes, and retrieving data by creating operation objects, and creates corresponding decorators for the operations. These operation objects are compiled and linked to the data retrieval service server, and the decorators are sent to the model training device, enabling the offline training model part to use these operation objects through the model training device to integrate other model logics and complete the training of various data retrieval models. At the same time, it supports the data retrieval service server to load the data retrieval model online for data retrieval, that is, online data retrieval can be achieved without additional business development.

[0153] For the introduction of the device provided by the embodiments of the present application, please refer to the above method embodiments, and the embodiments of the present application will not be elaborated here.

[0154] The embodiments of the present application provide an electronic device.

[0155] Please refer to Figure 10 , Figure 10 which is a schematic structural diagram of an electronic device provided by the present application. The electronic device may include:

[0156] A memory for storing computer programs;

[0157] A processor that can implement the steps of any of the above data retrieval methods when executing the computer program.

[0158] As Figure 10 shown, it is a schematic diagram of the composition structure of the electronic device. The electronic device may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 all complete communication with each other through the communication bus 13.

[0159] In the embodiments of the present application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices, etc. The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the data retrieval method.

[0160] The memory 11 is used to store one or more programs. The program may include program codes, and the program codes include computer operation instructions. In the embodiments of the present application, the memory 11 stores at least a program for implementing the following functions: creating operation objects and creating decorators corresponding to each operation object; wherein, the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object; sending each decorator to a model training device so that the model training device trains a data retrieval model based on each decorator; compiling and linking each operation object to a data retrieval service server so that the data retrieval service server loads the data retrieval model to perform a data retrieval operation. Or,

[0161] Receiving decorators corresponding to each operation object sent by an open-source machine learning platform; wherein, each operation object and the decorator corresponding to each operation object are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object; using each decorator to train a data retrieval model so that the data retrieval service server loads the data retrieval model to perform a data retrieval operation.

[0162] Or,

[0163] When compiling and linking to each operation object created by an open-source machine learning platform, loading and obtaining a data retrieval model from a model training device; wherein, the data retrieval model is trained by the model training device based on the decorators corresponding to each operation object, each operation object and the decorator corresponding to each operation object are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object;

[0164] Using the data retrieval model to perform a data retrieval operation.

[0165] Or,

[0166] Creating operation objects and decorators corresponding to each operation object through an open-source machine learning platform, sending each decorator to a model training device, and compiling and linking each operation object to a data retrieval service server; wherein, the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object; training a data retrieval model through the model training device based on each decorator; and loading the data retrieval model through the data retrieval service server to perform a data retrieval operation.

[0167] In a possible implementation manner, the memory 11 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function, etc.; the data storage area may store data created during use.

[0168] In addition, the memory 11 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device or other volatile solid-state storage devices.

[0169] The communication interface 12 may be an interface of a communication module for connecting to other devices or systems.

[0170] Of course, it should be noted that Figure 10 the structure shown does not constitute a limitation on the electronic device in the embodiments of the present application. In practical applications, the electronic device may include more or fewer components than Figure 10 shown, or combine certain components.

[0171] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above data retrieval methods can be implemented.

[0172] The computer-readable storage medium may include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0173] For the introduction of the computer-readable storage medium provided in the embodiments of the present application, please refer to the above method embodiments, and the embodiments of the present application will not be elaborated herein.

[0174] The various embodiments in the specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description in the method part.

[0175] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0176] The above has introduced the technical solution provided by this application in detail. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and modifications can still be made to this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data retrieval method, characterized in that, Applied to an open-source machine learning platform, including: Create operation objects and create decorators corresponding to each of the operation objects; wherein, the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object; Send each of the decorators to a model training device so that the model training device trains a data retrieval model based on each of the decorators; Compile and link each of the operation objects to a data retrieval service server so that the data retrieval service server loads the data retrieval model to perform a data retrieval operation.

2. The method according to claim 1, wherein The creating of the operation objects includes: Obtain the registration information corresponding to each of the operation objects; Create a kernel corresponding to each of the operation objects; Bind the registration information and the kernel corresponding to the same operation object to generate the operation object.

3. The method according to claim 1, characterized in that, The creating of the decorators corresponding to each of the operation objects includes: For each of the operation objects, load the operation object dynamic library corresponding to the operation object; Bind the operation object and the operation object dynamic library to generate the decorator corresponding to the operation object.

4. The method according to claim 1, wherein The compiling and linking of each of the operation objects to the data retrieval service server includes: For each of the operation objects, compile the operation object to generate a corresponding operation object static library; Link each of the operation object static libraries to the data retrieval service server.

5. A data retrieval method, characterized in that, Applied to a model training device, including: Receive the decorators corresponding to each operation object sent by the open-source machine learning platform; wherein, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object; Train a data retrieval model by using each of the decorators so that the data retrieval service server loads the data retrieval model to perform a data retrieval operation.

6. The method according to claim 5, wherein The training of the data retrieval model by using each of the decorators includes: Obtain a first feature sample of a retrieval element and a second feature sample of a recall element; Train the first feature sample and the second feature sample by using each of the decorators to obtain the data retrieval model.

7. A data retrieval method, characterized in that, Applied to a data retrieval service server, including: When compiling and linking to each operation object created by the open-source machine learning platform, load and obtain a data retrieval model from a model training device; wherein, the data retrieval model is trained by the model training device based on the decorators corresponding to each of the operation objects, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include a generate index operation object, a save index operation object, and a retrieve data operation object; Perform a data retrieval operation by using the data retrieval model.

8. A data retrieval method, characterized in that, Applied to a data retrieval system, the data retrieval system includes an open-source machine learning platform, a model training device, and a data retrieval service server, and the method includes: Create operation objects and decorators corresponding to each of the operation objects through the open-source machine learning platform, send each of the decorators to the model training device, and compile and link each of the operation objects to the data retrieval service server; wherein, the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object. Train a data retrieval model based on each of the decorators through the model training device. Load the data retrieval model through the data retrieval service server to perform a data retrieval operation.

9. An open-source machine learning platform, characterized in that, Comprises: A creation module, configured to create operation objects and create decorators corresponding to each of the operation objects; wherein, the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object. A sending module, configured to send each of the decorators to the model training device, so that the model training device trains a data retrieval model based on each of the decorators. A compilation module, configured to compile and link each of the operation objects to the data retrieval service server, so that the data retrieval service server loads the data retrieval model to perform a data retrieval operation.

10. A model training device, characterized in that, Comprises: A receiving module, configured to receive the decorators corresponding to each operation object sent by the open-source machine learning platform; wherein, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object. A training module, configured to train a data retrieval model by using each of the decorators, so that the data retrieval service server loads the data retrieval model to perform a data retrieval operation.

11. A data retrieval service server, characterized in that, Comprises: A loading module, configured to, when compiling and linking to each operation object created by the open-source machine learning platform, load and obtain a data retrieval model from the model training device; wherein, the data retrieval model is trained by the model training device based on the decorators corresponding to each of the operation objects, each of the operation objects and the decorators corresponding to each of the operation objects are created by the open-source machine learning platform, and the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object. A retrieval module, configured to perform a data retrieval operation by using the data retrieval model.

12. A data retrieval system, characterized in that, Comprises: An open-source machine learning platform, configured to create operation objects and decorators corresponding to each of the operation objects, send each of the decorators to the model training device, and compile and link each of the operation objects to the data retrieval service server; wherein, the operation objects include an index generation operation object, an index saving operation object, and a data retrieval operation object. The model training device, configured to train a data retrieval model based on each of the decorators. The data retrieval service server, configured to load the data retrieval model to perform a data retrieval operation.

13. An electronic device, characterized in that, Comprises: A memory, configured to store a computer program. A processor, configured to, when executing the computer program, implement the steps of the data retrieval method according to any one of claims 1 to 8.

14. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data retrieval method according to any one of claims 1 to 8 are implemented.

Images