Model training sample verification method and its device, equipment, medium, and product

Abstract

This application discloses a method, apparatus, device, medium, and product for validating model training samples. The method includes responding to a class annotation event acting on product query term samples and product title samples in a candidate sample pool; obtaining the first labeled product category for the product query term samples and product title samples; calling the first sample class annotation model to annotate the second labeled product category corresponding to the product query term samples and product title samples with the first labeled product category; pushing product query term samples or product title samples with the same first labeled product category and second labeled product category to the training sample library; and updating the first sample class annotation model based on the latest converged second sample class annotation model trained on the training sample library. This application achieves automated annotation and validation of training samples, and forms efficient utilization of model resources, effectively improving the inference capability of the model to be deployed online.

Description

Technical Field

[0001] The present application relates to the field of neural network models, and in particular to a method for verifying model training samples. In addition, the present application also relates to corresponding devices, equipment, non-volatile storage media and computer program products of the method. Background Art

[0002] There are various types of online services built based on neural network models in existing Internet platforms. For example, online services used to provide users with intelligent customer service generally have model services corresponding to neural network models used for semantic reasoning. Online services used to provide users with product classification or product query services generally have model services built based on neural network models. With the development of neural network technology, each platform needs to provide corresponding labeled sample data to the model service to drive the neural network model in the model service to use the new labeled sample data for training, thereby improving the business processing efficiency of online services built based on neural network models and optimizing the user experience.

[0003] However, existing platforms often do not verify the labeled sample data when pushing them to the neural network model for training. If the neural network model directly uses the unverified labeled sample data, it will undoubtedly pose a threat to the stability of the system operation of the online services provided by the neural network model in the platform.

[0004] In view of the problems existing in data verification of labeled sample data for training neural network models in existing online services, the applicant has made corresponding explorations in order to solve this problem. Summary of the invention

[0005] The purpose of this application is to provide a model training sample verification method to meet user needs. In addition, it also relates to corresponding devices, equipment, non-volatile storage media and computer program products of the method.

[0006] In order to achieve the purpose of this application, the following technical solutions are adopted:

[0007] A model training sample verification method proposed for the purpose of this application includes the following steps:

[0008] In response to a category annotation event acting on a product query word sample and a product title sample in a candidate sample pool, obtain a first annotated product category annotated by the event for the product query word sample and the product title sample;

[0009] The first sample category annotation model that has been trained to convergence is called to annotate the second annotated product category corresponding to the product query word samples and product title samples having the first annotated product category;

[0010] Pushing product query word samples or product title samples of the same first labeled product category and second labeled product category to the training sample library, driving the second sample category target annotation model that has not been trained to convergence to call the training sample library for training;

[0011] According to the second sample class target annotation model that is most recently trained to convergence based on the training sample library, the first sample class target annotation model is intervened to be updated.

[0012] In a further embodiment, the following pre-steps are included:

[0013] Calling the active learning sample screening model to drive the active learning sample screening model to classify each product title sample in the product title pool into a product category;

[0014] Obtaining commodity category classification data corresponding to each of the commodity title samples, wherein the commodity category classification data represents the classification probability of the commodity title samples and each commodity category;

[0015] The commodity title samples having a classification probability exceeding a preset threshold representing any commodity category in the commodity category classification data are stored in the training sample library.

[0016] In a further embodiment, the following pre-steps are included:

[0017] Obtaining user historical behavior information, parsing one or more product query word samples and their corresponding query interaction values ​​contained in the user historical behavior information;

[0018] The commodity query word samples whose query interaction values ​​exceed the preset screening value are stored in the candidate sample pool.

[0019] In a further embodiment, the step of responding to the category annotation event of the product query word sample and the product title sample in the candidate sample pool includes:

[0020] According to the preset sample scattering rule, any multiple product query word samples and product title samples stored in the candidate sample pool are obtained and pushed to the annotation end;

[0021] The first annotated commodity categories corresponding to the query word samples and the commodity title samples are obtained by the annotation end, and the first annotated commodity categories are stored in the candidate sample pool in correspondence with the query word samples or the commodity title samples.

[0022] In a further embodiment, the step of pushing the product query word samples or product title samples having the same first labeled product category and second labeled product category to the training sample library includes:

[0023] Return samples with different labels to the candidate sample pool

[0024] Determine whether the first marked product category of the product query word sample or the product title sample is the same as the corresponding second marked product category;

[0025] If they are not the same, the first annotated product category and the second annotated product category of the product query word sample or the product title sample are returned and stored in the candidate sample pool as mapping relationship data;

[0026] If they are the same, the commodity query word samples or commodity title samples that mark the commodity category are stored in the training sample library as mapping relationship data.

[0027] In a further embodiment, the step of calling the first sample category annotation model to annotate the second annotated product category corresponding to the product query word sample and the product title sample having the first annotated product category includes:

[0028] Performing text preprocessing on product query word samples or product title samples;

[0029] Calling a feature extraction model that has been trained to convergence, wherein the extraction model extracts a feature vector of the product query word sample or the product title sample;

[0030] The category classification model that has been trained to convergence is called to calculate the probability that the feature vector corresponds to each product category, and the product category with the highest probability is used as the second labeled product category of the product query term sample or product title sample.

[0031] A model training sample verification device proposed for the purpose of this application includes:

[0032] A first category annotation module is used to respond to the category annotation event acting on the product query word samples and product title samples in the candidate sample pool, and obtain the first annotated product category annotated by the event for these product query word samples and product title samples;

[0033] The second category target annotation module is used to call the first sample category target annotation model that has been trained to convergence, and annotate the second annotated product category corresponding to the product query word samples and product title samples with the first annotated product category;

[0034] A sample storage module, used to push product query word samples or product title samples of the same first labeled product category and second labeled product category to a training sample library, and drive the second sample category target annotation model that has not been trained to convergence to call the training sample library for training;

[0035] The model updating module is used to intervene in the first sample class target annotation model for updating according to the second sample class target annotation model that is most recently trained to convergence based on the training sample library.

[0036] In a further embodiment, the first type target injection module includes:

[0037] The sample push submodule is used to obtain any multiple product query word samples and product title samples stored in the candidate sample pool and push them to the annotation end according to the preset sample scattering rule;

[0038] The sample storage submodule is used to obtain the first labeled commodity categories corresponding to the query word samples and commodity title samples respectively acted on by the labeling end, and store these first labeled commodity categories and their query word samples or commodity title samples in the candidate sample pool in correspondence.

[0039] In a further embodiment, the second type target injection module includes:

[0040] A sample preprocessing submodule is used to perform text preprocessing on product query word samples or product title samples;

[0041] A feature vector extraction submodule is used to call a feature extraction model that has been trained to convergence, and the extraction model extracts a feature vector of the product query word sample or the product title sample;

[0042] The category target annotation submodule calls the category classification model that has been trained to convergence, calculates the probability that the feature vector corresponds to each product category, and uses the product category with the largest probability as the second labeled product category of the product query term sample or product title sample.

[0043] In a further embodiment, the sample storage module includes:

[0044] A labeling verification submodule, used to determine whether the first labeling commodity category of the commodity query word sample or the commodity title sample is the same as the corresponding second labeling commodity category;

[0045] A sample return submodule, for returning and storing the first labeled product category and the second labeled product category of the product query word sample or the product title sample as mapping relationship data to the candidate sample pool if they are not the same;

[0046] The training sample storage submodule is used to store the commodity query word samples or commodity title samples that mark the commodity category as mapping relationship data in the training sample library if they are the same.

[0047] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned model training sample verification method.

[0048] In order to solve the above technical problems, an embodiment of the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned model training sample verification method.

[0049] In order to solve the above technical problems, an embodiment of the present application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned model training sample verification method.

[0050] Compared with the prior art, the advantages of this application are as follows:

[0051] The present application can be used for the platform to automatically verify and screen the labeled sample data that needs to be input into the neural network model for training, by pushing the product title samples and product query word samples in the platform to the labeling end for product category annotation, and then pushing the product title samples and product query word samples labeled by the labeling end to the model built based on the neural network model for product category, and then verifying whether the product categories labeled by both parties for the samples are the same, so as to feed the same samples into the model that needs to be deployed online for e-commerce business services for model training; through multi-dimensional labeling and verification, it is determined whether the samples can be used as training samples for model training in the model that provides e-commerce business services for the platform, so as to ensure the data validity of the training samples, so as to improve the model training quality of the model, and then optimize the service quality of the corresponding e-commerce business services in the e-commerce platform, and improve the user experience of the e-commerce platform.

[0052] Secondly, the labeling model used to label candidate samples in this application is the same as the neural network architecture of the service model that needs to be deployed to the online service. After the service model calls the training samples labeled by the labeling model and trains them to convergence, the labeling model will be updated synchronously to form a recycling of model resources and improve the labeling accuracy of the model, thereby providing the service model with training samples with better data quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0054] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;

[0055] Figure 2 A flowchart of a typical embodiment of the model training sample verification method of the present application;

[0056] Figure 3A flowchart of a specific implementation method of the present application for screening product title samples;

[0057] Figure 4 A flowchart of a specific implementation method of the present application for screening product query word samples;

[0058] Figure 5 A flowchart diagram of a specific implementation method of the present application for determining the first labeled commodity category of a commodity title sample and a commodity query term sample;

[0059] Figure 6 A flowchart diagram of a specific implementation method of the present application for determining the second labeled commodity category of a commodity title sample and a commodity query term sample;

[0060] Figure 7 A flowchart of a specific implementation method of verifying a first labeled product category and a second labeled product category of a product title sample or a product query word sample in this application;

[0061] Figure 8 This is a principle block diagram of a typical embodiment of a model training sample verification device of the present application;

[0062] Fig. 9 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0064] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0065] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0066] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; traditional laptop and / or palmtop computers or other devices, which have and / or include radio frequency receivers. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playback function, or a smart TV, a set-top box and other devices.

[0067] The hardware referred to by the names such as "server", "client", and "working node" in this application is essentially an electronic device with capabilities equivalent to those of a personal computer. It is a hardware device that has the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. Computer programs are stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0068] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0069] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., so as to form a logically related service cluster to provide services for relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.

[0070] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0071] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium recognizable by a computer in the form of computer executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.

[0072] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0073] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0074] See also Figure 2 , a model training sample verification method of the present application, in its typical embodiment, comprises the following steps:

[0075] Step S11, responding to the category annotation event acting on the product query word samples and product title samples in the candidate sample pool, obtaining the first annotated product category annotated by the event for these product query word samples and product title samples:

[0076] The server responds to the category annotation event acting on the product query word samples and product title samples stored in the candidate sample pool, and obtains the first annotated product category annotated by the category annotation event for the product query word samples and product title samples.

[0077] The candidate sample pool stores a plurality of the product query word samples and product title samples, which are samples pre-screened by the server. For example, the query interaction value of the product query word samples stored in the candidate sample pool generally exceeds the preset screening value. The query interaction value refers to when a platform user searches for a target product on the platform through a product query word sample, after browsing or clicking on any product on the product browsing page, the query interaction value of the product query word sample will increase. For example, after a platform user clicks on a target product and enters its corresponding product details page, the query interaction value of the corresponding product query word sample will increase. Plus 1 or more, correspondingly, the value of the preset screening value is generally greater than; correspondingly, the product title samples stored in the candidate sample pool are generally the server calling the active learning sample screening model, and then driving the active learning sample screening model to classify each product title sample in the product title pool into a product category, so as to obtain the product category classification data corresponding to each of the product title samples, the product category classification data represents the classification probability of the product title samples and each product category, and then the product title samples having a classification probability of any product category represented by the product category classification data exceeding the preset threshold are stored in the training sample library.

[0078] The product query word sample refers to the product query word text entered by the platform user when conducting a product query event on the e-commerce platform. For example, the e-commerce platform generally provides product query services to the platform users. The platform user enters the product query word in the corresponding control to query the target product corresponding to the product query word on the e-commerce platform for browsing and shopping. The server collects the product query words to obtain the corresponding product query word samples and stores them in the candidate sample words.

[0079] The product title sample refers to the product title text corresponding to the product object in the e-commerce platform. The server stores the product title samples with inaccurate product categories in the candidate sample pool for subsequent verification, thereby preventing the product title samples with inaccurate product categories from being fed into the second sample category annotation model that needs to be deployed online for product query services for training, thereby affecting the service quality of the product query service corresponding to the second sample category annotation model.

[0080] The category annotation event generally refers to an event of manually annotating the product category of the product title samples and product query word samples. Specifically, the server pushes the product title samples and product query word samples stored in the candidate sample pool to the manual annotation end, or the manual annotation end obtains the corresponding product title samples and product query word samples from the candidate pool through a data communication link established with the server. After the manual annotation end obtains the product title samples and product query word samples for output and display, the annotator at the manual annotation end will annotate the product category of these product title samples and product query word samples to mark the first annotated product category corresponding to each of these product title samples and product query word samples, and then the server responds to the category annotation event with the first annotated product category annotated for these product title samples and product query word samples.

[0081] The first labeled product category generally refers to the product category manually labeled, and the product category generally refers to the smallest level product category in the e-commerce platform. For example, a product title sample with the text content of "Xiaomi smartphone" generally corresponds to the product category of smartphones or "Xiaomi brand smartphones" of a newer level category.

[0082] Step S12, calling the first sample category annotation model that has been trained to convergence, annotating the second annotated product category corresponding to the product query word samples and product title samples with the first annotated product category:

[0083] The server calls the first sample category annotation model that has been trained to convergence, so as to annotate the second annotated product category corresponding to the product query term sample or product title sample having the first product category through the first sample category annotation model.

[0084] The sample category annotation model refers to a neural network model used to classify the product title samples and product query word samples into product categories, and the sample category annotation model generally classifies the product title samples and product query word samples into product categories based on Fasttext and Bert. Specifically, the server feeds the product query word samples and product title samples with the first annotated product category into the first sample category annotation model. After the first sample category annotation model obtains the product query word samples and product title samples, it removes special characters and performs word segmentation preprocessing on these product query word samples and product title samples respectively, so as to call the feature extraction model that has been trained to convergence. The extraction model extracts the feature vector of the product query word sample or the product title sample, and then calls the category classification model that has been trained to convergence, calculates the probability that the feature vector corresponds to each product category in the e-commerce platform, and uses the product category with the largest probability as the second annotated product category of the product query word sample or the product title sample.

[0085] The first sample category annotation model and the subsequent second annotated product category model are neural network models constructed with the same network structure, and the first sample category annotation model is a neural network model trained to a convergence state, while the second annotated product category model is a neural network model that has not been trained to a convergence state. The second annotated product category model is a neural network model that calls a subsequent training sample library for training. For the specific implementation method, please refer to the relevant description in the subsequent steps, and this step will not be repeated.

[0086] As mentioned above, the second annotated product category of the product title sample and the product query word sample refers to the product category annotated by the first sample category annotation model through its neural network algorithm.

[0087] Step S13, push the product query word samples or product title samples of the same first labeled product category and second labeled product category to the training sample library, and drive the second sample category annotation model that has not been trained to convergence to call the training sample library for training:

[0088] After the server annotates the second annotated product category of the product query word sample or product title sample having the first annotated product category through the first sample category annotation model, it will determine whether the first annotated product category and the second annotated product category of the product query word sample or product title sample are the same product category. If they are the same, the product query word sample or product title text and its corresponding product category will be stored as mapping relationship data in the training sample library.

[0089] The training sample library stores one or more product query word samples and product title samples and their corresponding product categories, and each of the product query word samples and product title samples and their corresponding product categories are stored as mapping relationship data, so that the second category annotation model calls the query word and product title samples stored in the training sample library and their corresponding product types as training samples for training.

[0090] The second sample class target annotation model and the first sample class target annotation model are neural network models with the same network structure, wherein the second sample class target annotation model is generally in a state from untrained to trained, and calls the product title samples and product query word samples in the training sample library for training, which are verified by the server for product category with the help of the first sample class target annotation model. As described in the subsequent steps, when the second sample class target annotation model completes the staged training and converges through the training sample library, the server will intervene in the first sample class target annotation model based on the second sample class target annotation model for synchronous update to form a cyclic update of the model.

[0091] When the server determines that the first annotated product category of the product query word sample or the product title text is different from the second annotated product category, the server reflows and stores the product query word sample or the product title text into the candidate sample word, so that after synchronously updating the first sample category annotation model based on the second sample category annotation model, the first sample category annotation model will re-call the product title sample and the product query word sample reflowed and stored in the candidate sample pool for product category annotation.

[0092] Step S14, according to the second sample class annotation model that has been most recently trained to convergence based on the training sample library, the first sample class annotation model is intervened to be updated:

[0093] The second sample class target annotation model calls the mapping relationship data consisting of product title samples or product query term samples and their corresponding product categories stored in the training sample library to perform model training. When the second sample class target annotation model calls the training sample library to train to a convergence state, the server will intervene in the first sample class target annotation model according to the second sample class target annotation model to perform corresponding updates. For example, because the first sample class target annotation model and the second sample class target annotation model have the same neural network structure, the server can update the first sample class target annotation model with the latest weights in the second sample class target annotation model trained to a convergence state, or replace the first sample class target annotation model with the second sample class target annotation model.

[0094] After the server updates the first sample class annotation model according to the second sample class annotation model, it will continue to verify the product title samples or product query term samples stored in the candidate sample pool based on the latest updated first sample class annotation model, and then fill the training samples in the training sample library to improve the reasoning accuracy of the second sample class annotation model, and then update the first sample class annotation model accordingly, and so on, continuously completing the first sample class annotation model and the second sample class annotation model to improve the service built based on the second sample class annotation model and improve the user experience of the platform users in the e-commerce platform.

[0095] Through the typical implementation of this method, it can be known that this method can automatically verify and screen the labeled sample data that needs to be input into the neural network model for training for the platform, by pushing the product title samples and product query word samples in the platform to the labeling end for product category annotation, and then pushing the product title samples and product query word samples labeled by the labeling end to the model built based on the neural network model for product category, and then verifying whether the product categories labeled by both parties for the samples are the same, so as to feed the same samples into the model that needs to be deployed online for e-commerce business services for model training; through multi-dimensional labeling and verification, it is determined whether the samples can be used as training samples This method is used to perform model training in the model that provides e-commerce business services for the platform, ensure the data validity of the training samples, so as to improve the model training quality of the model, and then optimize the service quality of the corresponding e-commerce business services in the e-commerce platform, and improve the user experience in the e-commerce platform; secondly, the annotation model used to annotate the candidate samples in this method is the same as the neural network architecture of the service model to be deployed to the online service, and after the service model calls the training samples annotated by the annotation model to converge, the annotation model will be updated synchronously to form a recycling of model resources and improve the annotation accuracy of the model, thereby providing training samples with better data quality for the service model.

[0096] The above typical embodiments and their variations fully disclose the implementation scheme of the model training sample verification method of the present application. However, various variations of the method can still be derived by changing and amplifying some technical means. Other embodiments are briefly described as follows:

[0097] In one embodiment, please refer to Figure 3 , the method comprises the following pre-steps:

[0098] Step S08, calling the active learning sample screening model, driving the active learning sample screening model to classify each product title sample in the product title pool into a product category:

[0099] The server calls the active learning sample screening model to determine that the active learning sample screening model performs commodity category classification on each commodity title sample in the commodity title pool.

[0100] The product title samples stored in the product title pool are generally collected from the product title texts corresponding to the products existing in the e-commerce platform.

[0101] The active learning sample screening model is based on a model constructed by active learning. The active learning sample screening model uses query strategies such as uncertainty sampling, diversity sampling, expected model changes or committee queries. First, the active learning sample screening model generally determines the product type classification data corresponding to the product title sample through its own multiple classifiers. The product type classification data includes the classification probability of the product title sample in each product category on the e-commerce platform. For example, when there are product categories of clothing, electrical appliances and food on the e-commerce platform, and the product title text corresponding to the product category classification is a smart TV, the corresponding product type classification data is generally (clothing: 0.02, electrical appliances: 0.97, food: 0.01).

[0102] Step S09, obtaining the commodity category classification data corresponding to each of the commodity title samples, wherein the commodity category classification data represents the classification probability of the commodity title samples and each commodity category:

[0103] The product category classification data of each of the product title samples determined by the active learning sample screening model is obtained. As described in the relevant embodiment in step S08, the product classification data of the product title sample is used to characterize the classification probability of the product title sample and each product category in the e-commerce platform. The classification probability of the product title text and each product category represented in the product classification data is data that passes the normalization standard, that is, the sum of the classification probabilities of the product title text and each product category is 1.

[0104] Step S10, storing the commodity title samples having a classification probability of representing any commodity category in the commodity category classification data exceeding a preset threshold into the training sample library:

[0105] After obtaining the product category classification data of any product title sample in the product title pool, it will be determined whether there is any product category in the product category classification data whose classification probability exceeds a preset threshold. If so, the product title sample will be stored in the training sample library. If not, the product title sample will be stored in the candidate sample pool for sample labeling processing.

[0106] In this embodiment, each product title sample in the product title pool composed of product feature information collected on the e-commerce platform will be screened, and the product title samples that do not pass the screening will be stored in the candidate sample pool for further sample labeling processing, so as to save the computing power of subsequent sample labeling processing and improve its execution efficiency.

[0107] In one embodiment, please refer to Figure 4 , the method comprises the following pre-steps:

[0108] Step S09', obtaining the user's historical behavior information, parsing one or more product query word samples and their corresponding query interaction values ​​contained in the user's historical behavior information:

[0109] The server obtains the user historical behavior information in the e-commerce platform, which is generated by the platform users by using the product query service provided by the e-commerce platform. The user historical behavior information includes the product query word samples input by the platform users when using the product query service and the query interaction value generated by interacting with the products in the product browsing page after searching through the product query word samples.

[0110] The query interaction value records the numerical value of the platform user's interaction with the products displayed on the product browsing page after searching for products through product query words. For example, when the platform user enters the corresponding product query word sample in the product query page, it will jump to the product browsing page corresponding to the product query word sample. The product browsing page will display the products corresponding to the product query word sample. When the platform user clicks on the corresponding product through the product browsing page, the product interaction value corresponding to the product query word sample will increase in value, such as adding 1 or other value-added operations. That is, the product interaction value can be generally understood as the number of clicks.

[0111] Step S10', storing the commodity query word samples whose query interaction value exceeds the preset screening value in the candidate sample pool:

[0112] After parsing out one or more product query word samples and their corresponding product interaction values ​​contained in the user historical behavior information, the server will determine whether these product interaction values ​​exceed the preset filtering value, and store the product query word samples corresponding to the preset filtering value in the candidate sample pool.

[0113] In this embodiment, the product query word samples in the platform are screened by the product interaction value, and the interactive product query word samples are stored in the candidate sample pool to prevent meaningless product query word samples from being stored in the candidate sample pool and causing a waste of storage resources, and to improve the execution efficiency of the sample annotation processing.

[0114] In one embodiment, please refer to Figure 5 The step of responding to the category annotation events of the product query word samples and product title samples in the candidate sample pool includes:

[0115] Step S111: according to the preset sample scattering rule, any multiple product query word samples and product title samples stored in the candidate sample pool are obtained and pushed to the annotation end:

[0116] According to the preset sample scattering rule, the server obtains a sample set consisting of two sample types, namely, product query word samples and product title samples, from the candidate sample pool and pushes the sample set to the annotation end.

[0117] Step S112, obtaining the first annotated product categories corresponding to the query word samples and product title samples respectively acted on by the annotation end, and storing the first annotated product categories in correspondence with their query word samples or product title samples in the candidate sample pool:

[0118] The annotation end generally refers to the annotation end that performs manual verification. The annotation end will receive the product query word samples and product title samples obtained and pushed by the server from the candidate sample pool according to the preset scattering rules and display them on the corresponding sample annotation page, and then the annotation personnel at the annotation end will annotate the product category with the product query word samples and product title samples displayed on the sample annotation page. When any product query word sample or product title sample completes the product category annotation, the annotation end pushes the product category as the first annotated product category of the product query word sample or product title sample to the server, so that the server stores the first annotated product category in the candidate sample pool in correspondence with its query word sample or product title sample.

[0119] In this embodiment, the sample set constituted by the candidate sample pool is pushed to the manual annotation end to be broken up to prevent samples of the same sample type from being pushed to the manual annotation end for annotation, which affects the annotation accuracy of the annotators, and the manually annotated commodity categories are stored in the candidate sample pool accordingly to facilitate the annotation verification of the candidate commodity categories.

[0120] In one embodiment, please refer to Figure 6 The step of calling the first sample category annotation model to annotate the second annotated product category corresponding to the product query word sample and the product title sample having the first annotated product category includes:

[0121] Step S121, performing text preprocessing on product query word samples or product title samples:

[0122] After the first sample category annotation model obtains the product query word sample or product title sample pushed by the server, it will perform text preprocessing on the product query word sample or product title sample. The text preprocessing refers to formatting processing such as special character removal and word segmentation processing on the product title sample and product query word sample.

[0123] Step S122, calling a feature extraction model that has been trained to convergence, and extracting a feature vector of the product query word sample or product title sample by the extraction model:

[0124] The first sample category annotation model calls the feature extraction model that has been trained to convergence to extract the feature vectors of the product query word samples or product title samples that have completed text preprocessing. The feature extraction model generally extracts the feature vectors of the product query word samples and product title samples based on TF-IDF or Word2Vec.

[0125] Step S123, calling the category classification model that has been trained to convergence, calculating the probability that the feature vector corresponds to each product category, and taking the product category with the highest probability as the second labeled product category of the product query word sample or product title sample:

[0126] The category classification model is generally constructed based on a neural network model architecture such as Fasttext or Bert. The category classification model calculates the similarity between the feature vector of the product query word sample or the product title sample and the features of each product category existing in the e-commerce platform, and then determines the probability that the product query word sample or the product title sample corresponds to each of these product categories. The sum of the probabilities that the product query word sample or the product title sample corresponds to each of these product categories is generally 1.

[0127] In this embodiment, the first sample category annotation model performs product category annotation processing on product query term samples and product title samples that already have the first labeled product category, so as to subsequently verify whether the product categories of the samples are consistent, and then determine whether the samples need to be used as training samples for the model to be deployed online.

[0128] In one embodiment, please refer to Figure 7 The step of pushing the product query word samples or product title samples having the same first labeled product category and second labeled product category to the training sample library includes:

[0129] Step S131, determining whether the first marked commodity category of the commodity query word sample or the commodity title sample is the same as the corresponding second marked commodity category:

[0130] The server determines whether the product title sample or the first marked product category and the second product category of the product title sample are the same.

[0131] Step S132: If they are not the same, the first annotated product category and the second annotated product category of the product query word sample or the product title sample are stored back in the candidate sample pool as mapping relationship data:

[0132] When the server determines that the product title sample or the first annotated product category of the product title sample is different from the second product category, the server returns the product title sample or the product title sample to the candidate sample pool for verification by the annotation end, or re-annotates the product category by the first sample category annotation model that is most recently updated based on the second sample category annotation model.

[0133] Step S133: If they are the same, the product query word samples or product title samples marked with the product category are stored in the training sample library as mapping relationship data:

[0134] When the server determines that the product title sample or the first labeled product category of the product title sample is the same as the second product category, the product title sample or the product title sample is stored in the training sample library to drive the second sample category annotation model that has not been trained to convergence to obtain the product title sample or the product title sample from the training sample library for model training.

[0135] In this embodiment, when verifying the two commodity categories labeled for the samples, the samples with different labeled commodity categories are returned and stored in the candidate sample pool to recycle the samples, while the samples with the same labeled commodity categories are stored in the training sample library so that the second sample category annotation model can perform model training.

[0136] Furthermore, by functionalizing each step in the method disclosed in the above embodiments, a model training sample verification device of the present application can be constructed. According to this idea, please refer to Figure 8In a typical embodiment, the device includes: a first category annotation module 11, which is used to respond to the category annotation event acting on the commodity query word samples and the commodity title samples in the candidate sample pool, and obtain the first labeled commodity category labeled by the event for the commodity query word samples and the commodity title samples; a second category annotation module 12, which is used to call the first sample category annotation model that has been trained to convergence, and annotate the second labeled commodity category corresponding to the commodity query word samples and the commodity title samples with the first labeled commodity category; a sample storage module 13, which is used to push the commodity query word samples or the commodity title samples with the same first labeled commodity category and the second labeled commodity category to the training sample library, and drive the second sample category annotation model that has not been trained to convergence to call the training sample library for training; a model updating module 14, which is used to intervene in the first sample category annotation model for updating according to the second sample category annotation model that has been trained to convergence based on the training sample library.

[0137] In one embodiment, the first category annotation module 11 includes: a sample pushing submodule, which is used to obtain any multiple product query word samples and product title samples stored in the candidate sample pool and push them to the annotation end according to a preset sample scattering rule; a sample storage submodule, which is used to obtain the first annotated product category corresponding to each of the query word samples and product title samples acted on by the annotation end, and store these first annotated product categories and their query word samples or product title samples in the candidate sample pool in correspondence.

[0138] In one embodiment, the second category annotation module 12 includes: a sample preprocessing submodule, used to perform text preprocessing on product query word samples or product title samples; a feature vector extraction submodule, used to call a feature extraction model that has been trained to convergence, and the extraction model extracts the feature vector of the product query word sample or product title sample; a category annotation submodule, calling a category classification model that has been trained to convergence, calculating the probability that the feature vector corresponds to each product category, and taking the product category with the largest probability as the second labeled product category of the product query word sample or product title sample.

[0139] In one embodiment, the sample storage module 13 includes: a labeling verification submodule, used to determine whether the first labeled product category of the product query word sample or the product title sample is the same as its corresponding second labeled product category; a sample return submodule, used to return and store the first labeled product category and the second labeled product category of the product query word sample or the product title sample as mapping relationship data in the candidate sample pool if they are not the same; and a training sample storage submodule, used to store the product query word sample or the product title sample of the labeled product category as mapping relationship data in the training sample library if they are the same.

[0140] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the model training sample verification method. Fig. 9 , Fig. 9 This is a basic structural block diagram of the computer device in this embodiment.

[0141] like Fig. 9 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a model training sample verification method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a model training sample verification method. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In this embodiment, the processor is used to execute the specific functions of each module / submodule in the model training sample verification device of the present application, and the memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the model training sample verification device, and the server can call the program code and data of the server to execute the functions of all submodules.

[0143] The present application also provides a non-volatile storage medium, in which the model training sample verification method is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is run in the computer, thereby enabling one or more processors to execute the steps of the model training sample verification method in any of the above-mentioned embodiments.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0145] In summary, this application realizes automatic labeling and verification of training samples, and forms efficient utilization of model resources, effectively improving the reasoning ability of the model to be deployed online.

[0146] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0147] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0148] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A model training sample verification method, characterized in that: The steps include: In response to a category annotation event acting on a product query word sample and a product title sample in a candidate sample pool, obtain a first annotated product category annotated by the event for the product query word sample and the product title sample; The first sample category annotation model that has been trained to convergence is called to annotate the second annotated product category corresponding to the product query word samples and product title samples having the first annotated product category; Pushing product query word samples or product title samples of the same first labeled product category and second labeled product category to the training sample library, driving the second sample category target annotation model that has not been trained to convergence to call the training sample library for training; According to the second sample class target annotation model that is most recently trained to convergence based on the training sample library, the first sample class target annotation model is intervened to be updated.

2. The method according to claim 1, characterized in that The following pre-steps are included: Calling the active learning sample screening model to drive the active learning sample screening model to classify each product title sample in the product title pool into a product category; Obtaining commodity category classification data corresponding to each of the commodity title samples, wherein the commodity category classification data represents the classification probability of the commodity title samples and each commodity category; The commodity title samples having a classification probability exceeding a preset threshold representing any commodity category in the commodity category classification data are stored in the training sample library.

3. The method according to claim 1, characterized in that: The following pre-steps are included: Obtaining user historical behavior information, parsing one or more product query word samples and their corresponding query interaction values ​​contained in the user historical behavior information; The commodity query word samples whose query interaction values ​​exceed the preset screening value are stored in the candidate sample pool.

4. The method according to claim 1, characterized in that The step of responding to the category annotation event of the product query word sample and the product title sample in the candidate sample pool includes: According to the preset sample scattering rule, any multiple product query word samples and product title samples stored in the candidate sample pool are obtained and pushed to the annotation end; The first annotated commodity categories corresponding to the query word samples and the commodity title samples are obtained by the annotation end, and the first annotated commodity categories are stored in the candidate sample pool in correspondence with the query word samples or the commodity title samples.

5. The method according to claim 1, characterized in that The step of calling the first sample category annotation model to annotate the second annotated product category corresponding to the product query word sample and the product title sample having the first annotated product category includes: Performing text preprocessing on product query word samples or product title samples; Calling a feature extraction model that has been trained to convergence, wherein the extraction model extracts a feature vector of the product query word sample or the product title sample; The category classification model that has been trained to convergence is called to calculate the probability that the feature vector corresponds to each product category, and the product category with the highest probability is used as the second labeled product category of the product query term sample or product title sample.

6. The method according to claim 1, characterized in that The step of pushing the product query word samples or product title samples having the same first labeled product category and second labeled product category to the training sample library includes: Determine whether the first marked product category of the product query word sample or the product title sample is the same as the corresponding second marked product category; If they are not the same, the first annotated product category and the second annotated product category of the product query word sample or the product title sample are returned and stored in the candidate sample pool as mapping relationship data; If they are the same, the commodity query word samples or commodity title samples that mark the commodity category are stored in the training sample library as mapping relationship data.

7. A model training sample verification device, characterized in that: include: A first category annotation module is used to respond to the category annotation event acting on the product query word samples and product title samples in the candidate sample pool, and obtain the first annotated product category annotated by the event for these product query word samples and product title samples; The second category target annotation module is used to call the first sample category target annotation model that has been trained to convergence, and annotate the second annotated product category corresponding to the product query word samples and product title samples with the first annotated product category; A sample storage module, used to push product query word samples or product title samples of the same first labeled product category and second labeled product category to a training sample library, and drive the second sample category target annotation model that has not been trained to convergence to call the training sample library for training; The model updating module is used to intervene in the first sample class target annotation model for updating according to the second sample class target annotation model that is most recently trained to convergence based on the training sample library.

8. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 6.

9. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 6 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images