Sample information labeling method, device, equipment, program and storage medium

Abstract

The present invention provides a sample information labeling method, device, and electronic device. Related embodiments can be applied to various scenarios such as cloud technology, cloud security, and smart transportation. The method includes: obtaining positive sample information from a data source, and labeling the positive sample information to obtain labeled positive sample information; obtaining negative sample information from the data source; obtaining static statistical feature sets corresponding to the positive sample information and the negative sample information respectively; determining model parameters of the sample information labeling model; using the sample information labeling model, labeling the sample information in the data source to obtain labeled sample information; thereby saving labor costs and reducing the dependence of the quality of determining the relevance of the text to be processed on the quality of manual labeling.

Description

Technical Field

[0001] The present invention relates to information processing technology, and in particular to a sample information labeling method, device, electronic device, computer program product, and storage medium, so that the fields in which the present solution can be applied include but are not limited to autonomous driving, vehicle networking, smart transportation, and the like. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, it enables machines to have the functions of perception, reasoning and decision-making. Artificial Intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] In the related art, pre-trained language models have made breakthrough progress in machine reading comprehension tasks. The core idea is to pre-train language models on large-scale unsupervised text corpora to obtain semantic representations of texts. These semantic representations can be further applied to a series of natural language understanding tasks including machine reading comprehension in the form of features or fine-tuning, and this understanding process requires text-based relevance annotation; text relevance annotation refers to annotating text as one or more relevances in a relevance system. Text relevance annotation has a wide range of applications in a large number of business scenarios such as advertising, recommendation, and search. Determining the relevance of a text is an important part of text relevance annotation. In the traditional method of determining text relevance, the relevance of several texts is first manually annotated to obtain training samples, and then a machine learning model such as a neural network is trained based on the training samples to obtain a mapping model, and then the text to be processed is input into the mapping model, and the relevance of the text to be processed is determined by the mapping model. However, the process of manually annotating training samples consumes a lot of manpower. In addition, the mapping model is trained based on the manually annotated training samples, so the quality of determining the relevance of the text to be processed is seriously dependent on the quality of manual annotation. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a sample information annotation method, device, electronic device, software program and storage medium, which can determine the relevance of any sample information through machine learning technology, annotate the relevance of the sample information, and save the manual annotation of the relevance in the traditional method, save labor costs, reduce the dependence of the quality of determining the relevance of the text to be processed on the quality of manual annotation, improve the accuracy of text relevance annotation, and enhance the user experience.

[0005] An embodiment of the present invention provides a sample information labeling method, the method comprising:

[0006] Acquire positive sample information from a data source, and annotate the positive sample information to obtain annotated positive sample information;

[0007] Obtaining negative sample information from the data source;

[0008] Obtaining static statistical feature sets corresponding to the positive sample information and the negative sample information respectively;

[0009] Training a sample information annotation model based on the annotated positive sample information, the negative sample information, and the static statistical feature set to determine model parameters of the sample information annotation model;

[0010] The sample information in the data source is labeled using the sample information labeling model to obtain labeled sample information.

[0011] The embodiment of the present invention further provides a sample information labeling device, the device comprising:

[0012] An information transmission module, used to obtain positive sample information from a data source, and annotate the positive sample information to obtain annotated positive sample information;

[0013] An information processing module, used for obtaining negative sample information from the data source;

[0014] The information processing module is used to obtain static statistical feature sets corresponding to the positive sample information and the negative sample information respectively;

[0015] The information processing module is used to train the sample information annotation model based on the annotated positive sample information, the negative sample information and the static statistical feature set, and determine the model parameters of the sample information annotation model;

[0016] The information processing module is used to perform labeling processing on the sample information in the data source through the sample information labeling model to obtain labeled sample information.

[0017] In the above solution, the information processing module is used to randomly obtain at least one initial sample information from the data source, wherein the initial sample information at least includes: a question statement and an answer statement corresponding to the question statement;

[0018] The information processing module is used to annotate the initial sample information according to the semantic relevance between the question statement and the answer statement to obtain initial sample information with different semantic relevance;

[0019] The information processing module is used to filter positive sample information from the initial sample information of different semantic relevance levels;

[0020] The information processing module is used to label the positive example sample information based on the relevance degree of the positive example sample information to obtain labeled positive example sample information.

[0021] In the above solution, the information processing module is used to determine the screening dimension of the static statistical features in the static statistical feature set according to the type of the data source;

[0022] The information processing module is used to filter sample information from the positive sample information and the negative sample information of the data source according to the filtering dimension of the static statistical feature;

[0023] The information processing module is used to perform one-hot encoding processing on the sample information to form static statistical features corresponding to the positive sample information and the negative sample information respectively;

[0024] The information processing module is used to combine the static statistical features of different screening dimensions to form a static statistical feature set corresponding to the positive sample information and the negative sample information respectively.

[0025] In the above solution, the information processing module is used for, when the type of the data source is a search question and answer data source, the static statistical feature set includes at least one of the following:

[0026] The length, click rate and number of searches of the positive sample information, and the length, click rate and number of searches of the negative sample information;

[0027] When the type of the data source is an intelligent chat data source, the static statistical feature set includes at least one of the following:

[0028] The length and search times of the positive sample information, and the length and search times of the negative sample information.

[0029] In the above solution, the information processing module is used to obtain unclicked sample information from the data source as negative sample information; or,

[0030] The information processing module is used to filter negative sample information from initial sample information with different semantic relevance levels.

[0031] In the above solution, the information processing module is used to combine the labeled positive sample information, the negative sample information and the static statistical feature set to form a training sample set;

[0032] The information processing module is used to determine the convergence function of the sample information annotation model;

[0033] The information processing module is used to train the bidirectional attention neural network of the sample information annotation model based on the training sample set, and determine the network parameters of the bidirectional attention neural network;

[0034] The information processing module is used to train the feature encoding network of the sample information annotation model based on the training sample set, and determine the network parameters of the feature encoding network;

[0035] The information processing module is used to update the network parameters of the bidirectional attention neural network and the network parameters of the feature encoding network until the convergence conditions corresponding to the convergence function are reached.

[0036] In the above solution, the information processing module is used to obtain the sample information to be processed and the static statistical features in the data source, wherein the sample information to be processed includes: question statements and answer statements;

[0037] The information processing module is used to trigger the corresponding word segmentation library according to the text parameter information carried by the sample information to be processed;

[0038] The information processing module is used to perform word segmentation processing on the sample information to be processed through the triggered word segmentation library dictionary to form different word-level feature vectors;

[0039] The information processing module is used to perform noise removal on the different word-level feature vectors to form a word-level feature vector set corresponding to the sample information to be processed;

[0040] The information processing module is used to process the word-level feature vector set and the static statistical features through the sample information annotation model to determine the correlation between the question sentence and the answer sentence;

[0041] The information processing module is used to perform labeling processing on the sample information to be processed according to the correlation between the question statement and the answer statement, so as to obtain the correlation labeling result of the sample information to be processed.

[0042] An embodiment of the present invention further provides an electronic device, the electronic device comprising:

[0043] A memory for storing executable instructions;

[0044] The processor is used to implement the aforementioned sample information labeling method when running the executable instructions stored in the memory.

[0045] An embodiment of the present invention further provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed by a processor, the aforementioned sample information labeling method is implemented.

[0046] An embodiment of the present invention further provides a computer-readable storage medium storing executable instructions, which implement the aforementioned sample information labeling method when executed by a processor.

[0047] The embodiments of the present invention have the following beneficial effects:

[0048] The present invention obtains positive sample information from a data source and annotates the positive sample information to obtain annotated positive sample information; obtains negative sample information from the data source; obtains static statistical feature sets corresponding to the positive sample information and the negative sample information respectively; trains a sample information annotation model based on the annotated positive sample information, the negative sample information and the static statistical feature set to determine model parameters of the sample information annotation model; annotates the sample information in the data source through the sample information annotation model to obtain annotated sample information, thereby being able to determine the relevance of any sample information through machine learning technology, annotate the relevance of the sample information, and save the link of manually annotating the relevance in the traditional method, saving labor costs, reducing the dependence of the quality of determining the relevance of the text to be processed on the quality of manual annotation, improving the accuracy of text relevance annotation, and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a usage scenario of the sample information labeling method provided by an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of the composition structure of a sample information labeling device provided in an embodiment of the present invention;

[0051] Figure 3 An optional flowchart of a sample information labeling method provided in an embodiment of the present invention;

[0052] Figure 4 A schematic diagram of the structure of a sample information labeling model in an embodiment of the present invention;

[0053] Figure 5 An optional flowchart of a sample information labeling method provided in an embodiment of the present invention;

[0054] Figure 6 Schematic diagram of the application environment of the sample information labeling model in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.

[0056] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0057] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0058] 1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0059] 2) Based on, used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order of execution of the multiple operations executed.

[0060] 3) Softmax: A very common and important function in machine learning, especially in multi-classification scenarios. It maps some inputs to real numbers between 0 and 1 and normalizes them to ensure that the sum is 1.

[0061] 4) Neural Network (NN): Artificial Neural Network (ANN), referred to as neural network or quasi-neural network, is a mathematical model or computational model that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) in the field of machine learning and cognitive science, and is used to estimate or approximate functions.

[0062] 5) Clicklog: The search engine backend will record the docid and click count of each input query and clicked doc. The record format is as follows:

[0063] <query,docid> \tNumber of searches, number of clicks, and docid sorting position.

[0064] 6) Bidirectional Attention Neural Network Model (BERT Bidirectional Encoder Representationsfrom Transformers) Bidirectional Attention Neural Network Model proposed by Google.

[0065] 7) Token: Before any actual processing of the input text, it needs to be segmented into language units such as words, punctuation marks, numbers, or pure alphanumeric characters. These units are called tokens.

[0066] 8) Softmax: Normalized exponential function, which is a generalization of the logic function. It can "compress" a K-dimensional vector containing any real number into another K-dimensional real vector, so that each element is between [0,1] and the sum of all elements is 1.

[0067] Among them, the embodiments of the present invention can be implemented in combination with cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or a local area network to realize data calculation, storage, processing and sharing. It can also be understood as a general term for network technology, information technology, integration technology, management platform technology and application technology based on cloud computing business model applications. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites, so cloud technology needs to be supported by cloud computing.

[0068] Through cloud technology and the sample information annotation method provided in the present application, the target media information domain can be used as a recommendation reference, and the target media information domain can be recorded in the corresponding cloud server. When the target object browses media information in different terminals, the corresponding media information can be recommended to the target object through the target media information domain stored in the cloud server as a recommendation reference, so that the target object can obtain more accurate media information recommendation results.

[0069] It should be noted that cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called the "cloud". From the user's point of view, the resources in the "cloud" can be infinitely expanded, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a basic capability provider for cloud computing, a cloud computing resource pool platform will be established, referred to as a cloud platform, generally referred to as Infrastructure as a Service (IaaS), and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (which can be virtualized machines, including operating systems), storage devices and network devices.

[0070] Before introducing the sample information annotation method provided by this application, we first briefly explain the defects of media information recommendation in related technologies. When related technologies perform relevance annotation, the user enters a question statement (query), and the search engine background recalls a series of document collections containing the question statement information based on the pre-established index, and then sorts these candidate documents by calculating the relevance scores of the question statement and the candidate document collection, and re-sorts them by integrating ranking factors such as document quality, authority, and timeliness, and returns the final ranking results to the user. The key search ranking algorithm is the relevance ranking algorithm, which requires a lot of R&D personnel to train the model.<query,doc> Related labeled data is obtained by manual labeling by data outsourcing companies in related technologies. However, considering the high economic cost and low efficiency of manual labeling, it often leads to the failure to obtain a sufficient number of labeled samples in time during model training and development.

[0071] In order to solve the above defects in media information recommendation, this application provides a sample information annotation method. Figure 1 , Figure 1 A schematic diagram of a usage scenario of a media information processing method based on a sample information annotation model provided in an embodiment of the present invention, wherein a client of an application software with a text input function is provided on a terminal (including a terminal 10-1 and a terminal 10-2), a user can input a corresponding question statement through the provided text input client, and the text input client can also receive a corresponding text relevance annotation result and display the received text relevance annotation result to the user; the terminal is connected to a server 200 through a network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission.

[0072] As an example, the server 200 is used to deploy a sample information labeling device and train a sample information labeling model, that is, to obtain positive sample information from a data source, and label the positive sample information to obtain labeled positive sample information; to obtain negative sample information from the data source; to obtain static statistical feature sets corresponding to the positive sample information and the negative sample information, respectively; to train a sample information labeling model based on the labeled positive sample information, the negative sample information and the static statistical feature set to determine model parameters of the sample information labeling model; and to label the sample information in the data source through the sample information labeling model to obtain labeled sample information.

[0073] Among them, the sample information annotation method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0074] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] In the embodiments of the present application, the artificial intelligence software technologies mainly involved include the above-mentioned speech processing technology and machine learning. For example, it may involve speech recognition technology (Automatic Speech Recognition, ASR) in speech technology (Speech Technology), including speech signal preprocessing (Speech signal preprocessing), speech signal frequency domain analysis (Speech signal frequency analyzing), speech signal feature extraction (Speech signal feature extraction), speech signal feature matching / recognition (Speech signal feature matching / recognition), speech training (Speech training), etc.

[0076] For example, it may involve machine learning (ML), which is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning usually includes technologies such as deep learning. Deep learning includes artificial neural networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc.

[0077] It can be understood that the sample information annotation method and voice processing provided in this application can be applied to intelligent devices (Intelligent device), and the intelligent device can be any device with information display function, such as a smart terminal, a smart home device (such as a smart speaker, a smart washing machine, etc.), a smart wearable device (such as a smart watch), an in-vehicle intelligent central control system (displaying media information to users through small programs that perform different tasks) or an AI smart medical device (displaying treatment cases by displaying media information), etc.

[0078] The structure of the sample information labeling device according to the embodiment of the present invention is described in detail below. The sample information labeling device can be implemented in various forms, such as a dedicated terminal with a sample information labeling processing function, or a server provided with a sample information labeling device processing function, such as the preceding Figure 1 Server 200 in. Figure 2 The schematic diagram of the composition structure of the sample information labeling device provided by the embodiment of the present invention can be understood as follows: Figure 2 Only an exemplary structure of the sample information labeling device is shown, not all structures, and can be implemented as needed. Figure 2 Partial or complete structure shown.

[0079] The sample information labeling device provided in the embodiment of the present invention includes: at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. The various components in the sample information labeling device are coupled together through a bus system 205. It can be understood that the bus system 205 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 2 Various buses are labeled as bus system 205 .

[0080] The user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0081] It is understood that the memory 202 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The memory 202 in the embodiment of the present invention can store data to support the operation of the terminal (such as 10-1). Examples of these data include: any computer program for operating on the terminal (such as 10-1), such as an operating system and an application. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application can include various applications.

[0082] In some embodiments, the sample information labeling device provided by the embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the sample information labeling device provided by the embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method of the sample information labeling model provided by the embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0083] As an example of a sample information labeling device provided in an embodiment of the present invention being implemented by a combination of software and hardware, the sample information labeling device provided in an embodiment of the present invention can be directly embodied as a combination of software modules executed by a processor 201. The software module can be located in a storage medium, and the storage medium is located in a memory 202. The processor 201 reads the executable instructions included in the software module in the memory 202, and combines with necessary hardware (for example, including the processor 201 and other components connected to the bus 205) to complete the training method of the sample information labeling model provided in an embodiment of the present invention.

[0084] As an example, the processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0085] As an example of hardware implementation of the sample information labeling device provided in an embodiment of the present invention, the device provided in an embodiment of the present invention can be directly executed by a processor 201 in the form of a hardware decoding processor. For example, the training method of the sample information labeling model provided in an embodiment of the present invention can be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0086] The memory 202 in the embodiment of the present invention is used to store various types of data to support the operation of the sample information labeling device. Examples of such data include: any executable instructions for operating on the sample information labeling device, such as executable instructions, and the program for implementing the training method of the sample information labeling model in the embodiment of the present invention can be included in the executable instructions.

[0087] In other embodiments, the sample information labeling device provided by the embodiments of the present invention may be implemented in a software manner. Figure 2 The sample information annotation device stored in the memory 202 is shown, which may be software in the form of a program and a plug-in, and includes a series of modules. As an example of a program stored in the memory 202, a sample information annotation device may be included, and the sample information annotation device includes the following software modules:

[0088] Information transmission module 2081 and information processing module 2082. When the software modules in the sample information labeling device are read into the RAM by the processor 201 and executed, the training method of the sample information labeling model provided by the embodiment of the present invention will be implemented, wherein the functions of each software module in the sample information labeling device include:

[0089] The information transmission module 2081 is used to obtain positive sample information from a data source, and to annotate the positive sample information to obtain annotated positive sample information.

[0090] The information processing module 2082 is used to obtain negative sample information from the data source.

[0091] The information processing module 2082 is used to obtain static statistical feature sets corresponding to the positive sample information and the negative sample information respectively.

[0092] The information processing module 2082 is used to train the sample information annotation model based on the annotated positive sample information, the negative sample information and the static statistical feature set, and determine the model parameters of the sample information annotation model.

[0093] The information processing module 2082 is used to perform labeling processing on the sample information in the data source through the sample information labeling model to obtain labeled sample information.

[0094] according to Figure 2 The electronic device shown, in one aspect of the present application, the present application also provides a computer program product or a computer program, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the electronic device executes the different embodiments and combinations of embodiments provided in various optional implementations of the above-mentioned sample information labeling method.

[0095] Combination Figure 2 The sample information labeling device shown in the figure illustrates the sample information labeling method provided by the embodiment of the present invention, see Figure 3 , Figure 3 An optional flow chart of the sample information labeling method provided in an embodiment of the present invention is provided. It can be understood that: Figure 3 The steps shown can be performed by various electronic devices running the sample information labeling device, for example, a dedicated terminal or server with a sample information labeling device. Figure 3 The steps shown are explained.

[0096] Step 301: The sample information labeling device obtains positive sample information from a data source, and labels the positive sample information to obtain labeled positive sample information.

[0097] In some embodiments of the present invention, obtaining positive sample information from a data source and labeling the positive sample information to obtain labeled positive sample information can be achieved in the following manner:

[0098] Randomly obtain at least one initial sample information from the data source, wherein the initial sample information at least includes: a question statement and a reply statement corresponding to the question statement; annotate the initial sample information according to the semantic relevance between the question statement and the reply statement to obtain initial sample information with different semantic relevance; filter positive sample information from the initial sample information with different semantic relevance; annotate the positive sample information based on the relevance of the positive sample information to obtain annotated positive sample information. The semantic relevance between the question statement query and the reply statement doc can include 5 different levels, namely:

[0099] 5 (Excellent results can be prioritized) ---- The text semantics completely matches and fully meets the main needs of the user's search query. 4 (Satisfactory results can be prioritized) ---- The text semantics highly matches and highly meets the user's search query needs. 3 (Qualified results are ranked lower than 4 and 5) ---- The text is related or partially related, the semantics are partially related, and it meets part or a small part of the search needs, and the semantics do not drift. 2 (General results are ranked at the back or not shown) ---- The text is related or partially related, and it is in the same rough field as the search needs, but the semantics drift and does not meet the search query intention. 1 (Poor results are not shown) ---- The text semantics are not related at all.

[0100] Among them, 1 is the lowest level and 5 is the highest level. The semantic relevance degree between the question sentence query and the answer sentence doc is 5, which means that the semantic relevance degree between the question sentence query and the answer sentence doc is the highest.

[0101] Step 302: The sample information labeling device obtains negative sample information from the data source.

[0102] In some embodiments of the present invention, obtaining negative sample information can be achieved in the following ways:

[0103] The unclicked sample information is obtained from the data source as negative sample information; or the negative sample information is screened from the initial sample information with different semantic relevance levels. The unclicked sample information obtained from the data source can be selected as negative sample information according to the preset ratio of negative samples, or the sample information of grades 1-3 can be selected as negative sample information with reference to the above embodiment to increase the speed of collecting negative sample information and save the training time of the sample labeling model.

[0104] Step 303: The sample information labeling device obtains static statistical feature sets corresponding to the positive sample information and the negative sample information respectively.

[0105] In some embodiments of the present invention, obtaining the static statistical feature sets corresponding to the positive sample information and the negative sample information respectively can be implemented in the following manner:

[0106] According to the type of the data source, determine the screening dimension of the static statistical features in the static statistical feature set; according to the screening dimension of the static statistical features, screen the sample information in the positive sample information and the negative sample information of the data source; perform one-hot encoding on the sample information to form the static statistical features corresponding to the positive sample information and the negative sample information respectively; combine the static statistical features of different screening dimensions to form the static statistical feature sets corresponding to the positive sample information and the negative sample information respectively. Among them, the process of encoding the sample information by one-hot encoding is to use an N-bit state register to encode N sample information, each state has its independent register bit, and at any time, only one of them is valid. This part can be completed with the help of sklearn.

[0107] Specifically, in order to make the different features in the static statistical feature set dense during encoding and ensure the training accuracy of the sample information standard, the following takes the number of searches for the question sentence query as an example to introduce the onehot encoding process:

[0108] When the search times of question statement query1 is 49 times, and the search times of question statement query2 is 50 times, if the value is directly used to represent the feature vector, the queries with similar search times will be completely different and sparse in this feature dimension. Moreover, because the value ranges of different feature dimensions vary greatly, for example, the real-time search times of the search applet of the instant messaging client are in the millions, if the original feature values ​​are directly used for encoding, it will cause an explosion of feature values ​​and affect the training accuracy of the sample information labeling model. Therefore, the above defects are solved through onehot encoding with interval mapping. For example: different encoding values ​​are set for the search times (i.e., query pv, abbreviated as qv) intervals of different question statements, that is, assuming that there are a total of 4 dictionaries with value ranges, refer to Table 1:

[0109]

[0110]

[0111] Table 1

[0112] According to the above table, different qv value ranges correspond to the corresponding bin positions being coded as 1, and the other bits being coded as 0. For example, for all queries whose qv in the log is less than 100 times, they belong to the 4th bin in terms of qv, that is, the 4th bit is 1, and their codes are all 0001.

[0113] Other features in the static statistical feature set can also be one-hot encoded in different grades according to the encoding method of the query search times in the above embodiment (for example, they are all divided into 4 grades, that is, the encoding length of each type of feature is a fixed length of 4), and then directly spliced. For example, the feature encoding process for a certain question statement query-A is as follows:

[0114] Query-A length onehot encoding: 0010; Query-A search times onehot encoding: 0001; Query-A click rate feature onehot encoding: 0010. Combining the above onehot encodings, we can see that the feature encoding of query-A can be expressed as:

[0010]

[0001]

[0010] .

[0115] Continue to encode the features of a reply statement doc-A clicked under Query-A by onehot encoding, where the length of Doc-A is onehot encoded: 0100; the number of clicks on Doc-A is onehot encoded: 1000; therefore, the feature encoding of Doc-A can be expressed as:

[0100]

[1000] ; then the feature encodings of Query-A and Doc-A are concatenated, that is:

[0010]

[0001]

[0010]

[0100]

[1000] . It should be noted that numerical attributes can be directly used as calculated features, while non-numerical attributes need to be converted into computable numerical features first. For example, if there are 10 categories of samples, the 10 categories are numbered 1-10 at a time and converted into a one-hot sparse vector for feature calculation. For example, if sample A belongs to the third category among the 10 categories, then the feature of the sample class attribute is [0, 0, 1, 0, 0, 0, 0, 0, 0].

[0116] In some embodiments of the present invention, when the type of the data source is a search question and answer data source, the static statistical feature set includes at least one of the following: the length, click-through rate, and number of searches of the positive sample information, and the length, click-through rate, and number of searches of the negative sample information.

[0117] In some embodiments of the present invention, when the type of the data source is an intelligent chat data source, the static statistical feature set includes at least one of the following: the length and search times of the positive sample information, and the length and search times of the negative sample information. Thus, according to the type of data source, the user can flexibly adjust the static statistical feature type in the static statistical feature set to reduce the training time of the sample annotation model and facilitate the rapid deployment of the sample annotation model.

[0118] Step 304: The sample information labeling device trains a sample information labeling model based on the labeled positive sample information, the negative sample information and the static statistical feature set to determine model parameters of the sample information labeling model.

[0119] refer to Figure 4 , Figure 4 It is a structural diagram of the sample information labeling model in the embodiment of the present invention, wherein the sample information labeling model includes: a network of a bidirectional attention neural network and a feature encoding network, during training, the labeled positive sample information, the negative sample information and the static statistical feature set are combined to form a training sample set; the convergence function of the sample information labeling model is determined; based on the training sample set, the bidirectional attention neural network of the sample information labeling model is trained to determine the network parameters of the bidirectional attention neural network; based on the training sample set, the feature encoding network of the sample information labeling model is trained to determine the network parameters of the feature encoding network; the network parameters of the bidirectional attention neural network and the network parameters of the feature encoding network are updated until the convergence condition corresponding to the convergence function is reached. Figure 4 As shown in the figure, the text semantic vectors of the question sentence query and the answer sentence doc are extracted through the network of the bidirectional attention neural network, and the CLS vector is obtained as the semantic feature vector of the text. Then, the pooling layer is connected to extract the sentence feature vectors u and v of the query and doc. The difference vector of the vectors u and v is calculated through the cross features of u and v, i.e. uv, and concatenated with u and v to form the similarity calculation vector representation corresponding to the query and doc. After that, it is concatenated with the output result of the feature encoding network to finally form the annotation of the relevance of any sample.

[0120] In some embodiments of the present invention, after the sample information annotation model is trained, for a certain high click rate<query,doc> Yes, if the relevance classification automatically marked by the sample information labeling model is very low (i.e. irrelevant), it may mean that<query,doc> The high click rate of the<query,doc> The sample information is used as the training and use of other sample information labeling models.

[0121] In some embodiments of the present invention, when statically counting features, various types of user behavior data matched by the corresponding client can be collected through different program components, and the original log of the user behavior data can be effectively extracted, such as the user's device number (user account), media information type information, media information browsing time, and media information browsing completeness parameters. Among them, the user's historical click behavior and the browsing time of the corresponding information will be recorded through the subscription service and stored in Redis, and the online recommendation system will pull the corresponding user's historical click behavior when the user request arrives.

[0122] Step 305: The sample information labeling device labels the sample information in the data source through the sample information labeling model to obtain labeled sample information.

[0123] See also Figure 5 , Figure 5 An optional flow chart of the sample information labeling method provided in an embodiment of the present invention is provided. It can be understood that: Figure 5 The steps shown can be performed by various electronic devices running the sample correlation annotation device, such as a dedicated terminal with a sample correlation annotation function, a server or a server cluster with a sample processing function. Figure 5 The steps shown are explained.

[0124] Step 501: Obtain the sample information to be processed and static statistical features in the data source, wherein the sample information to be processed includes: question statements and answer statements.

[0125] Step 502: triggering a corresponding word segmentation library according to the text parameter information carried by the sample information to be processed.

[0126] Step 503: Perform word segmentation processing on the sample information to be processed through the triggered word segmentation library dictionary to form different word-level feature vectors.

[0127] In some embodiments of the present invention, the language habits and operation habits of different users are not the same. For different users, different word segmentation methods need to be adjusted to adapt to the language habits of different users. Especially for Chinese, Chinese characters are used as the basic ideographic units, while the smallest semantic units with real meanings are words. Since there is no space between words like in English words to separate them, in a sentence of text, it is not certain which characters form words. Therefore, word segmentation of Chinese text is an important task. Moreover, for text processing instruction text, it contains some things that are only valuable for natural language understanding. For the sample information annotation model, to query relevant content, it is necessary to determine which are the truly valuable retrieval bases. Therefore, through the word segmentation process shown in steps 502 - 503, a set of word-level feature vectors corresponding to the text can be formed, avoiding meaningless word-level feature vectors in the set of word-level feature vectors, such as "de", "di", and "de".

[0128] Step 504: Denoise the different word-level feature vectors to form a set of word-level feature vectors corresponding to the sample information to be processed.

[0129] In some embodiments of the present invention, a dynamic noise threshold matching the usage environment of the sample information annotation model can be determined; denoise the problem text set according to the dynamic noise threshold, and trigger a dynamic word segmentation strategy matching the dynamic noise threshold; perform word segmentation on the problem text according to the dynamic word segmentation strategy matching the dynamic noise threshold to form a set of dynamic word-level feature vectors corresponding to the problem text. Among them, due to the different usage environments of the sample information annotation model (for example, the object using the corpus), the dynamic noise thresholds matching the usage environments of the sample information annotation model are also different. For example, in the usage environment of academic translation, the problem text and the corresponding reply sentences displayed on the terminal only include the problem text and the corresponding reply sentences of academic papers. The dynamic noise threshold matching the usage environment of the sample information annotation model needs to be less than the dynamic noise threshold in the reading environment of entertainment information text.

[0130] Determine a fixed noise threshold corresponding to the use environment of the sample information annotation model; perform denoising on the question text set according to the fixed noise threshold, and trigger a fixed word segmentation strategy that matches the fixed noise threshold; perform word segmentation on the question text according to the fixed word segmentation strategy that matches the fixed noise threshold, and obtain a fixed word-level feature vector set corresponding to the question text. Among them, when the sample information annotation model is solidified in the corresponding hardware mechanism, such as an in-vehicle terminal or a smart medical system, and the use environment is professional terminology text information (or text information in a certain field), since the noise is relatively single, by fixing the fixed noise threshold corresponding to the sample information annotation model, the processing speed of the sample information annotation model can be effectively improved, the user's waiting time can be reduced, and the user's experience can be improved.

[0131] Step 505: Process the word-level feature vector set and the static statistical features through the sample information annotation model to determine the correlation between the question sentence and the answer sentence.

[0132] Step 506: Based on the correlation between the question statement and the answer statement, the sample information to be processed is annotated to obtain a correlation labeling result of the sample information to be processed.

[0133] The chat corpus annotation method provided by the embodiment of the present invention is described below using a chat corpus annotation model encapsulated in a WeChat applet, wherein: Figure 6 FIG. 1 is a schematic diagram of an application environment of a sample information annotation model in an embodiment of the present invention, wherein Figure 6 As shown, with the development of human-computer interaction technology, more and more intelligent products based on human-computer interaction technology have emerged. For example, when entering a question statement through the search applet provided by the instant messaging client, the sample information annotation method provided by this application can be used to annotate the sample information in the data source, and the annotated sample information can be used to generate corresponding answer information. In this process, the sample information annotation method provided by this application is used to determine the relevance of any sample information, annotate the relevance of the sample information, and save labor costs. When the question text entered by the user is query = "A company financial report", for example, the corresponding reply statement refers to Table 2:

[0134] Table 2

[0135]

[0136] By using the sample information annotation model provided in this application, only reply statements with relevance levels 5 and 4 can be provided, so that users can obtain reply statements that better meet their needs.

[0137] Beneficial technical effects:

[0138] The present invention obtains positive sample information from a data source and annotates the positive sample information to obtain annotated positive sample information; obtains negative sample information from the data source; obtains static statistical feature sets corresponding to the positive sample information and the negative sample information respectively; trains a sample information annotation model based on the annotated positive sample information, the negative sample information and the static statistical feature set to determine model parameters of the sample information annotation model; annotates the sample information in the data source through the sample information annotation model to obtain annotated sample information, thereby being able to determine the relevance of any sample information through machine learning technology, annotate the relevance of the sample information, and save the link of manually annotating the relevance in the traditional method, saving labor costs, reducing the dependence of the quality of determining the relevance of the text to be processed on the quality of manual annotation, improving the accuracy of text relevance annotation, and improving the user experience.

[0139] The above descriptions are merely embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sample information labeling method, characterized in that: The method comprises: Acquire positive sample information from a data source, and annotate the positive sample information to obtain annotated positive sample information; Obtaining negative sample information from the data source; Determining, according to the type of the data source, a screening dimension of the static statistical features in the static statistical feature set; Filtering sample information from the positive sample information and the negative sample information of the data source according to the filtering dimension of the static statistical feature; Performing one-hot encoding processing on the sample information to form static statistical features corresponding to the positive sample information and the negative sample information respectively; Combining the static statistical features of different screening dimensions to form static statistical feature sets corresponding to the positive sample information and the negative sample information respectively; Training a sample information annotation model based on the annotated positive sample information, the negative sample information, and the static statistical feature set to determine model parameters of the sample information annotation model; The sample information in the data source is labeled using the sample information labeling model to obtain labeled sample information.

2. The method according to claim 1, characterized in that The obtaining of positive sample information from a data source and labeling of the positive sample information to obtain labeled positive sample information includes: Randomly acquiring at least one initial sample information from the data source, wherein the initial sample information at least includes: a question statement and an answer statement corresponding to the question statement; Annotating the initial sample information according to the semantic relevance between the question sentence and the answer sentence to obtain initial sample information with different semantic relevance; Screening positive sample information from the initial sample information of different semantic relevance levels; Based on the relevance degree of the positive example sample information, the positive example sample information is labeled to obtain labeled positive example sample information.

3. The method according to claim 1, characterized in that The method further comprises: When the type of the data source is a search question and answer data source, the static statistical feature set includes at least one of the following: The length, click rate and number of searches of the positive sample information, and the length, click rate and number of searches of the negative sample information; When the type of the data source is an intelligent chat data source, the static statistical feature set includes at least one of the following: The length and search times of the positive sample information, and the length and search times of the negative sample information.

4. The method according to claim 1, characterized in that: The obtaining negative sample information from the data source includes: Acquire unclicked sample information from the data source as negative sample information; or, Filter negative sample information from initial sample information with different degrees of semantic relevance.

5. The method according to claim 1, characterized in that The training of the sample information annotation model based on the annotated positive sample information, the negative sample information and the static statistical feature set to determine the model parameters of the sample information annotation model includes: Combining the labeled positive sample information, the negative sample information, and the static statistical feature set to form a training sample set; Determining a convergence function of the sample information labeling model; Based on the training sample set, training the bidirectional attention neural network of the sample information labeling model to determine the network parameters of the bidirectional attention neural network; Based on the training sample set, training the feature encoding network of the sample information annotation model to determine the network parameters of the feature encoding network; The network parameters of the bidirectional attention neural network and the network parameters of the feature encoding network are updated until the convergence condition corresponding to the convergence function is reached.

6. The method according to claim 1, characterized in that The step of labeling the sample information in the data source by using the sample information labeling model to obtain labeled sample information includes: Acquire the sample information to be processed and static statistical features in the data source, wherein the sample information to be processed includes: question statements and answer statements; According to the text parameter information carried by the sample information to be processed, trigger the corresponding word segmentation library; Performing word segmentation processing on the sample information to be processed by using the triggered word segmentation library dictionary to form different word-level feature vectors; Performing denoising processing on the different word-level feature vectors to form a word-level feature vector set corresponding to the sample information to be processed; Processing the word-level feature vector set and the static statistical features through the sample information annotation model to determine the correlation between the question sentence and the answer sentence; According to the relevance between the question statement and the answer statement, the sample information to be processed is labeled to obtain a relevance labeling result of the sample information to be processed.

7. A sample information labeling device, characterized in that: The device comprises: An information transmission module, used to obtain positive sample information from a data source, and annotate the positive sample information to obtain annotated positive sample information; An information processing module, used for obtaining negative sample information from the data source; The information processing module is used to determine the screening dimension of the static statistical features in the static statistical feature set according to the type of the data source; screen the sample information in the positive sample information and the negative sample information of the data source according to the screening dimension of the static statistical features; perform one-hot encoding processing on the sample information to form the static statistical features corresponding to the positive sample information and the negative sample information respectively; combine the static statistical features of different screening dimensions to form the static statistical feature sets corresponding to the positive sample information and the negative sample information respectively; The information processing module is used to train the sample information annotation model based on the annotated positive sample information, the negative sample information and the static statistical feature set, and determine the model parameters of the sample information annotation model; The information processing module is used to perform labeling processing on the sample information in the data source through the sample information labeling model to obtain labeled sample information.

8. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the sample information labeling method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor is used to implement the sample information labeling method described in any one of claims 1 to 6 when running the executable instructions stored in the memory.

10. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by the processor, the sample information labeling method according to any one of claims 1 to 6 is implemented.

Images