A text classification method, device, electronic device and storage medium

Abstract

The present application relates to the field of computer technology, and in particular to a text classification method, device, electronic device and storage medium for quickly and accurately labeling samples. The method includes: obtaining multiple texts to be labeled, and determining the initial labeling information of each of the texts to be labeled according to the category of the corresponding search results and the relevant feedback behavior information; constructing multiple query samples and multiple reference samples based on the texts to be labeled and the initial labeling information; determining the target labeling information of each query sample based on the initial labeling information of the query samples and the reference samples; constructing a training sample data set based on the query samples and the respective target labeling information, and training a text classification model. Since the present application adjusts the initial labeling information of the corresponding query sample through the reference sample, and constructs the training sample data set based on the adjusted query sample, it can improve the accuracy of sample labeling, thereby improving the accuracy of text classification.

Description

Background Art

[0002] At present, text classification tasks based on deep learning models rely on a large number of high-quality labeled samples. However, in the early stages of text classification tasks, the time and labor costs of obtaining sufficient and high-quality training samples are often very high. In the cold start phase of the project, there is not enough labeled data, which limits the upper limit of the model fitting ability and the speed of model iteration.

[0003] In related technologies, there are usually two ways to deal with cold start. One is to manually annotate the text, but manual annotation is often costly and has a long annotation cycle. The other is to use small sample learning methods, but small sample learning relies on the generalization performance of the model and the results are relatively uncontrollable.

[0004] Therefore, in the cold start phase of text classification tasks, how to quickly and accurately label samples is a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a text classification method, device, electronic device and storage medium to improve the accuracy of sample annotation, thereby improving the accuracy of text classification.

[0006] A text classification method provided in an embodiment of the present application includes:

[0007] Acquire multiple texts to be annotated, and determine initial annotation information of each of the texts to be annotated according to the categories of the search results and related feedback behavior information corresponding to each of the texts to be annotated;

[0008] Based on the texts to be annotated and their respective initial annotation information, a plurality of query samples and a plurality of reference samples are constructed, wherein the reference samples are used to adjust the initial annotation information of the corresponding query samples;

[0009] Determining target labeling information of each query sample based on initial labeling information of each query sample and each reference sample;

[0010] A training sample data set is constructed based on each query sample and respective target annotation information, and a text classification model is trained based on the training sample data set, wherein the text classification model is used to classify the text to be classified.

[0011] An embodiment of the present application provides a text classification device, comprising:

[0012] An acquisition unit, used for acquiring a plurality of texts to be annotated, and determining initial annotation information of each of the texts to be annotated according to the categories of the search results and related feedback behavior information corresponding to each of the texts to be annotated;

[0013] A construction unit, configured to construct a plurality of query samples and a plurality of reference samples based on the texts to be annotated and their respective initial annotation information, wherein the reference samples are used to adjust the initial annotation information of the corresponding query samples;

[0014] A determination unit, configured to determine target labeling information of each query sample based on initial labeling information of each query sample and each reference sample;

[0015] The training unit is used to construct a training sample data set based on the query samples and their respective target annotation information, and to train a text classification model based on the training sample data set, wherein the text classification model is used to classify the text to be classified.

[0016] Optionally, the initial annotation information includes: a first probability that the corresponding text to be annotated corresponds to each category, and the feedback behavior information includes the number of clicks and exposures of each search result; the acquisition unit is specifically used to:

[0017] For each text to be annotated, perform the following operations:

[0018] Determine the exposure rate and click rate of each category corresponding to the text to be annotated based on the total number of exposures and the total number of clicks of the search results of the same category corresponding to the text to be annotated;

[0019] Based on the exposure rate and click rate of the one to-be-annotated text corresponding to each category, a first probability that the one to-be-annotated text corresponds to each category is determined.

[0020] Optionally, the acquiring unit is specifically used for:

[0021] Determine the exposure rate of each category corresponding to the text to be annotated based on the total exposure times of search results of the same category corresponding to the text to be annotated and the total exposure times of all search results corresponding to the text to be annotated;

[0022] Based on the total number of clicks on search results of the same category corresponding to a text to be annotated and the total number of clicks on all search results corresponding to the texts to be annotated, the click rates of the categories corresponding to the text to be annotated are determined.

[0023] Optionally, the determining unit is specifically configured to:

[0024] Performing character segmentation on each query sample and each reference sample according to a preset character segmentation method to obtain query sample characters contained in each query sample and reference sample characters contained in each reference sample;

[0025] Based on the query sample characters respectively included in each query sample and the reference sample characters respectively included in each reference sample, obtaining the query sample vectors respectively corresponding to the query samples and the reference sample vectors respectively corresponding to the reference samples;

[0026] Based on the similarities between each query sample vector and each reference sample vector, the reference samples corresponding to each query sample are obtained.

[0027] Optionally, the determining unit is specifically configured to:

[0028] Encoding the query sample characters contained in each query sample and the reference sample characters contained in each reference sample based on the trained language model to obtain at least one query sample character vector corresponding to each query sample and at least one reference sample character vector corresponding to each reference sample;

[0029] Based on at least one query sample character vector corresponding to each query sample, a query sample vector corresponding to each query sample is determined, and based on at least one reference sample character vector corresponding to each reference sample, a reference sample vector corresponding to each reference sample is determined.

[0030] Optionally, each to-be-segmented text is segmented according to at least one of the following preset character segmentation methods, where the to-be-segmented text is a reference sample or a query sample:

[0031] For each query sample, perform the following operations:

[0032] Perform character segmentation on a text to be segmented according to a preset character granularity;

[0033] The text to be segmented is segmented by taking entity words as whole characters and non-entity words according to the preset character granularity.

[0034] Optionally, each query sample corresponds to at least one query sample vector, each reference sample corresponds to at least one reference sample vector, and the determining unit is specifically configured to:

[0035] Based on the character segmentation method corresponding to each query sample vector, select reference sample vectors with the same character segmentation method as each query sample vector;

[0036] Based on the similarities between each query sample vector and each corresponding reference sample vector, the reference samples corresponding to each query sample are obtained.

[0037] Optionally, obtain a trained language model by:

[0038] Acquire at least one corpus consisting of texts to be evaluated from historical data, and segment the corpus into character sets according to a preset character segmentation method;

[0039] The language model to be trained is trained based on the character set to obtain a trained language model.

[0040] Optionally, the initial annotation information includes: a first probability that the corresponding text to be annotated corresponds to each category; and the determination unit is specifically configured to:

[0041] For each query sample, perform the following operations:

[0042] Based on the first probabilities of the reference samples corresponding to a query sample and the similarity between the corresponding reference samples and the query sample, determine the second probabilities of the query sample corresponding to each category, the query sample corresponding to at least one second probability;

[0043] Based on the at least one second probability and the first probability of the query sample, target labeling information of the query sample is determined.

[0044] Optionally, the determining unit is specifically configured to:

[0045] Determine, based on the sum of the at least one second probability and the first probability of the one query sample, a third probability of the category of each search result corresponding to the query sample;

[0046] The category corresponding to the highest third probability is used as the target labeling information of the query sample.

[0047] Optionally, the text classification model includes at least one text binary classification model, and the training sample data set of each text binary classification model includes a positive sample set and a negative sample set; the training unit is specifically used for:

[0048] Taking the query samples whose target annotation information is the target category in the query samples as the positive sample set;

[0049] According to the stratified sampling method, among various query samples whose target annotation information in the query samples is a non-target category, a preset proportion of query samples are respectively extracted to form a negative sample set.

[0050] An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned text classification methods.

[0051] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of any one of the above-mentioned text classification methods.

[0052] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned text classification methods.

[0053] The beneficial effects of this application are as follows:

[0054] The embodiments of the present application provide a text classification method, device, electronic device and storage medium. Specifically, first, a plurality of texts to be annotated are obtained, and the initial annotation information of each text to be annotated is determined according to the category of each search result corresponding to each text to be annotated and the related feedback behavior information, so that the category distribution of the text to be annotated can be preliminarily obtained according to the category of the search result corresponding to the text to be annotated; then, based on each text to be annotated and their respective initial annotation information, a plurality of query samples and a plurality of reference samples are constructed, and based on the initial annotation information of each query sample and each reference sample, the target annotation information of each query sample is determined, and the initial annotation information of the query sample can be adjusted based on the initial annotation information of the reference sample to improve the accuracy of the query sample annotation; finally, a training sample data set is constructed based on each query sample and their respective target annotation information, and a text classification model is trained based on the training sample data set. Based on the above method, rapid sample automatic annotation can be achieved, the sample annotation accuracy can be improved, and then the text classification model is trained based on the generated training samples, which can improve the text classification accuracy.

[0055] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0057] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0058] Figure 2 is a flowchart of an implementation method of a text classification method in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a search result corresponding to a text to be evaluated in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of generating a query sample vector in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of generating a reference sample vector in an embodiment of the present application;

[0062] Figure 6 A schematic diagram of a corresponding relationship between a query sample and a reference sample in an embodiment of the present application;

[0063] Figure 7 A specific flow chart of a text classification method in an embodiment of the present application;

[0064] Fig. 8A A logical schematic diagram of a text classification method in an embodiment of the present application;

[0065] Figure 8B A schematic diagram of a labeling information adjustment process in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of the structure of a text classification device in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0068] Fig.11 The present invention is a schematic diagram of the hardware structure of another electronic device to which the embodiments of the present application are applied. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0070] The following is an introduction to some concepts involved in the embodiments of the present application.

[0071] Text classification: It is the process of associating a given text with one or more categories based on the characteristics (content or attributes) of the text under a predefined classification system. It is a research topic that closely combines pattern recognition and natural language processing. The effectiveness of text classification depends on a large number of high-quality annotated samples. The quality of the training samples determines the upper limit of the model's fitting ability and the speed of model iteration.

[0072] Reference samples: A set of samples that are initially labeled using some means (such as object search behavior). The annotation information of this sample set has a certain degree of accuracy, but there is still a certain degree of error, and the influence of noise needs to be further removed.

[0073] Query samples: samples that need to obtain annotation information. Query samples initiate queries to reference samples, adjust their own annotation information based on the annotation information of reference samples, and build training sample datasets based on the adjusted query samples to train text classification models.

[0074] Faiss (Facebook AI Similarity Search): is a library for efficient similarity search and dense vector clustering. It provides efficient similarity search and clustering for dense vectors, supports searching billions of vectors, and is an approximate nearest neighbor search library. It contains multiple algorithms for searching vector sets of arbitrary size, as well as computer programs for algorithm evaluation and parameter adjustment. The size of the vector set is determined by the random-access memory (RAM) memory.

[0075] The embodiments of the present application relate to artificial intelligence (AI), natural language processing (NLP) and machine learning (ML) technology, and are designed based on computer vision technology and machine learning in artificial intelligence.

[0076] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0077] Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology mainly includes computer vision technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions. With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, robots, smart medical care, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0078] Natural language processing is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0079] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Compared with data mining, which seeks mutual characteristics between big data, machine learning focuses more on algorithm design, allowing computers to automatically "learn" rules from data and use the rules to predict unknown data.

[0080] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are in all fields of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning. The machine translation quality assessment model in the embodiment of the present application is obtained by training with machine learning or deep learning technology. Based on the text classification method in the embodiment of the present application, the accuracy of the machine translation quality assessment model can be improved.

[0081] The following is a brief introduction to the design concept of the embodiment of the present application:

[0082] At present, text classification tasks based on deep learning models rely on a large number of high-quality labeled samples. However, in the early stages of text classification tasks, the time and labor costs of obtaining sufficient and high-quality training samples are often very high. In the cold start phase of the project, there is not enough labeled data, which limits the upper limit of the model fitting ability and the speed of model iteration.

[0083] In related technologies, there are usually two ways to handle cold start. One is to manually annotate text, but manual annotation is often costly and takes a long time. The other is to use small sample learning methods, but small sample learning relies on the generalization performance of the model and the results are relatively uncontrollable. Therefore, in the cold start stage of text classification tasks, how to quickly and accurately annotate samples is a problem that needs to be solved urgently.

[0084] In view of this, the embodiment of the present application provides a text classification method, device, electronic device and storage medium. First, multiple texts to be annotated are obtained, and the initial annotation information of each text to be annotated is determined according to the category of each search result corresponding to each text to be annotated and the related feedback behavior information. The category distribution of the text to be annotated can be preliminarily obtained according to the search result category corresponding to the text to be annotated; then, multiple query samples and multiple reference samples are constructed based on each text to be annotated and their initial annotation information, and the target annotation information of each query sample is determined based on the initial annotation information of each query sample and each reference sample. The initial annotation information of the query sample can be adjusted based on the initial annotation information of the reference sample to improve the accuracy of the query sample annotation; finally, a training sample data set is constructed based on each query sample and their target annotation information, and a text classification model is trained based on the training sample data set. Based on the above method, fast sample automatic annotation can be achieved, the sample annotation accuracy can be improved, and then the text classification model is trained based on the generated training samples, which can improve the text classification accuracy.

[0085] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0086] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0087] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, car terminals and other devices; a text classification-related client can be installed on the terminal device, which can be software (such as a browser, video software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for text classification, which is not specifically limited in the present application. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0088] It should be noted that the method in each embodiment of the present application can be executed by an electronic device, and the electronic device can be a server 120 or a terminal device 110, that is, the method can be executed by the server 120 or the terminal device 110 alone, or can be executed by the server 120 and the terminal device 110 together. For example, when executed by the server 120 alone, the server 120 obtains multiple texts to be annotated, and determines the initial annotation information of each text to be annotated according to the category of each search result corresponding to each text to be annotated and the related feedback behavior information; constructs multiple query samples and multiple reference samples based on each text to be annotated and their initial annotation information, and determines the target annotation information of each query sample based on the initial annotation information of each query sample and each reference sample; constructs a training sample data set based on each query sample and their target annotation information, and trains a text classification model based on the training sample data set.

[0089] It should be noted that different training samples are used in different text classification scenarios. Taking the scenario of browser search keyword classification as an example, the text to be evaluated is the search keyword, and the target annotation information in the training sample can be categories such as TV series and novels; in the scenario of news text classification, the text to be evaluated is news text, and the target annotation information in the training sample can be categories such as technology and entertainment.

[0090] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0091] In an optional implementation, the communication network is a wired network or a wireless network.

[0092] It should be noted that Figure 1The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0093] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the text classification method disclosed in the embodiment of the present application, the data involved, such as the text to be evaluated, query samples, reference samples, etc., can be stored on the blockchain.

[0094] In addition, the embodiments of the present application can be applied to various scenarios, including not only text classification scenarios, but also scenarios including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. In these scenarios, some text classification-related applications may be involved. For example, in the smart transportation scenario, users can classify the location text entered by users in the map navigation application, classify popular locations and non-popular locations, and provide users with more targeted choices.

[0095] It is understandable that in the specific implementation of this application, related data such as user information is involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0096] The text classification method provided by the exemplary embodiment of the present application is described below in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0097] See also Figure 2 As shown, it is an implementation flow chart of a text classification method provided in an embodiment of the present application. Taking the execution subject as a server as an example, the specific implementation process of the method includes the following steps S21-S24:

[0098] S21: The server obtains a plurality of texts to be annotated, and determines initial annotation information of each text to be annotated according to the categories of search results corresponding to each text to be annotated and related feedback behavior information;

[0099] Among them, the text to be annotated can be different types of texts such as news text and translation text, which are not specifically limited here. This application is mainly explained by taking the text to be annotated as the search keyword as an example. For example, the search keyword can be "2020 All-Star Game", "Little Vegetable Grower", etc. The search result corresponding to the text to be annotated can be the corresponding video under each search keyword. The feedback behavior information includes the number of clicks and exposures of each search result, and the initial annotation information includes the first probability of each category corresponding to the corresponding text to be annotated.

[0100] Specifically, the text 1 to be annotated is "2020 All-Star Game". A search is performed based on the 2020 All-Star Game to obtain video 1, video 2, and video 3. The category of video 1 is sports, with 1000 exposures and 700 clicks. The category of video 2 is sports, with 1200 exposures and 800 clicks. The category of video 3 is games, with 1100 exposures and 60 clicks. Based on the categories, clicks, and exposures of each video corresponding to the text 1 to be evaluated, the first probability that the text 1 to be evaluated corresponds to sports and games is determined.

[0101] S22: The server constructs multiple query samples and multiple reference samples based on the texts to be annotated and their respective initial annotation information;

[0102] The reference sample is used to adjust the initial annotation information of the corresponding query sample.

[0103] Specifically, each reference sample and each query sample consists of a text to be annotated and the corresponding initial annotation information. Both the reference sample and the query sample are derived from the user's search log. The search keywords of T days are annotated as texts to be evaluated and the initial annotation information is obtained. Multiple query samples are constructed based on the search keywords of T days and their respective initial annotation information. The search keywords of T-1 day are annotated as texts to be evaluated and the initial annotation information is obtained. Multiple reference samples are constructed based on the search keywords of T-1 day and their respective initial annotation information.

[0104] S23: The server determines the target annotation information of each query sample based on the initial annotation information of each query sample and each reference sample;

[0105] Specifically, based on the initial labeling information of each reference sample corresponding to each query sample, the initial labeling information of each query sample is adjusted to obtain the target labeling information. For example, if the reference samples corresponding to query sample 1 are reference sample 1, reference sample 2, and reference sample 3, the initial labeling information of query sample 1 is adjusted based on the initial labeling information of reference sample 1, reference sample 2, and reference sample 3 to obtain the target labeling information of query sample 1.

[0106] S24: The server constructs a training sample data set based on each query sample and respective target annotation information, and performs text classification model training based on the training sample data set.

[0107] The text classification model is used to classify the text to be classified, and the text classification model is trained based on the training data set to obtain a trained text classification model.

[0108] In an embodiment of the present application, based on the search result category corresponding to the text to be annotated, the category distribution of the text to be annotated is preliminarily obtained, and based on the initial annotation information of the reference sample, the initial annotation information of the query sample is adjusted to improve the accuracy of the query sample annotation, which can achieve rapid automatic sample annotation and improve the sample annotation accuracy. Furthermore, based on the generated training samples, the text classification model is trained to improve the text classification accuracy.

[0109] In an optional implementation, for each text to be annotated, initial annotation information of the text to be annotated is obtained in the following manner:

[0110] First, based on the total number of exposures of search results of the same category corresponding to a text to be annotated and the total number of exposures of all search results corresponding to the text to be annotated, the exposure rate of each category corresponding to the text to be annotated is determined; secondly, based on the total number of clicks of search results of the same category corresponding to the text to be annotated and the total number of clicks of all search results corresponding to the texts to be annotated, the click-through rate of each category corresponding to the text to be annotated is determined; finally, based on the exposure rate and click-through rate of the text to be annotated corresponding to each category, the first probability of the text to be annotated corresponding to each category is determined.

[0111] Specifically, Figure 3 As shown, it is a schematic diagram of search results corresponding to the text to be annotated 1. The total number of exposures of category 1 is 40, and the total number of exposures of all search results is 100. Then the exposure rate of the text to be annotated 1 corresponding to category 1 is 0.4. Similarly, the exposure rate of the text to be annotated 1 corresponding to category 2 is 0.35, and the exposure rate of the text to be annotated 1 corresponding to category 3 is 0.25; the total number of clicks of category 1 is 25, and the total number of clicks of all search results is 50. Then the exposure rate of the text to be annotated 1 corresponding to category 1 is 0.5. Similarly, the click rate of the text to be annotated 1 corresponding to category 2 is 0.4, and the click rate of the text to be annotated 1 corresponding to category 3 is 0.1; the exposure rate and click rate of each category are added together to calculate the average value to obtain the first probability. The first probability of the text to be annotated 1 corresponding to category 1 is 0.45, the first probability of the text to be annotated 1 corresponding to category 2 is 0.375, and the first probability of the text to be annotated 1 corresponding to category 3 is 0.175.

[0112] The initial annotation information can be expressed as a probability distribution form, such as: dist = {category 1: prob1, category 2: prob2, ..., category C: probC}, where C is the total number of categories, prob is the probability value that the text to be annotated belongs to this category, and the value range is [0, 1]. Then the initial annotation information of the text to be annotated 1 = {category 1: 0.45, category 2: 0.375, category 3: 0.175}.

[0113] It should be noted that the method of annotating the text to be annotated is not limited to the method based on the number of clicks and exposures in the embodiments of the present application, but can also be manual annotation, as well as other machine labeling methods, such as predicting results through a neural network model, which is not specifically limited here.

[0114] In an embodiment of the present application, the text to be annotated is preliminarily annotated based on the category of the search results corresponding to the text to be annotated, as well as the number of exposures and clicks, and the category distribution of the text to be annotated is preliminarily obtained, so that text annotation can be quickly and automatically performed at the initial stage of the text classification project, thereby alleviating the lack of annotation samples.

[0115] In an optional implementation, the user's search keywords in the past T days are counted as the text to be evaluated, the search keywords that have only been exposed but not clicked are filtered out, the text to be evaluated is automatically annotated, initial annotation information is obtained, and a query sample is constructed through the search keywords of T days and the corresponding initial annotation information; the user's search keywords in the past T-1 days are counted as the text to be evaluated, the search keywords that have only been exposed but not clicked are filtered out, the text to be evaluated is automatically annotated, initial annotation information is obtained, and a reference sample is constructed through the search keywords of T-1 days and the corresponding initial annotation information.

[0116] In an optional implementation manner, each reference sample corresponding to each query sample is obtained in the following manner:

[0117] First, character segmentation is performed on each query sample and each reference sample according to a preset character segmentation method to obtain query sample characters contained in each query sample and reference sample characters contained in each reference sample; based on the query sample characters contained in each query sample and the reference sample characters contained in each reference sample, a query sample vector corresponding to each query sample and a reference sample vector corresponding to each reference sample are obtained; based on the similarity between each query sample vector and each reference sample vector, a reference sample corresponding to each query sample is obtained.

[0118] In the embodiment of the present application, by representing the reference sample and the query sample as vectors, the reference sample corresponding to the query sample can be quickly obtained based on the similarity between the vectors.

[0119] In an optional implementation, the text to be segmented is a reference sample or a query sample, and the preset character segmentation method includes at least one of the following:

[0120] Method 1: Segment a text to be segmented according to a preset character granularity;

[0121] Among them, segmentation according to the preset character granularity means segmentation according to each character, where each Chinese character and special symbol is regarded as a character, and continuous data numbers and English are segmented according to spaces or punctuation marks. For example, the segmentation result of "2020 All-Star Game" is: [2020, All, Ming, Star, Game].

[0122] Method 2: Segment a text to be segmented by taking entity words as whole characters and non-entity words according to preset character granularity.

[0123] Among them, entity words include name categories (program names, place names, personal names, product names, etc.), attribute categories (series words, version words, etc.), and special expression phrases, etc. For example, the segmentation result of "2020 All-Star Game" is: [2020, All-Star Game].

[0124] In an embodiment of the present application, the preset character segmentation method includes segmentation according to a preset character granularity, and treating entity words as whole characters and non-entities as segmented according to a preset character opening. Character segmentation is performed on the text to be segmented from two segmentation dimensions, and segmented characters of more dimensions can be obtained, thereby improving the accuracy of the correspondence between the query sample and the reference sample.

[0125] In an optional implementation, after character segmentation is performed on the query sample and the reference sample respectively based on a preset character segmentation method, the query sample vector and the reference sample vector are obtained in the following manner:

[0126] First, based on the trained language model, query sample characters contained in each query sample and reference sample characters contained in each reference sample are encoded to obtain at least one query sample character vector corresponding to each query sample and at least one reference sample character vector corresponding to each reference sample;

[0127] Then, based on at least one query sample character vector corresponding to each query sample, a query sample vector corresponding to each query sample is determined, and based on at least one reference sample character vector corresponding to each reference sample, a reference sample vector corresponding to each reference sample is determined.

[0128] Specifically, by segmenting the query sample according to the above two character segmentation methods, characters of two segmentation methods corresponding to each query sample can be obtained. For example, query sample 1 is "Best Singer of the Year". After character segmentation, the following are obtained: segmentation character set 1 [Best, Singer, Singer of the Year] and segmentation character set 2 [Best, Singer of the Year]. The segmentation character set 1 is input into the trained language model. The language model represents each segmentation character as a D-dimensional vector, where D is the word vector dimension of each segmentation character. Then, query sample 1 can be represented as a 5×D-dimensional vector. The 5×D-dimensional vectors are added row by row and averaged to obtain a D-dimensional vector as the first sentence vector of query sample 1, i.e., the first query sample vector. The segmentation character set 2 is input into the trained language model. Query sample 1 can be represented as a 2×D-dimensional vector. The 2×D-dimensional vectors are added row by row and averaged to obtain a D-dimensional vector as the second sentence vector of query sample 1, i.e., the second query sample vector. Correspondingly, each reference sample also obtains two reference sample vectors in the above method, which will not be described in detail here.

[0129] In an embodiment of the present application, two query sample vectors corresponding to each query sample and two reference sample vectors corresponding to each reference sample are obtained based on a trained language model, which can improve the accuracy of obtaining query sample vectors and reference sample vectors.

[0130] See also Figure 4 , which is a schematic diagram of generating a query sample vector in an embodiment of the present application. First, the query sample is segmented according to two character segmentation methods to obtain segmented character sets of two segmentation dimensions. Then, the two segmented character sets are encoded through language models to obtain a character vector for each character. Finally, the character vectors are added row by row to obtain the average value to obtain two query sample vectors corresponding to the query sample.

[0131] See also Figure 5 , which is a schematic diagram of generating a reference sample vector in an embodiment of the present application. First, the reference sample is segmented according to two character segmentation methods to obtain segmented character sets of two segmentation dimensions. Then, the two segmented character sets are encoded through language models to obtain a character vector for each character. Finally, the character vectors are added row by row to obtain the average value to obtain two reference sample vectors corresponding to the reference sample.

[0132] In an optional implementation, the trained language model is obtained by:

[0133] First, a corpus consisting of at least one text to be evaluated is obtained from historical data, and the corpus is segmented into a character set according to a preset character segmentation method; then, a language model to be trained is trained based on the character set to obtain a trained language model.

[0134] Specifically, search keywords with a certain time span are obtained from user search logs to form a large-scale corpus set D. The time span can be selected as one week, half a month, etc. According to the above two preset character segmentation methods, the samples in the corpus set D are segmented into character sets, and the word2vec (word vector embedding model) of the two segmentation dimensions are trained respectively to obtain the trained language model.

[0135] It should be noted that based on the above method, two language models with segmentation dimensions are obtained. When the language model is used to encode the query sample characters or the reference sample characters, the query sample characters or the reference sample characters need to be encoded using the language model with the same segmentation dimension.

[0136] It should be noted that the language model in this application is not limited to the listed word2vec, but can also be tfidf (text frequency and inverse document frequency index), doc2vec (document embedding model), neural network feature extraction, etc., which are not specifically limited here.

[0137] In an embodiment of the present application, a corpus set of search keywords is obtained based on the user's search log, and a language model is trained based on the corpus set, so that a language model trained in two segmentation dimensions can be obtained, which can improve the encoding accuracy of the language model.

[0138] In an optional implementation, after obtaining two query sample vectors corresponding to each query sample and two reference sample vectors corresponding to each reference sample, the reference samples corresponding to each query sample are obtained in the following manner:

[0139] Firstly, based on the character segmentation method corresponding to each query sample vector, reference sample vectors with the same character segmentation method as that of each query sample vector are screened out; then, based on the similarity between each query sample vector and the corresponding reference sample vector, reference samples corresponding to each query sample are obtained.

[0140] Specifically, for the sentence vectors of the two segmentation dimensions of each query sample, based on the character segmentation method corresponding to each query sample vector, the reference sample vectors with the same character segmentation method as each query sample vector are screened out, and the nearest neighbor algorithm KNN is used to find the N reference samples with the highest similarity. The similarity measurement method between the query sample vector and the reference sample vector can be selected using cosine similarity. The similarity is a value in the range of [-1,1]. The larger the value, the more similar the vectors are. Due to the large amount of data here, the open source vector retrieval tool Faiss is used to improve the search speed.

[0141] It should be noted that the method of obtaining the reference sample corresponding to the query sample in the present application is not limited to the method using cosine similarity, and can also be other methods that can calculate the distance between vectors, which is not specifically limited here.

[0142] In the embodiment of the present application, by calculating the similarity between the query sample vector and the reference sample vector with the same character segmentation method, the correspondence between the query sample and the reference sample can be obtained, thereby improving the accuracy of the correspondence between the query sample and the reference sample.

[0143] See also Figure 6 , which is a schematic diagram of the correspondence between a query sample and a reference sample in an embodiment of the present application. After obtaining query sample vectors of two segmentation dimensions according to two character segmentation methods, based on the similarity between vectors, N reference sample vectors with the highest similarity corresponding to each query sample vector are obtained respectively, and each query sample vector has the same character segmentation method as the corresponding reference sample vector, and finally two reference samples corresponding to the query sample are obtained.

[0144] In an optional implementation, after obtaining the reference sample corresponding to the query sample, the target annotation information of each query sample is obtained in the following manner:

[0145] For each query sample, the following operations are performed respectively: based on the first probability of each reference sample corresponding to the query sample and the similarity between the corresponding reference sample and the query sample, the second probability of each category corresponding to the query sample is determined, and the query sample corresponds to at least one second probability; based on at least one second probability and the first probability of the query sample, the target labeling information of the query sample is determined.

[0146] Specifically, for the two segmentation dimension reference samples of the recalled query sample, the weighted average of the category distribution of the N reference samples of the query sample is calculated as the target labeling information of the query sample. The calculation method is as follows:

[0147]

[0148] Among them, x j is the jth query sample, q i For this purpose, the query sample is the probability value generated by combining the results of N reference samples in category i, where N is the number of corresponding reference samples, x k is the kth reference sample, s is the similarity between the query sample and the reference sample, c i is the probability value of the reference sample in category i; if the query sample and the reference sample are more similar, the category probability distribution of the reference sample will have a greater impact on the category probability distribution of the query sample. Finally, each query sample can obtain the second probability of the two segmentation dimensions.

[0149] For example, the second probability of the first segmentation dimension corresponding to query sample a is {category 1: 0.4, category 2: 0.3, category 3: 0.3}, and the second probability of the second segmentation dimension is {category 1: 0.6, category 2: 0.3, category 3: 0.1}.

[0150] In an embodiment of the present application, the initial annotation information of the query sample is adjusted based on the initial annotation information of two reference samples corresponding to the query sample and its own initial annotation information. If the similarity between the reference sample and the query sample is high, the target annotation information depends largely on the initial annotation information of the reference sample. If the similarity between the reference sample and the query sample is low, the target annotation information depends largely on the initial annotation information of the query sample itself, so that more accurate annotation information of the query sample can be obtained.

[0151] In an optional implementation, based on the sum of at least one second probability and a first probability of a query sample, a third probability of the category of each search result corresponding to the query sample is determined; and the category corresponding to the highest third probability is used as the target labeling information of the query sample.

[0152] Specifically, the second probability obtained from the two segmentation dimensions and the category probability distribution of the query sample itself (i.e., the first probability) are superimposed to obtain the third probability. The categories in the third probability finally calculated are sorted from high to low according to the probability value, and the category with the highest probability value is taken as the final label of the query sample (i.e., the target annotation information) to construct a positive and negative sample set for model training. If there are samples with a high similarity to the query sample among the N reference samples, the target annotation information of the query sample depends largely on the category distribution of these reference samples; conversely, if the similarity between the reference sample and the query sample is low, the query sample will refer more to its own annotation results.

[0153] Still taking the query sample a listed above as an example, the first probability of query sample a is {category 1: 0.5, category 2: 0.3, category 3: 0.2}, the first probability is added to the probabilities of the same category corresponding to the two second probabilities, and the third probability is obtained as {category 1: 1.5, category 2: 0.9, category 3: 0.6}, then category 1 is used as the target labeling information of query sample a.

[0154] It should be noted that the above only takes the example of obtaining the third probability by summing the first probability and at least one second probability based on the category probability. In fact, weighted averaging and other methods can also be used, which is not specifically limited here.

[0155] It should be noted that the method of adjusting the initial annotation information of the query sample based on the initial annotation information of the reference sample is not limited to the above-mentioned method based on the initial annotation information of the two reference samples and the initial annotation information of the query sample itself. The method of determining the target annotation information can also be a method based only on the initial annotation information of the reference sample, which is not specifically limited here.

[0156] In an embodiment of the present application, the category with the highest probability is used as the target annotation information of the query sample, and relatively accurate annotation information of the query sample is automatically obtained, providing data support for model training and rapid iteration of the model.

[0157] In an optional implementation, the text classification model includes at least one text binary classification model, and the training sample data set of each text binary classification model includes a positive sample set and a negative sample set. The positive sample set and the negative sample set are obtained by:

[0158] The query samples whose target annotation information in the query samples is the target category are taken as the positive sample set; according to the stratified sampling method, the query samples with preset proportions are respectively extracted from each type of query samples whose target annotation information in the query samples is the non-target category to form the negative sample set.

[0159] Specifically, it is assumed that there are C categories in the classification task. According to the text to be classified input by the user, the user's input needs to be divided into one or several of the C categories. Here, the multi-label classification problem is converted into C binary classification problems, that is, C text binary classification models are constructed, and the positive class in each binary classification model is the target class, and the negative class is the non-target class. The positive and negative samples of each binary classification model are constructed as follows: select the samples whose label information (i.e., target annotation information) in the query sample is the target class as the positive sample set, and the samples whose label information in the query sample is not the target class as the candidate set of negative samples; the candidate set of negative samples often contains multiple categories. According to the stratified sampling method, a certain proportion of samples are randomly selected from each category, and the negative samples obtained by sampling all non-target classes are merged together to form the final negative sample set. Finally, the positive and negative sample sets together constitute the training sample set for the training of the text classification model. The text classification model can select various models based on convolutional neural networks, recurrent neural networks, and encoders, such as textCNN (text classification network), which is not specifically limited here.

[0160] In an embodiment of the present application, query samples and target annotation information are obtained by automatically annotating the text to be evaluated, and a training sample data set is constructed. The query samples are input into a text classification model, and the model parameters are adjusted based on the model output results and the target annotation information to obtain a trained text classification model and improve the accuracy of text classification.

[0161] See also Figure 7 As shown, it is a specific flow chart of a text classification method in this application, taking the text to be annotated as the search keyword and the execution subject as the server as an example, specifically including the following steps S701-S708:

[0162] S701: The server obtains the search keyword to be annotated;

[0163] S702: The server obtains the video exposure result of the search result page;

[0164] S703: The server obtains category information of the search keyword based on the category, exposure times, and click times of each video;

[0165] S704: The server constructs reference samples and query samples based on the search keywords and corresponding categories;

[0166] S705: The server represents the query sample and the reference sample as feature vectors respectively;

[0167] S706: The server searches for N reference samples with the highest similarity to the query sample;

[0168] S707: The server adjusts the category information of the query sample based on the category information of the N reference samples;

[0169] S708: The server constructs a training sample data set through the adjusted query samples to train a text classification model.

[0170] In an embodiment of the present application, first, a user's search keyword text set and video category information in the user's search result exposure page are obtained, and the category distribution of the user's search keywords is calculated according to the video type, so as to preliminarily label the category of the user's search keyword; a part of the preliminarily labeled samples is used as reference samples to construct a reference sample set; the remaining preliminarily labeled samples retain the initial labeling information to form a query sample set; the samples in the pseudo-expert sample set and the query sample set are respectively represented in vector form, and for each query sample vector, the N reference samples with the highest similarity are found, and the labeling of the query sample itself is adjusted with reference to the labeling information of these N reference samples; finally, the labeling result of the query sample set is obtained, and the query sample set is used as a training sample to complete the training of the text classification model.

[0171] Based on the text classification method in this application, it is possible to classify the user's search keyword intentions, and to explore the user's potential core search needs. According to different user needs, the search engine will use different retrieval strategies and ultimately return content that meets the user's needs. In the four intent categories of "education", "music", "games", and "sports", the labeled sample training model obtained by automatic labeling using the method in this application can achieve a basic effect of an F1 score of around 0.85 on the manually evaluated test set, and can be initially put into online applications. The training samples and models accumulated in this process can serve as a relatively good iterative starting point to support subsequent model optimization. At the same time, the method proposed in this application can realize automated sample labeling, and therefore automated online model iterative updates.

[0172] See also Fig. 8A , which is a logical diagram of a text classification method in an embodiment of the present application. First, multiple texts to be annotated are input, and the texts to be annotated are preliminarily annotated based on the search results of the texts to be annotated, and multiple query samples and multiple reference samples are obtained. Taking query sample 1 "2020 All-Star Game" as an example, the corresponding initial annotation information is: first probability = {category 1: 0.5, category 2: 0.3, category 3: 0.2}, character segmentation is performed based on character segmentation method one to obtain query sample vector a, and character segmentation is performed based on character segmentation method two to obtain query sample vector b; through vector similarity, the reference sample vector 1 and reference sample vector 2 corresponding to the query sample vector a are obtained, that is, query sample 1 corresponds to reference sample 1 and reference sample 2, and the first probability of query sample 1 is adjusted based on reference sample 1 and reference sample 2 to obtain the first second probability of query sample 1 corresponding to each category = {sports: 0.4, games: 0.3, Music: 0.3}; through vector similarity, obtain reference sample vector 3 and reference sample vector 4 corresponding to query sample vector b, that is, query sample 1 corresponds to reference sample 3 and reference sample 4, adjust the first probability of query sample 1 based on reference sample 3 and reference sample 4, and obtain the second second probability of query sample 1 corresponding to each category = {sports: 0.6, games: 0.3, music: 0.1}; based on the two second probabilities corresponding to query sample 1 and its own first probability, obtain the third probability corresponding to query sample 1 = {sports: 1.5, games: 0.9, music: 0.6}, then the target annotation information of query sample 1 is sports, that is, the category corresponding to query sample 1 is sports. Based on the above method, multiple texts to be annotated can be classified, and a training sample data set can be constructed based on the query sample and the corresponding target annotation information for training a text classification model.

[0173] The embodiment of the present application provides a method for quickly and automatically obtaining sample annotation information, and preliminarily obtains the category of the search keyword based on the video type exposed by the search keyword in the search results page; however, this automated method will have some noise, so an unsupervised method is subsequently used to correct the category of the search keyword to obtain more accurate category annotation information, thereby providing data support for model training and rapid iteration of the model.

[0174] See also Figure 8B , which is a schematic diagram of a labeling information adjustment process in an embodiment of the present application. The input text to be labeled is "2021 Western Conference Finals". After preliminary labeling based on the category, exposure times and number of search results of "2021 Western Conference Finals", the initial labeling information of "2021 Western Conference Finals" is obtained as {Sports: 0.6, Game 2: 0.3, Music: 0.1}. Further, character segmentation and encoding are performed on "2021 Western Conference Finals". According to cosine similarity, the "2021 Western Conference Finals" is obtained. "The corresponding reference samples "2021 Eastern Conference Semifinals" and "World Cup Finals" are adjusted based on the initial labeling information of "2021 Eastern Conference Finals" {Sports: 0.5, Game 2: 0.3, Music: 0.2} and the initial labeling information of "World Cup Finals" {Sports: 0.4, Game 2: 0.4, Music: 0.2}, respectively, to obtain the corresponding target labeling information as sports, that is, the corresponding category of "2021 Western Conference Finals" is sports.

[0175] Based on the same inventive concept, the present application also provides a text classification device. Fig. 9 As shown, it is a schematic diagram of the structure of a text classification device 900, which may include:

[0176] The acquisition unit 901 is used to acquire a plurality of texts to be annotated, and determine initial annotation information of each text to be annotated according to the category of each search result corresponding to each text to be annotated and the related feedback behavior information;

[0177] A construction unit 902 is used to construct multiple query samples and multiple reference samples based on the texts to be annotated and their respective initial annotation information, where the reference samples are used to adjust the initial annotation information of the corresponding query samples;

[0178] A determination unit 903, configured to determine target labeling information of each query sample based on initial labeling information of each query sample and each reference sample;

[0179] The training unit 904 is used to construct a training sample data set based on each query sample and respective target annotation information, and to train a text classification model based on the training sample data set. The text classification model is used to classify the text to be classified.

[0180] Optionally, the initial annotation information includes: the first probability that the corresponding text to be annotated corresponds to each category, and the feedback behavior information includes the number of clicks and exposures of each search result; the acquisition unit 901 is specifically used to:

[0181] For each text to be annotated, perform the following operations:

[0182] Determine the exposure rate and click rate of each category of the text to be annotated based on the total number of exposures and the total number of clicks of the search results of the same category corresponding to the text to be annotated;

[0183] Based on the exposure rate and click rate of a text to be annotated corresponding to each category, a first probability that the text to be annotated corresponds to each category is determined.

[0184] Optionally, the acquiring unit 901 is specifically configured to:

[0185] Determine the exposure rate of each category corresponding to the text to be annotated based on the total exposure times of the search results of the same category corresponding to the text to be annotated and the total exposure times of all search results corresponding to the text to be annotated;

[0186] Based on the total number of clicks on search results of the same category corresponding to a text to be annotated and the total number of clicks on all search results corresponding to each text to be annotated, the click rate of each category corresponding to a text to be annotated is determined.

[0187] Optionally, the determining unit 903 is specifically configured to:

[0188] Performing character segmentation on each query sample and each reference sample according to a preset character segmentation method to obtain query sample characters contained in each query sample and reference sample characters contained in each reference sample;

[0189] Based on the query sample characters contained in each query sample and the reference sample characters contained in each reference sample, respectively, a query sample vector corresponding to each query sample and a reference sample vector corresponding to each reference sample are obtained;

[0190] Based on the similarities between each query sample vector and each reference sample vector, the reference samples corresponding to each query sample are obtained.

[0191] Optionally, the determining unit 903 is specifically configured to:

[0192] Encoding the query sample characters contained in each query sample and the reference sample characters contained in each reference sample based on the trained language model to obtain at least one query sample character vector corresponding to each query sample and at least one reference sample character vector corresponding to each reference sample;

[0193] Based on at least one query sample character vector corresponding to each query sample, a query sample vector corresponding to each query sample is determined, and based on at least one reference sample character vector corresponding to each reference sample, a reference sample vector corresponding to each reference sample is determined.

[0194] Optionally, each to-be-segmented text is segmented according to at least one of the following preset character segmentation methods, and the to-be-segmented text is a reference sample or a query sample:

[0195] For each query sample, perform the following operations:

[0196] Perform character segmentation on a text to be segmented according to a preset character granularity;

[0197] The text to be segmented is segmented by taking entity words as whole characters and non-entity words according to the preset character granularity.

[0198] Optionally, each query sample corresponds to at least one query sample vector, and each reference sample corresponds to at least one reference sample vector, and the determining unit 903 is specifically configured to:

[0199] Based on the character segmentation method corresponding to each query sample vector, select reference sample vectors with the same character segmentation method as each query sample vector;

[0200] Based on the similarities between each query sample vector and each corresponding reference sample vector, the reference samples corresponding to each query sample are obtained.

[0201] Optionally, obtain a trained language model by:

[0202] Acquire at least one corpus consisting of texts to be evaluated from historical data, and segment the corpus into character sets according to a preset character segmentation method;

[0203] The language model to be trained is trained based on the character set to obtain a trained language model.

[0204] Optionally, the initial annotation information includes: a first probability that the corresponding text to be annotated corresponds to each category; the determination unit 903 is specifically used to:

[0205] For each query sample, perform the following operations:

[0206] Based on the first probabilities of the reference samples corresponding to the query sample and the similarity between the corresponding reference samples and the query sample, determine the second probabilities of the query sample corresponding to the categories, and the query sample corresponds to at least one second probability;

[0207] Based on the at least one second probability and the first probability of a query sample, target labeling information of the query sample is determined.

[0208] Optionally, the determining unit 903 is specifically configured to:

[0209] Determine a third probability of the category of each search result corresponding to the query sample based on the sum of at least one second probability and a first probability of a query sample;

[0210] The category corresponding to the highest third probability is used as the target labeling information of the query sample.

[0211] Optionally, the text classification model includes at least one text binary classification model, and the training sample data set of each text binary classification model includes a positive sample set and a negative sample set; the training unit 904 is specifically used for:

[0212] The query samples whose target annotation information is the target category in the query samples are taken as the positive sample set;

[0213] According to the stratified sampling method, among the query samples whose target annotation information is non-target categories, a preset proportion of query samples are respectively extracted to form a negative sample set.

[0214] In the embodiment of the present application, firstly, a plurality of texts to be annotated are obtained, and the initial annotation information of each text to be annotated is determined according to the category of each search result corresponding to each text to be annotated and the related feedback behavior information, so that the category distribution of the text to be annotated can be preliminarily obtained according to the category of the search result corresponding to the text to be annotated; then, based on each text to be annotated and their respective initial annotation information, a plurality of query samples and a plurality of reference samples are constructed, and based on the initial annotation information of each query sample and each reference sample, the target annotation information of each query sample is determined, and the initial annotation information of the query sample can be adjusted based on the initial annotation information of the reference sample to improve the accuracy of the query sample annotation; finally, a training sample data set is constructed based on each query sample and their respective target annotation information, and a text classification model is trained based on the training sample data set. Based on the above method, rapid automatic sample annotation can be achieved, the accuracy of sample annotation can be improved, and then the text classification model can be trained based on the generated training samples, which can improve the accuracy of text classification.

[0215] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0216] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0217] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.10 As shown, it includes a memory 1001 , a communication module 1003 and one or more processors 1002 .

[0218] The memory 1001 is used to store computer programs executed by the processor 1002. The memory 1001 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0219] The memory 1001 may be a volatile memory, such as a random-access memory (RAM); the memory 1001 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1001 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1001 may be a combination of the above memories.

[0220] The processor 1002 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1002 is used to implement the above-mentioned text classification method when calling the computer program stored in the memory 1001 .

[0221] The communication module 1003 is used to communicate with terminal devices and other servers.

[0222] The specific connection medium between the memory 1001, the communication module 1003 and the processor 1002 is not limited in the embodiment of the present application. Fig.10 In the embodiment, the memory 1001 and the processor 1002 are connected via a bus 1004. The bus 1004 is connected to the processor 1002 via a bus 1004. Fig.10 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1004 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.10 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0223] The memory 1001 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the text classification method of the embodiment of the present application. The processor 1002 is used to execute the above-mentioned text classification method, such as Figure 2 shown.

[0224] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.11 As shown, it includes: a communication component 1110, a memory 1120, a display unit 1130, a camera 1140, a sensor 1150, an audio circuit 1160, a Bluetooth module 1170, a processor 1180 and other components.

[0225] The communication component 1110 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0226] The memory 1120 can be used to store software programs and data. The processor 1180 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1120. In the present application, the memory 1120 can store an operating system and various application programs, and can also store a computer program for executing the text classification method of the embodiment of the present application.

[0227] The display unit 1130 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1130 may include a display screen 1132 disposed on the front of the terminal device 110. The display unit 1130 may be used to display the machine translation user interface in the embodiment of the present application, etc.

[0228] The display unit 1130 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 1130 may include a touch screen 1131 arranged on the front of the terminal device 110, which can collect user touch operations on or near it.

[0229] The touch screen 1131 may be covered on the display screen 1132, or the touch screen 1131 and the display screen 1132 may be integrated to realize the input and output functions of the terminal device 110, and the integrated touch screen may be referred to as a touch display screen. In the present application, the display unit 1130 may display the application and the corresponding operation steps.

[0230] The camera 1140 can be used to capture still images, and users can post comments on the images taken by the camera 1140 through the application. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 1180 to convert it into a digital image signal.

[0231] The terminal device may further include at least one sensor 1150, such as an acceleration sensor 1151, a distance sensor 1152, a fingerprint sensor 1153, and a temperature sensor 1154. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0232] The audio circuit 1160, the speaker 1161, and the microphone 1162 can provide an audio interface between the user and the terminal device 110. The audio circuit 1160 can transmit the electrical signal converted from the received audio data to the speaker 1161, which is converted into a sound signal for output. On the other hand, the microphone 1162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1160 and converted into audio data, and then the audio data is output to the communication component 1110 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 1120 for further processing.

[0233] The Bluetooth module 1170 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol.

[0234] The processor 1180 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1120 and calling data stored in the memory 1120. In some embodiments, the processor 1180 may include at least one processing unit; the processor 1180 may also integrate an application processor and a baseband processor. In the present application, the processor 1180 can run an operating system, an application program, a user interface display and touch response, and a text classification method of an embodiment of the present application. In addition, the processor 1180 is coupled to the display unit 1130.

[0235] In some possible implementations, various aspects of the text classification method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the text classification method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may perform the following steps: Figure 2 Follow the steps shown in .

[0236] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0237] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0238] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0239] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0240] Computer programs for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program may be executed entirely on the user electronic device, partially on the user device, as a separate software package, partially on the user electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In the case of a remote electronic device, the remote electronic device may be connected to the user electronic device via any type of network including a local area network (LAN) or a wide area network (WAN), or may be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0241] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0242] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0243] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0244] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0245] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0246] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0247] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0248] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A text classification method, characterized in that: The method includes: Acquire multiple texts to be annotated, and determine initial annotation information of each text to be annotated according to the categories of each search result corresponding to each text to be annotated and related feedback behavior information; the initial annotation information includes a first probability that the text to be annotated corresponds to each category; Based on the texts to be annotated and their respective initial annotation information, a plurality of query samples and a plurality of reference samples are constructed, wherein the reference samples are used to adjust the initial annotation information of the corresponding query samples; For each query sample, based on the first probabilities of the reference samples corresponding to the query sample, the similarities between the corresponding reference samples and the query sample, and the first probability of the query sample, determine the target labeling information of the query sample; A training sample data set is constructed based on each query sample and respective target annotation information, and a text classification model is trained based on the training sample data set. The text classification model is used to classify the text to be classified.

2. The method according to claim 1, characterized in that The feedback behavior information includes the number of clicks and exposures of each search result; The determining of the initial annotation information of each of the texts to be annotated according to the categories of the search results corresponding to each of the texts to be annotated and the related feedback behavior information includes: For each text to be annotated, perform the following operations: Determine the exposure rate and click rate of each category corresponding to the text to be annotated based on the total number of exposures and the total number of clicks of the search results of the same category corresponding to the text to be annotated; Based on the exposure rate and click rate of the one to-be-annotated text corresponding to each category, a first probability that the one to-be-annotated text corresponds to each category is determined.

3. The method according to claim 2, characterized in that The determining of the exposure rate and click rate of each category corresponding to the text to be annotated based on the total exposure times and the total click times of the search results of the same category corresponding to the text to be annotated includes: Determine the exposure rate of each category corresponding to the text to be annotated based on the total exposure times of search results of the same category corresponding to the text to be annotated and the total exposure times of all search results corresponding to the text to be annotated; Based on the total number of clicks on search results of the same category corresponding to a text to be annotated and the total number of clicks on all search results corresponding to the texts to be annotated, the click rates of the categories corresponding to the text to be annotated are determined.

4. The method according to claim 1, characterized in that The reference samples corresponding to each query sample are obtained in the following way: Performing character segmentation on each query sample and each reference sample according to a preset character segmentation method to obtain query sample characters contained in each query sample and reference sample characters contained in each reference sample; Based on the query sample characters respectively included in each query sample and the reference sample characters respectively included in each reference sample, obtaining the query sample vectors respectively corresponding to the query samples and the reference sample vectors respectively corresponding to the reference samples; Based on the similarities between each query sample vector and each reference sample vector, the reference samples corresponding to each query sample are obtained.

5. The method according to claim 4, characterized in that The obtaining, based on the query sample characters respectively included in each query sample and the reference sample characters respectively included in each reference sample, the query sample vectors respectively corresponding to the query samples and the reference sample vectors respectively corresponding to the reference samples, comprises: Encoding the query sample characters contained in each query sample and the reference sample characters contained in each reference sample based on the trained language model to obtain at least one query sample character vector corresponding to each query sample and at least one reference sample character vector corresponding to each reference sample; Based on at least one query sample character vector corresponding to each query sample, a query sample vector corresponding to each query sample is determined, and based on at least one reference sample character vector corresponding to each reference sample, a reference sample vector corresponding to each reference sample is determined.

6. The method according to claim 4, characterized in that Each to-be-segmented text is segmented according to at least one of the following preset character segmentation methods, where the to-be-segmented text is a reference sample or a query sample: For each query sample, perform the following operations: Perform character segmentation on a text to be segmented according to a preset character granularity; The text to be segmented is segmented by taking entity words as whole characters and non-entity words according to the preset character granularity.

7. The method according to claim 4, characterized in that Each query sample corresponds to at least one query sample vector, and each reference sample corresponds to at least one reference sample vector. The obtaining of reference samples corresponding to each query sample based on the similarity between each query sample vector and each reference sample vector includes: Based on the character segmentation method corresponding to each query sample vector, select reference sample vectors with the same character segmentation method as each query sample vector; Based on the similarities between each query sample vector and each corresponding reference sample vector, the reference samples corresponding to each query sample are obtained.

8. The method according to claim 5, characterized in that Get the trained language model by: Acquire at least one corpus consisting of texts to be evaluated from historical data, and segment the corpus into character sets according to a preset character segmentation method; The language model to be trained is trained based on the character set to obtain a trained language model.

9. The method according to any one of claims 1 to 8, characterized in that: The determining the target labeling information of the query sample based on the first probabilities of the reference samples corresponding to the query sample, the similarity between the corresponding reference samples and the query sample, and the first probability of the query sample includes: Based on the first probabilities of the reference samples corresponding to a query sample and the similarity between the corresponding reference samples and the query sample, determine the second probabilities of the query sample corresponding to each category, the query sample corresponding to at least one second probability; Based on the at least one second probability and the first probability of the query sample, target labeling information of the query sample is determined.

10. The method according to claim 9, characterized in that The step of determining target labeling information of the query sample based on the at least one second probability and the first probability of the query sample is as follows: Determine, based on the sum of the at least one second probability and the first probability of the one query sample, a third probability of the category of each search result corresponding to the query sample; The category corresponding to the highest third probability is used as the target labeling information of the query sample.

11. The method according to any one of claims 1 to 8, characterized in that: The text classification model includes at least one text binary classification model, and the training sample data set of each text binary classification model includes a positive sample set and a negative sample set; the training sample data set is constructed based on each query sample and respective target annotation information, including: Taking the query samples whose target annotation information is the target category in the query samples as the positive sample set; According to the stratified sampling method, among various query samples whose target annotation information in the query samples is a non-target category, a preset proportion of query samples are respectively extracted to form a negative sample set.

12. A text classification device, characterized in that: The device includes: An acquisition unit is used to acquire a plurality of texts to be annotated, and determine initial annotation information of each text to be annotated according to the categories of each search result corresponding to each text to be annotated and related feedback behavior information; the initial annotation information includes a first probability that the text to be annotated corresponds to each category; A construction unit, configured to construct a plurality of query samples and a plurality of reference samples based on the texts to be annotated and their respective initial annotation information, wherein the reference samples are used to adjust the initial annotation information of the corresponding query samples; A determination unit, configured to determine, for each query sample, target labeling information of the query sample based on a first probability of each reference sample corresponding to the query sample, a similarity between the corresponding reference sample and the query sample, and the first probability of the query sample; The training unit is used to construct a training sample data set based on the query samples and their respective target annotation information, and to train a text classification model based on the training sample data set, wherein the text classification model is used to classify the text to be classified.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The method comprises a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any one of the methods of claims 1 to 11.

15. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 11.

Images