A term processing method, apparatus, device and readable storage medium

Abstract

This application discloses a word processing method, apparatus, device, and readable storage medium. The method includes: acquiring training sample sentences; performing word annotation processing on the training sample sentences based on the relevance between the training sample sentences and documents in a document database to obtain a word level label for each sample word in the training sample sentences; acquiring word features for each sample word, whereby the word features reflect the frequency of occurrence and word category information of each sample word; and training an initial word classification model based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model, whereby the target word classification model is used to classify the word level of each word contained in the sentence. Using this application embodiment can improve word annotation efficiency.

Description

Technical Field

[0001] This application relates to the field of word processing technology, and in particular to a word processing method, apparatus, device, and readable storage medium. Background Technology

[0002] Word processing technology is widely used in various technical fields, including word search. Since a sentence contains multiple words, each with a different meaning, the search results for each word in a sentence will vary. Currently, the common practice is to manually assess the importance of each word in a sentence and then manually label each word based on its importance. This labeling is then incorporated into subsequent word searches. However, manual word labeling is inefficient; therefore, improving its efficiency is a pressing issue. Summary of the Invention

[0003] This application provides a word processing method, apparatus, device, and readable storage medium, which can improve word annotation efficiency.

[0004] Firstly, this application provides a word processing method, including:

[0005] Obtain training sample sentences, and based on the relevance between the training sample sentences and documents in the document database, perform word tagging on the training sample sentences to obtain the word level label for each sample word in the training sample sentences.

[0006] The word features of each sample word are obtained, and the word features of each sample word are used to reflect the frequency of occurrence of each sample word and the word category information of each sample word;

[0007] Based on the word level label and word features of each sample word in the training sample sentence, an initial word classification model is trained to obtain a target word classification model. This target word classification model is used to classify the word level of each word contained in the sentence.

[0008] Secondly, this application provides a word processing apparatus, the apparatus comprising:

[0009] The tag acquisition unit is used to acquire training sample sentences, and based on the relevance between the training sample sentences and documents in the document database, to perform word tagging processing on the training sample sentences to obtain the word level tag of each sample word in the training sample sentences.

[0010] The feature acquisition unit is used to acquire the word features of each sample word. The word features of each sample word are used to reflect the frequency of occurrence of each sample word and the word category information of each sample word.

[0011] The model training unit is used to train an initial word classification model based on the word level label and word features of each sample word in the training sample sentence, and obtain a target word classification model. The target word classification model is used to classify the word level of each word contained in the sentence.

[0012] Thirdly, this application provides a computer device, including: a processor, a memory, and a network interface;

[0013] The processor is connected to a memory and a network interface. The network interface is used to provide data communication functions, the memory is used to store computer program code, and the processor is used to call the computer program code to cause the computer device containing the processor to execute the word processing method.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having the processor performs the above-described word processing method.

[0015] Fifthly, this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the word processing methods provided in the various alternative embodiments of the first aspect of this application.

[0016] In this embodiment, training sample sentences are obtained, and word tagging is performed on the training sample sentences based on the relevance between the training sample sentences and documents in the document database to obtain the word level label for each sample word in the training sample sentences. Since the relevance between documents in the document database and the training sample sentences is combined to tag each sample word in the training sample sentences, the accuracy of word tagging can be improved, thereby improving the accuracy of word level label determination. By obtaining the word features of each sample word, which reflect the frequency of occurrence and word category information of each sample word, an initial word classification model is trained based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model. Subsequently, the target word classification model can be used to classify the word level of each word contained in any sentence, improving the efficiency of word level determination. By using the target word classification model to classify words and thus perform word tagging, word tagging efficiency can be improved without the need for manual classification and tagging. Furthermore, by combining information such as word frequency and word category to classify word levels, the accuracy of word classification can be improved, thereby improving the accuracy of determining word levels. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a network architecture diagram of a word processing system provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a word processing method provided in an embodiment of this application;

[0020] Figure 3 This is a schematic diagram illustrating the determination of word level labels for multiple words in a training sample sentence, provided in an embodiment of this application.

[0021] Figure 4 This is a flowchart illustrating a word processing method provided in an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of the composition structure of a word processing device provided in an embodiment of this application;

[0023] Figure 6This is a schematic diagram of the composition structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The technical solution of this application can be applied to scenarios where multiple words in a sentence are labeled to determine the word level of each word. For example, it can be applied to search systems, where labeling each word in a search sentence determines the word level of each word, allowing for a more accurate search by combining the word levels of each word. The technical solution of this application can also be applied to any scenario requiring word level determination, and this application does not limit the application to such scenarios.

[0026] Please refer to Figure 1 , Figure 1 This is a network architecture diagram of a word processing system provided in an embodiment of this application, such as... Figure 1 As shown, the computer device can interact with the terminal device, and the number of terminal devices can be one or at least two. For example, when there are multiple terminal devices, the terminal devices may include... Figure 1 The system includes terminal devices 101a, 101b, and 101c. Taking terminal device 101a as an example, computer device 102 can acquire training sample sentences. Based on the relevance between the training sample sentences and documents in the document database, it performs word tagging on the training sample sentences to obtain a word level label for each sample word in the training sample sentences. Further, computer device 102 can acquire the word features of each sample word. Further, computer device 102 can train an initial word classification model based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model. Optionally, computer device 102 can also use the target word classification model to classify and predict the word level of each word in any query sentence, obtain the word level of each word in the query sentence, and perform document search based on the word level of each word in the query sentence, sending the searched documents to terminal device 101a.

[0027] The computer equipment mentioned in this application embodiment can refer to a server or a terminal device, or it can be a system composed of a server and a terminal device. This application embodiment does not limit this. A terminal device can be an electronic device, including but not limited to mobile phones, tablets, laptops, and in-vehicle devices. The server mentioned above can be a single physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-to-everything (V2X) communication, content delivery networks (CDNs), and big data and artificial intelligence platforms. When the server is a single physical server, it can perform word processing independently. When there are multiple physical servers, they can collaborate to perform word tagging, word classification, and other processing.

[0028] Further, please see Figure 2 , Figure 2 This is a flowchart illustrating a word processing method provided in an embodiment of this application; as shown below. Figure 2 As shown, this word processing method can be applied to computer devices, and the word processing method includes, but is not limited to, the following steps:

[0029] S101, Obtain training sample sentences. Based on the relevance between the training sample sentences and documents in the document database, perform word tagging on the training sample sentences to obtain the word level label for each sample word in the training sample sentences.

[0030] The technical solution of this application can be applied to search systems for word search, sentence search, or document search, etc. In search systems, word processing is an important task, generally including word segmentation, intent recognition, named entity recognition, and word weighting. This application mainly focuses on word weighting, which assigns different weights to each word in the query statement according to certain rules, thereby affecting subsequent word recall, word ranking, and ultimately the search results and the user's search experience. Intent recognition is generally used to determine the main meaning of the query statement, while named entity recognition is generally used to identify which named entity words are included in the query statement.

[0031] Current word weighting methods typically assign weights to each word within a sentence based on its context. However, in actual searches, the semantic relationships between words are complex. For example, sentence 1 is "[X sports brand, shampoo]", and sentence 2 is "[a certain shampoo brand, shampoo]". Both sentences involve a brand plus a specific category, but X sports brand's main products are clothing, not toiletries. Therefore, the word "shampoo" in sentence 1 is far more important than "shampoo" in sentence 2. Similarly, in the sentence "[suitcase, rectangle]", since suitcases are generally rectangular, the importance of "rectangle" in this sentence could be zero. Because these patterns between words in sentences are complex, machine learning combined with search click mining is needed to solve this problem. However, constructing labels for the model training data (e.g., word level labels for training samples) is a complex issue.

[0032] Currently, the common practice for labeling model training data is to directly use the inverse document frequency (IVF) and co-occurrence frequency (COF) of words as labels. However, this method is relatively simple and struggles to guarantee that the labels accurately reflect word importance. Furthermore, IVF and COF may conflict, making the accuracy of labeling solely based on these two factors low and failing to accurately reflect word importance. For example, IVF might indicate that a less frequent occurrence of a word suggests greater importance, but a less frequent occurrence might also indicate a lower COF, meaning the word's importance is lower. Similarly, when a user enters incorrect information in a search system, such as garbled text, the garbled text has a higher IVF because search system documents generally don't contain garbled text. However, the garbled text has a lower COF. Therefore, the two features of IVF and COF may conflict, making it inaccurate to use them as word labels.

[0033] Therefore, in this embodiment, the importance of each word in a sentence can be determined based on the search results of the search system. If relevant results can still be found after removing a certain word, it indicates that the removed word is not important; conversely, if no relevant results can be found after removing a certain word, it indicates that the removed word is relatively important. In this way, word annotation data, i.e., word level labels, can be automatically obtained. Furthermore, feature mining can be performed on each word in the sentence to obtain word features. Then, the word annotation data and word features can be combined to train a model, which can then be used to predict word weights.

[0034] In this embodiment, training sample sentences can be obtained, and each word in the training sample sentences can be labeled to obtain a word level label for each sample word in the training sample sentences. Specifically, training sample sentences can be obtained, and based on the relevance between the training sample sentences and documents in the document database, word labeling processing can be performed on the training sample sentences to obtain a word level label for each sample word in the training sample sentences.

[0035] When acquiring training sample sentences, they can be downloaded from a relevant database or obtained from a terminal device; this embodiment does not limit this. The document database can, for example, refer to the database corresponding to a search system, and can store a large number of documents, sentences, or words, etc. A document can consist of multiple sentences, and a sentence can consist of multiple words. Word tagging processing is used to annotate the word level label of each word in the training sample sentences; that is, word tagging processing can annotate the importance of each word in the training sample sentences. The higher the relevance between the training sample sentences and the documents in the document database, the higher the word level indicated by the word level label of the sample words in the training sample sentences. The lower the relevance between the training sample sentences and the documents in the document database, the lower the word level indicated by the word level label of the sample words in the training sample sentences. Word level can be used to reflect the importance of words; the higher the word level, the higher the importance of the word. The lower the word level, the lower the importance of the word.

[0036] Optionally, word ranking can include multiple levels, such as four levels: Level 1, Level 2, Level 3, and Level 4. Words at Level 1 are more important than words at Level 2, words at Level 2 are more important than words at Level 3, and words at Level 3 are more important than words at Level 4. Optionally, word ranking labels can include at least two levels, such as 0, 1, 2, and 3. A lower level corresponds to a lower word ranking, meaning lower word importance. A higher level corresponds to a higher word ranking, meaning higher word importance. For example, Level 1 could correspond to 3 levels, Level 2 to 2 levels, Level 3 to 1 level, and Level 4 to 0 levels.

[0037] In one embodiment, the word level label of each sample word in the training sample statement can be determined as follows: the training sample statement is segmented to obtain multiple sample words; each sample word in the multiple sample words is removed to obtain the unprocessed sample words associated with each sample word; the document corresponding to the unprocessed sample words associated with each sample word is determined from the document database, and each unprocessed sample word associated with each sample word corresponds to at least one document; the correlation between the training sample statement and the document corresponding to the unprocessed sample words associated with each sample word is obtained, and the word level label of each sample word in the multiple sample words is determined based on the correlation.

[0038] The process involves word segmentation to divide the training sample statement into multiple sample words, and word removal to remove words from each of these sample words, resulting in the corresponding unprocessed sample words associated with each sample word. Determining the document corresponding to each unprocessed sample word from the document database can be achieved by inputting the unprocessed sample word associated with each sample word into the document database and searching for the document. For each unprocessed sample word associated with a sample word, at least one document can be found. Therefore, each unprocessed sample word associated with multiple sample words corresponds to at least one document. The relevance between the training sample statement and the document corresponding to each unprocessed sample word can be used to indicate the degree of relevance between the training sample statement and the document. The higher the relevance between the training sample statement and the document, the higher the word importance of each sample word. The lower the relevance between the training sample statement and the document, the lower the word importance of each sample word.

[0039] Optionally, determining the word level label for each sample word among multiple sample words based on relevance could mean that the relevance and the word level indicated by the word level label are directly proportional; for example, the higher the relevance, the higher the word level indicated by the word level label; and the lower the relevance, the lower the word level indicated by the word level label. Determining the word level label for each word through relevance can improve the accuracy of word level determination, thereby improving the accuracy of word importance determination, and ultimately improving the accuracy of word weighting.

[0040] Understandingly, a search system can refer to a system that takes a statement or word as input and outputs documents or statements associated with that statement or word. By searching for the unprocessed sample words associated with each sample word, the documents corresponding to those unprocessed sample words can be determined. Based on the relevance between the training sample statement and the documents corresponding to the unprocessed sample words associated with each sample word, the word ranking label for each sample word in the training sample statement can be determined, reflecting the importance of each sample word. This method of determining word ranking labels, by combining the search documents corresponding to each sample word and the relevance between the training sample statement and the search documents, accurately reflects the importance of each word in the training sample data. Since this method can automatically perform word labeling on any training sample statement, it is less costly than manual labeling and improves word labeling efficiency.

[0041] In one embodiment, taking any one of the multiple sample words included in the training sample data as the target sample word as an example, the process of determining the sample words to be processed associated with the target sample word is explained as follows: the target sample word can be processed by word removal to obtain the sample words to be processed associated with the target sample word. The sample words to be processed associated with the target sample word include sample words other than the target sample word among the multiple sample words.

[0042] Optionally, for example, if the multiple sample words include a first sample word, a second sample word, and a third sample word, then the method for performing word removal processing on each of the multiple sample words to obtain the unprocessed sample words associated with each sample word may include:

[0043] Word removal is performed on the first sample word from multiple sample words to obtain the unprocessed sample words associated with the first sample word; word removal is performed on the second sample word from multiple sample words to obtain the unprocessed sample words associated with the second sample word; word removal is performed on the third sample word from multiple sample words to obtain the unprocessed sample words associated with the third sample word.

[0044] Among them, the unprocessed sample words associated with the first sample word include the second sample word and the third sample word; the unprocessed sample words associated with the second sample word include the first sample word and the third sample word; and the unprocessed sample words associated with the third sample word include the first sample word and the second sample word.

[0045] For example, if the training sample sentence is "urban cultural exchange," and the training sample data includes multiple sample words such as [city, culture, exchange], removing each word from these sample words yields the unprocessed sample words associated with the removed words. For instance, after removing the sample word "city," the unprocessed sample words are [culture, exchange]. Searching for [culture, exchange] in a search system will then yield search results for [culture, exchange], including documents related to [culture, exchange]. Similarly, removing the sample word "culture" will result in [city, exchange], which will also yield search results for [city, exchange], including documents related to [city, exchange]. Furthermore, removing the sample word "exchange" will result in [city, culture], which will also yield search results for [city, culture], including documents related to [city, culture].

[0046] Furthermore, the correlation between the training sample phrase "urban cultural exchange" and the documents corresponding to the words after removing each sample word can be obtained. For example, the correlation between "urban cultural exchange" and the documents corresponding to [culture, exchange] can be obtained, and the word level label of the sample word [city] can be determined based on the correlation. Similarly, the correlation between "urban cultural exchange" and the documents corresponding to [city, exchange] can be obtained, and the word level label of the sample word [culture] can be determined based on the correlation.

[0047] In one embodiment, feature parameters can be obtained between sample words in the training sample statements and the documents corresponding to the sample words to be processed associated with each sample word. Based on these feature parameters, a word ranking label for each sample word among multiple sample words can be determined. These feature parameters can reflect the similarity between sample words in the training sample statements and the documents corresponding to the sample words to be processed associated with each sample word. The feature parameters between sample words and the documents corresponding to the sample words to be processed associated with each sample word may include, but are not limited to: the number of identical words in the documents corresponding to the sample words and the associated sample words; the number of operations on the documents corresponding to the sample words and the associated sample words in historical search records; the category similarity between the statement category of the training sample statement and the document category of the documents corresponding to the sample words and the associated sample words; and so on.

[0048] Historical search records refer to historical search information within the search system, such as search terms, searched documents, and search counts. The number of operations performed on documents corresponding to each sample term in the historical search records refers to the number of times the target operation was executed on the documents corresponding to each sample term in the historical search records when searching for the training sample statement. For example, if 50 target operations (such as clicks or views) were performed on the documents corresponding to each sample term in the historical search records when searching for the training sample statement, the number of operations would be 50. Category similarity reflects the similarity between statement and document categories. For example, a category similarity greater than a similarity threshold indicates similarity between the statement and document categories. A category similarity less than or equal to the similarity threshold indicates dissimilarity between the statement and document categories. The statement category to which the training sample statement belongs reflects the domain to which the training sample statement belongs. This could include, but is not limited to, categories such as games, sports, movies, food, and outdoor sports. The document category can reflect the area the document belongs to, such as games, sports, movies, food, outdoor sports, etc. Optionally, if the statement category and document category are the same, the category similarity is considered to be greater than the similarity threshold; if the statement category and document category are different, the category similarity is considered to be less than or equal to the similarity threshold.

[0049] Optionally, for example, documents where the number of identical words in the documents corresponding to each sample word is greater than a threshold can be considered relevant to the sample word; documents where the number of identical words in the documents corresponding to each sample word is less than or equal to a threshold can be considered irrelevant. Alternatively, documents where the number of operations on the documents corresponding to each sample word in historical search records is greater than a threshold can be considered relevant to the sample word; documents where the number of operations on the documents corresponding to each sample word in historical search records is less than or equal to a threshold can be considered irrelevant. Alternatively, documents where the category similarity between the sentence category of the training sample sentence and the document category of the documents corresponding to each sample word is greater than a similarity threshold can be considered relevant to the training sample sentence; documents where the category similarity between the sentence category of the training sample sentence and the document category of the documents corresponding to each sample word is less than or equal to a similarity threshold can be considered irrelevant.

[0050] When it is determined that the document corresponding to the sample word associated with each sample word is relevant to the training sample sentence, the word level label indicating the corresponding sample word in the training sample sentence can be considered to indicate a high word level. When it is determined that the document corresponding to the sample word associated with each sample word is irrelevant to the training sample sentence, the word level label indicating the corresponding sample word in the training sample sentence can be considered to indicate a low word level.

[0051] It is understood that any one of the three feature parameters mentioned above can be used to determine the relevance between the training sample statement and the document corresponding to the sample word to be processed associated with each sample word, or any two of the three feature parameters can be combined to determine the relevance between the training sample statement and the document corresponding to the sample word to be processed associated with each sample word, or the three feature parameters can be combined to determine the relevance between the training sample statement and the document corresponding to the sample word to be processed associated with each sample word. This application embodiment does not limit this.

[0052] Optionally, taking the determination of the relevance between the training sample statement and the documents corresponding to the unprocessed sample words associated with each sample word by combining the three feature parameters mentioned above, and using the determination of the word level label of each sample word among multiple sample words as an example: The number of words with the same word in the documents corresponding to the unprocessed sample words associated with each sample word in the training sample statement can be obtained; the documents corresponding to the unprocessed sample words associated with each sample word based on the training sample statement in the historical search records can be obtained, and the number of operations performed on the target operation for that document can be obtained; the statement category to which the training sample statement belongs, and the document category to which the documents corresponding to the unprocessed sample words associated with each sample word belong, can be obtained, and the category similarity between the statement category and the document category can be obtained; based on the number of words, the number of operations, and the category similarity, the associated documents of each sample word can be determined from the documents corresponding to the unprocessed sample words associated with each sample word; the ratio between the number of associated documents of each sample word and the total number of documents corresponding to the unprocessed sample words associated with each sample word can be obtained, and the word level label of each sample word among multiple sample words can be determined based on the ratio.

[0053] For example, if the number of words exceeds a threshold, the number of operations exceeds a threshold, and the category similarity exceeds a threshold, then the document is identified as an associated document for each sample word. For instance, if the number of associated documents is 30, the ratio between the 30 associated documents for each sample word and the total number of documents corresponding to the 50 processed sample words associated with each sample word is 60%. Based on this ratio, a word level label is determined for each sample word, for example, a word level label of 1. Optionally, in practical applications, to reduce computational load, when determining the total number of documents corresponding to the processed sample words associated with each sample word, if the total number is large, some documents with higher order can be selected. The earlier the document is selected, the greater the correlation between the document and the training sample statement, indicating a higher correlation. For example, documents with an order less than or equal to 50 can be selected, resulting in a total of 50 documents corresponding to the processed sample words associated with each sample word. Subsequently, when calculating relevance, only the correlation between the selected documents and the training sample statement needs to be calculated, thereby reducing computational load and improving data processing efficiency. If the total number is small, the correlation between each document and the training sample sentences can be calculated, thereby improving the calculation accuracy.

[0054] In this embodiment, the relevance of each document in the training sample sentence and the documents corresponding to the unprocessed sample words associated with each sample word is determined, along with the proportion of relevant documents to the total number of documents associated with each sample word. A lower proportion indicates a greater impact of removing each sample word on the intent of the training sample sentence, meaning the removed word is more important. Therefore, the word level label for each word in the training sample sentence can be determined based on this proportion. Optionally, if the proportion is lower than a first proportion threshold but higher than a second proportion threshold, the word level label for the corresponding sample word can indicate an extremely high word level, for example, a level 3 label. If the proportion is lower than the second proportion threshold but higher than a third proportion threshold, the word level label for the corresponding sample word can indicate a high word level, for example, a level 2 label. If the proportion is lower than the third proportion threshold but higher than a fourth proportion threshold, the word level label for the corresponding sample word can indicate a low word level, for example, a level 1 label. If the proportion is lower than the fourth proportion threshold, the word level label for the corresponding sample word can indicate an extremely low word level, for example, a level 0 label.

[0055] Please see Figure 3 , Figure 3This is a schematic diagram illustrating the determination of word level labels for multiple words in a training sample sentence according to an embodiment of this application. The training sample sentence is "international cultural exchange." Each sample word in the training sample sentence undergoes word removal processing, resulting in three unprocessed sample words associated with each sample word: [culture, exchange], [international, exchange], and [international, culture]. Search result 1 can refer to the ratio between the number of associated documents corresponding to the unprocessed sample word [culture, exchange] and the total number of documents corresponding to [culture, exchange], for example, a ratio of 0%. Search result 2 can refer to the ratio between the number of associated documents corresponding to the unprocessed sample word [international, exchange] and the total number of documents corresponding to [international, exchange], for example, a ratio of 80%. Search result 3 can refer to the ratio between the number of associated documents corresponding to the unprocessed sample word [international, culture] and the total number of documents corresponding to [international, culture], for example, a ratio of 0%. Then, the word level label of the sample word "[International]" in the training sample sentence can be determined as level 3, meaning that the importance of "[International]" in the training sample sentence is extremely high; the word level label of the sample word "[Culture]" in the training sample sentence is level 1, meaning that the importance of "[Culture]" in the training sample sentence is low; and the word level label of the sample word "[Communication]" in the training sample sentence is level 3, meaning that the importance of "[Communication]" in the training sample sentence is extremely high. Thus, the word level label of each sample word in the training sample sentence can be determined.

[0056] In one embodiment, a relevance model can be used to calculate the relevance between training sample statements and the documents corresponding to the sample words to be processed associated with each sample word. Specifically, a large amount of sample data, such as training sample statements and sample documents, can be used to train the relevance model in advance, enabling it to determine the relevance of the training sample statements and sample documents. For example, when training the relevance model, training sample statements and sample documents can be input, and the model can process them, outputting a relevance result between the training sample statements and sample documents. This result may include whether the results are related or not. The relevance model is then trained based on the difference between the relevance labels of the training sample statements and sample documents and the relevance result output by the model. By training the relevance model with a large amount of sample data, the accuracy of the model's processing can be improved, thereby increasing the accuracy of relevance determination when subsequently calculating the relevance between the training sample statements and the documents corresponding to the sample words to be processed associated with each sample word.

[0057] Optionally, the relevance model may include, but is not limited to, the Transformer model (self-attention model), the pre-trained language representation model (Bidirectional Encoder Representation from Transformers, BERT), the Convolutional Neural Network (CNN), the Recurrent Neural Network (RNN), the Residual Network (ResNet), and so on.

[0058] S102, obtain the word features of each sample word.

[0059] In this embodiment, the training sample sentence can be segmented to obtain multiple sample words contained in the training sample sentence; each of the multiple sample words can then be processed to obtain word features corresponding to each sample word, i.e., word features for each sample word. Word features can reflect the importance of words to a certain extent; for example, the word features of each sample word can reflect the frequency of occurrence and the word category information of each sample word. Word processing can include, but is not limited to, word occurrence statistics, part-of-speech prediction, named entity prediction, etc.

[0060] In one embodiment, the word features of each sample word include the term frequency (TF) of each sample word. For example, the TF of each sample word can be obtained by obtaining the number of times each sample word appears in the document database and the total number of words in the document database, and determining the TF of each sample word by the ratio between the number of times the word appears and the total number of words.

[0061] The frequency of each sample word in the document database can refer to the number of documents containing each sample word stored in the database. Each sample word can appear multiple times in the same document. When each sample word appears only once in each document, the frequency can be the sum of the frequencies of each sample word in all documents containing it. When each sample word appears multiple times in each document, the frequency can be the sum of the frequencies of each sample word in all documents containing it. The total number of words in the document database can refer to the sum of the number of words in all documents contained in the database, or the sum of the number of words in all statements contained in the database, or the sum of the number of words in all words contained in the database, or at least two of the following: the number of words in all documents contained in the database, the number of words in all statements contained in the database, and the number of words in all words contained in the database. The ratio between the frequency of each sample word in the document database and the total number of words in the document database reflects the frequency of each sample word in the database. Generally, a higher frequency indicates that the sample word is used more often, has a lower word ranking in statements, and is less important. By combining word frequency to determine word level, we can analyze word level from the perspective of word frequency and improve the accuracy of word level determination.

[0062] Optionally, when calculating the IF, the number of times each sample word appears in all documents in the document database (i.e., the number of times the word appears) n, and the total number of documents in the document database (i.e., the total number of words) N, are counted. The ratio of the two is then determined as the IF, i.e., IF = n / N.

[0063] In one embodiment, the word features of each sample word include the inverse document frequency (IDF) of each sample word. The inverse document frequency is used to reflect the importance of each sample word in the document. For example, the inverse document frequency of each sample word can be obtained by obtaining the number of documents in the document database that contain each sample word and the total number of documents in the document database, and determining the ratio between the total number of documents and the number of documents as the inverse document frequency of each sample word.

[0064] In this context, "document count" can refer to the number of documents in the document database containing each sample word, while "total number of documents in the document database" can refer to the sum of all documents in the database. Inverse document frequency (IVF) indicates the frequency with which each sample word appears in the documents of the database; the less frequently a sample word appears, the more important it is. By combining the IVF of each sample word to determine its word ranking, the accuracy of word ranking determination can be improved through analysis of IVF.

[0065] Optionally, when calculating IDF, the number of documents (document count) d of each sample word appearing in the document database is counted, and the total number of documents (i.e., total number of documents) D in the document database is counted. The ratio of the two is determined as IDF, i.e., IDF = D / d.

[0066] Optionally, the word features of each sample word can also include the Term Frequency-Inverse Document Frequency (TF-IDF) of each sample word. This means that the importance of each sample word can be comprehensively measured by combining its term frequency and inverse document frequency. TF-IDF calculates the weight of each sample word by comprehensively considering its TF and IDF, and is calculated as the product of its term frequency and inverse document frequency. The calculation method is: TF-IDF = TF * IDF. In other words, when any sample word appears multiple times in only a few documents in the document database, its word weight is the highest, for example, the first weight, meaning the word level indicated by the word level label is the highest, and therefore this sample word can provide the strongest discriminative ability for documents. When any sample word appears infrequently in a document in the document database, or when it appears in many documents, its weight is the second highest, for example, the second weight. When any sample word appears in all documents in the document database, its weight is the lowest, for example, the third weight. The first weight is greater than the second weight, and the second weight is greater than the third weight. By combining the term frequency-inverse document frequency of each sample word to determine its word ranking, the accuracy of word ranking can be improved.

[0067] In one embodiment, the word features of each sample word include the inverse query frequency (IQF) of each sample word, which reflects the importance of each sample word in the training sample sentences. For example, the IQF of each sample word can be obtained by obtaining the number of sentences containing each sample word in the document database and the total number of sentences in the document database, and determining the IQF of each sample word by the ratio between the total number of sentences and the total number of sentences.

[0068] The number of sentences can refer to the total number of sentences in the document database containing each sample word, or the total number of sentences in the document database can refer to the sum of the number of sentences in the document database. Inverse sentence frequency (IQF) indicates the importance of each sample word; the less frequently it appears in the sentences of the document database, the more important it is. In other words, IQF can represent the importance of any sample word in the training sample sentences; the fewer sentences containing that sample word, the stronger its sentence category feature, and thus the greater its importance. By combining the IQF of each sample word to determine its word ranking, the accuracy of word ranking determination can be improved by analyzing the IQF aspect.

[0069] Optionally, the inverse statement frequency can be expressed as the inverse query frequency. The inverse query frequency is calculated by first counting the number of queries (i.e., the number of statements) q for each sample word and the total number of queries (i.e., the total number of statements) Q. That is, the inverse query frequency is calculated as: IQF = Q / q.

[0070] In one embodiment, the word features of each sample word include the co-occurrence frequency of each sample word, which reflects the importance of each sample word. For example, the co-occurrence frequency of each sample word can be obtained by: statistically analyzing the documents associated with the statements containing each sample word in a document database; obtaining a first number of documents in the associated documents where the target operation is performed; determining a second number of documents containing each sample word at the target position in the documents where the target operation is performed; and determining the ratio between the second number and the first number as the co-occurrence frequency of each sample word.

[0071] The target operation may include, but is not limited to, click operations and view operations. The first quantity may refer to the number of documents in the document database associated with each sample word whose target operation is being performed. The target location may include, but is not limited to, document titles such as main titles and subtitles, document tables of contents, document summaries, and document headers and footers. For example, the second quantity may refer to the number of times each sample word is contained in the titles of the documents whose target operation is being performed. By combining the co-occurrence frequency of each sample word to determine the word's rank, the accuracy of word rank determination can be improved by analyzing the word's rank based on its co-occurrence frequency.

[0072] Optionally, the co-occurrence frequency can be measured by calculating the common words in the training sample sentences and the documents clicked by the user from the document database based on the training sample sentences, thus measuring the importance of each sample word in the training sample sentences. For example, one can first count the sentences containing each sample word and the corresponding sentence-document pairs (i.e., the documents that the user clicked in the documents associated with each sample word), that is, the total number of sentence-document pairs (i.e., the first count) is N. qd For each statement-document pair, the total number of statement-document pairs (i.e., the second count) that calculates whether each sample word appears in the title of the document containing that statement is n. qd The ratio between the second quantity and the first quantity is calculated as the co-occurrence frequency of each sample word, i.e., the co-occurrence frequency is freq = n. qd / n qd .

[0073] For example, if the document database of a search system stores 10,000 statements, and 500 of these statements contain any one of the sample words, and each statement corresponds to one document, then the 500 statements correspond to 500 documents. Furthermore, based on historical search records, it can be determined which of these 500 documents the user clicked. For instance, if the user clicked on 50 documents out of the 500 statements, then the co-occurrence frequency is 50 / 500 = 0.1.

[0074] In one embodiment, the word features of each sample word include its word length. For example, the word length can be obtained by acquiring the number of characters contained in each sample word, which serves as its word length. The number of characters can refer to the number of Chinese characters, the number of letters, or the sum of both. By combining the word length of each sample word to determine its word level, the word level can be analyzed from the perspective of word length, thereby improving the accuracy of word level determination.

[0075] In one embodiment, the word features of each sample word include the part-of-speech tag of each sample word. For example, the part-of-speech tag of each sample word can be obtained by performing part-of-speech tag prediction on each sample word.

[0076] The part-of-speech (POS) can include, but is not limited to, nouns, verbs, adjectives, etc. By predicting the POS of each sample word, the POS of each sample word can be determined. This POS value can then be used to determine the word's ranking, thus increasing the importance of word ranking determination. Optionally, a POS model can be used to predict the POS of each sample word. POS models can include, but are not limited to, pre-trained language representation models (Bidirectional Encoder Representation from Transformers, BERT), text classification models (Text Convolutional Neural Network, TextCNN), Long Short-Term Memory (LSTM), etc.

[0077] In one embodiment, the word features of each sample word include the word category of each sample word. For example, the word category of each sample word can be obtained by performing named entity prediction on each sample word to obtain the word category of each sample word.

[0078] Named entity prediction can refer to Named Entity Recognition (NER), which identifies entities with specific meanings in text, such as names of people, places, organizations, and proper nouns. Word categories can refer to the categories of named entity words, including but not limited to categories such as people, groups, time, location, none, and brands. By identifying the word category of each sample word, the word level of each sample word in the training sample sentence can be determined, thereby improving the accuracy of word level determination. Optionally, a named entity prediction model can be used to perform named entity prediction on each sample word to obtain the word category of each sample word. Named entity prediction models can include, but are not limited to, Convolutional Neural Network (CNN), Relational Graph Convolutional Network (R-GCN), Deep Neural Networks (DNN), and so on.

[0079] In one embodiment, the word features of each sample word may include multiple features such as word frequency, inverse document frequency, inverse sentence frequency, co-occurrence frequency, word length, part-of-speech, and word category. For example, it may include any combination of two features, any combination of three features, any combination of four features, any combination of five features, etc., so that words can be analyzed from multiple dimensions to determine the word level and improve the accuracy of word level determination.

[0080] S103. Based on the word level label and word features of each sample word in the training sample sentences, train the initial word classification model to obtain the target word classification model.

[0081] In this embodiment, since obtaining the word level label of each sample word in the training sample sentence is equivalent to obtaining the real data of each sample word in the training sample sentence, as well as the word features of each sample word, the initial word classification model can be trained using the word features and word level labels of each sample word to obtain the target word classification model. The target word classification model is used to classify and predict the word level of each word contained in the sentence. By training the initial word classification model to obtain the target word classification model, the target word classification model can be used to classify the word level of each word contained in any sentence, determine the word level of each word, that is, the weight of each word, which facilitates subsequent applications.

[0082] In one embodiment, the initial word classification model can be trained as follows: based on the initial word classification model, the word features of each sample word in the training sample sentence are classified and predicted to obtain the sample output word level of each sample word in the training sample sentence; based on the difference between the sample output word level and the word level label of each sample word in the training sample sentence, the initial word classification model is trained to obtain the target word classification model.

[0083] In this model, the word ranking label for each word in the training sample sentences refers to the true value, while the output word ranking refers to the model's predicted value. By comparing the difference between the true and predicted word rankings, the loss of the initial word classification model can be determined. The model parameters are then adjusted based on this loss to reduce it, ensuring it falls below a threshold. When the loss is below the threshold, this initial word classification model is saved as the target word classification model. When the loss is greater than or equal to the threshold, the model parameters are continuously adjusted to keep the loss below the threshold. By continuously adjusting the model parameters based on the difference between the true and predicted values, the stability of the target word classification model can be improved. When any subsequent sentence is obtained, the target word classification model can be automatically used to determine the word ranking of each word in that sentence, thus determining the importance of each word.

[0084] Optionally, the target word classification model can be constructed using a distributed gradient boosting framework (Light Gradient Boosting Machine, LightGBM) based on decision tree algorithms. For example, the LightGBM framework can employ various classification models to achieve classification prediction, including but not limited to Gradient Boosting Decision Tree (GBDT) and Logistic Regression (LR). The LightGBM framework supports highly efficient parallel training, offering faster training speeds, lower memory consumption, better accuracy, and distributed processing, thus enabling rapid processing of massive amounts of data.

[0085] Optionally, when training the initial word classification model, the training sample sentences can be pre-segmented to obtain each sample word in the training sample words, and feature extraction can be performed on each sample word to obtain word features for each sample word. For example, word features may include word length, named entity category, word frequency, inverse document frequency, inverse sentence frequency, co-occurrence frequency, and word level label, as shown in Table 1. Thus, the word features and word level labels in Table 1 can be used to train the initial word classification model to obtain the target word classification model.

[0086] Table 1

[0087]

[0088] In this embodiment, the word level labels and word features of the training sample sentences can be obtained using the above method. These word level labels and word features can be used to train an initial word classification model. For example, each word in the training sample sentence can be split into three sample words: "international cultural exchange," "culture," and "exchange." Each sample word has corresponding word features and word level labels. Further, LightGBM is used for model training, and the training loss can be multi-class softmax. Multi-class softmax can output the probability of a sample word belonging to each of the multiple word levels. For example, word levels correspond to four tiers: 3, 2, 1, and 0. In the training sample sentence "international cultural exchange," multi-class softmax can output the probabilities of the word "international," "culture," and "exchange" belonging to the four tiers. For example, if the probability of "international" being at tier 3 is greater than the probability of it being at the other three tiers, then the word level of "international" can be tier 3. For example, if the probability of "culture" being in level 1 is greater than the probability of it being in any of the other three levels, then the word level for "culture" can be level 1. Similarly, if the probability of "communication" being in level 3 is greater than the probability of it being in any of the other three levels, then the word level for "communication" can be level 3. For instance, the initial word classification model outputs "international cultural communication" with three sample words in levels 3, 1, and 3, respectively. Further, optionally, the levels can be mapped to weight values. For example, level 3 can be mapped to a weight value of 1, level 2 to a weight value of 0.5, level 1 to a weight value of 0.3, level 0 to a weight value of 0, and so on. Mapping levels to weight values ​​facilitates the subsequent determination of word importance; that is, the larger the weight value, the greater the word's importance, and the smaller the weight value, the less important the word.

[0089] Optionally, after obtaining the word level labels and word features corresponding to the training sample sentences, the tree model in the LightGBM framework has many split points. Each split point will split; for example, the first split point selects points with word lengths greater than 1 and less than 3 for splitting. The second split point, based on the first split point, splits based on inverse document frequency, for example, selecting points with inverse document frequencies greater than 6 and less than 8 for a second split, and so on, until finally stopping at a certain split point, which is called a leaf node. A loss can be trained based on the values ​​of the leaf nodes. In this process, by calculating the relationship between the loss and the split points, a decision tree can be obtained. Each point on the decision tree is a splitting condition for a feature. By obtaining the word level labels of each sample word in the training sample sentences and fitting the word level labels based on the word features of each sample word, the word features are correlated to a certain extent. Therefore, the target word classification model can be used to classify and predict the word level of words based on the word features.

[0090] In this embodiment, the number of training sample sentences can be multiple. This embodiment mainly uses one training sample sentence as an example. The word level label and word feature of each sample word in the training sample sentence are determined. The word level label and word feature corresponding to one training sample sentence are used to train the initial word classification model. For other training sample sentences in multiple training sample sentences, this method can also be used to process other training sample sentences to train the initial word classification model. This embodiment will not describe this in detail.

[0091] In this embodiment, word weight training data can be automatically constructed, that is, the label of each sample word in the training sample sentence can be automatically constructed. Compared with manually labeling each sample word in the training sample, this can reduce the manual labeling cost and improve the labeling efficiency. In addition, since the distribution of documents under the search term is considered when determining the word level label in this embodiment, the labeled word level label reflects the weight of the word more reasonably, and therefore the final word level label is more accurate.

[0092] In this embodiment, training sample sentences are obtained, and word tagging is performed on the training sample sentences based on the relevance between the training sample sentences and documents in the document database to obtain the word level label for each sample word in the training sample sentences. Since the relevance between documents in the document database and the training sample sentences is combined to tag each sample word in the training sample sentences, the accuracy of word tagging can be improved, thereby improving the accuracy of word level label determination. By obtaining the word features of each sample word, which reflect the frequency of occurrence and word category information of each sample word, an initial word classification model is trained based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model. Subsequently, the target word classification model can be used to classify the word level of each word contained in any sentence, improving the efficiency of word level determination. By using the target word classification model to classify words and thus perform word tagging, word tagging efficiency can be improved without the need for manual classification and tagging. Furthermore, by combining information such as word frequency and word category to classify word levels, the accuracy of word classification can be improved, thereby improving the accuracy of determining word levels.

[0093] Further, please see Figure 4 , Figure 4 This is a flowchart illustrating a word processing method provided in an embodiment of this application; as shown below. Figure 4 As shown, this word processing method can be applied to computer devices, and the word processing method includes, but is not limited to, the following steps:

[0094] S201, Obtain the word features of each word among the multiple words contained in the query statement.

[0095] In this embodiment of the application, a target word classification model can be trained in the aforementioned manner. Therefore, the target word classification model can be used to classify and predict each word in the query statement, thereby obtaining the word level of each word in the query statement.

[0096] Specifically, the query statement can be segmented to obtain multiple words contained in the query statement. Each word in these multiple words can then be processed to obtain word features for each word. These word features may include one or more of the following: word frequency, inverse document frequency, inverse statement frequency, co-occurrence frequency, word length, part-of-speech tag, and word category. The specific method for obtaining these word features can refer to the method for obtaining the word features of each sample word in step S102 above, which will not be described in detail here.

[0097] S202, Based on the target word classification model, classify and predict the word features of each word in the query statement to obtain the word level of each word in the query statement.

[0098] In this embodiment, based on the target word classification model, the word features of each word in the query statement are classified and predicted. This determines the probability of each word belonging to multiple word levels, and the word level corresponding to the highest probability is determined as the word level of each word. For example, multiple word levels include first level, second level, third level, and fourth level. The importance of the word corresponding to the first level is higher than that of the word corresponding to the second level, the importance of the word corresponding to the second level is higher than that of the word corresponding to the third level, and the importance of the word corresponding to the third level is higher than that of the word corresponding to the fourth level. The higher the importance of a word, the greater its corresponding word weight can be. The lower the importance of a word, the smaller its corresponding word weight can be.

[0099] S203: Based on the word level of each word in the query statement, perform word adjustment processing on the query statement to obtain the search statement, and perform document search in the document database based on the search statement.

[0100] In this embodiment, since the word level of each word in the query statement is obtained, the query statement can be adjusted based on the word level of each word to obtain the search statement. A document search is then performed in the document database based on the search statement. The word adjustment process includes at least one of word sorting or word filtering. Word sorting can refer to sorting multiple words according to their word levels, while word filtering can refer to filtering out words in the query statement whose word levels are less than or equal to a level threshold.

[0101] In one embodiment, the word adjustment process includes word sorting, which sorts the words in the query statement based on the word level of each word to obtain the search statement. Each word in the search statement corresponds to a sequence number, and the smaller the sequence number, the higher the importance level of the corresponding word. Based on the sequence number corresponding to each word in the search statement, a document search is performed in the document database.

[0102] In this embodiment, by sorting each word based on its level, document searches can be performed using the corresponding word's sequence number, i.e., by combining the importance of the word with the document search. This results in more search results for words with higher levels of importance, thereby improving search accuracy. For words with lower levels of importance, fewer results can be retrieved, or their search results can be displayed later in the search results, making it easier to quickly identify the search results for words with higher levels of importance and improving data viewing efficiency.

[0103] In one embodiment, the word adjustment process includes word filtering, which can filter the words in the query statement based on the word level of each word to obtain the search statement; and perform a document search in the document database based on the search statement.

[0104] Filtering words in a query can mean removing words whose level is less than or equal to a level threshold, while words in the search query must have a level greater than the threshold. By filtering words based on their level, lower-level words can be removed, allowing for the use of higher-level words in subsequent searches, thus improving search efficiency.

[0105] In this embodiment, training sample sentences are obtained, and word annotation is performed on the training sample sentences based on the relevance between the training sample sentences and documents in the document database to obtain a word level label for each sample word in the training sample sentences. Since the relevance between documents in the document database and the training sample sentences is combined to annotate each sample word in the training sample sentences, the accuracy of word annotation can be improved, thereby improving the accuracy of word level label determination. By obtaining the word features of each sample word, which reflect the frequency of occurrence and word category information of each sample word, an initial word classification model is trained based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model. This target word classification model can then be used to classify the word level of each word in any sentence, improving the efficiency of word level determination. By using the target word classification model to classify words and thus annotate them, word annotation efficiency can be improved without manual classification and annotation. Furthermore, since the word level classification is based on multiple aspects such as word frequency and word category, the accuracy of word classification can be improved, thereby improving the accuracy of word level determination. Furthermore, sorting each word in the query based on its ranking improves search accuracy. Also, filtering based on word ranking removes less important words, thus improving search efficiency.

[0106] The methods of the embodiments of this application have been described above, and the apparatus of the embodiments of this application will be described below.

[0107] See Figure 5 , Figure 5This is a schematic diagram of the composition structure of a word processing device provided in an embodiment of this application. The word processing device can be deployed on a computer device; the word processing device can be used to execute the corresponding steps in the word processing method provided in the embodiment of this application. The word processing device 50 includes:

[0108] The tag acquisition unit 501 is used to acquire training sample sentences, and based on the relevance between the training sample sentences and documents in the document database, to perform word tagging processing on the training sample sentences to obtain the word level tag of each sample word in the training sample sentences.

[0109] The feature acquisition unit 502 is used to acquire the word features of each sample word. The word features of each sample word are used to reflect the frequency of occurrence of each sample word and the word category information of each sample word.

[0110] The model training unit 503 is used to train an initial word classification model based on the word level label and word features of each sample word in the training sample sentence, and obtain a target word classification model. The target word classification model is used to classify the word level of each word contained in the sentence.

[0111] Optionally, the label acquisition unit 501 is specifically used for:

[0112] The training sample sentence is segmented into words to obtain multiple sample words in the training sample sentence.

[0113] Each of the multiple sample words is subjected to word removal processing to obtain the sample words to be processed associated with each sample word;

[0114] The document database is used to determine the document corresponding to each sample word and the sample word to be processed associated with it. Each sample word and the sample word to be processed associated with it corresponds to at least one document.

[0115] Obtain the relevance between the training sample sentence and the document corresponding to the sample word to be processed associated with each sample word, and determine the word level label of each sample word among the multiple sample words based on the relevance.

[0116] Optionally, any one of the multiple sample words is represented as the target sample word; the tag acquisition unit 501 is specifically used for:

[0117] The target sample word is subjected to word removal processing to obtain the unprocessed sample words associated with the target sample word. The unprocessed sample words associated with the target sample word include sample words other than the target sample word among the multiple sample words.

[0118] Optionally, the label acquisition unit 501 is specifically used for:

[0119] Obtain the number of words in the document corresponding to the sample words in the training sample sentence and the sample words to be processed associated with each sample word that have the same word;

[0120] The number of times the target operation was performed on the document corresponding to the unprocessed sample word associated with each sample word when searching for the training sample sentence in the historical search records;

[0121] Obtain the sentence category to which the training sample sentence belongs, and the document category to which the document corresponding to the sample word to be processed associated with each sample word belongs, and obtain the category similarity between the sentence category and the document category;

[0122] Based on the number of words, the number of operations, and the category similarity, determine the associated documents of each sample word from the documents corresponding to the sample words to be processed associated with each sample word;

[0123] The ratio between the number of associated documents for each sample word and the total number of documents corresponding to the sample words to be processed associated with each sample word is obtained, and the word level label of each sample word among the multiple sample words is determined based on the ratio.

[0124] Optionally, the word features of each sample word include the word frequency of each sample word, the inverse document frequency of each sample word, the inverse sentence frequency of each sample word, the co-occurrence frequency of each sample word, the word length of each sample word, the part-of-speech of each sample word, and the word category of each sample word.

[0125] The feature acquisition unit 502 is specifically used for:

[0126] Obtain the number of times each sample word appears in the document database, as well as the total number of words in the document database. Determine the word frequency of each sample word as the ratio between the number of times the word appears and the total number of words.

[0127] Obtain the number of documents in the document database that contain each sample word, and the total number of documents in the document database. Determine the ratio between the total number of documents and the number of documents as the inverse document frequency of each sample word. The inverse document frequency is used to reflect the importance of each sample word in the document.

[0128] Obtain the number of sentences containing each sample word in the document database, and the total number of sentences in the document database. Determine the ratio between the total number of sentences and the number of sentences as the inverse sentence frequency of each sample word. The inverse sentence frequency is used to reflect the importance of each sample word in the training sample sentences.

[0129] The statistical document database contains documents associated with each sample word's statements. A first number of documents in the associated documents where the target operation was performed is obtained. A second number of documents containing each sample word at the target position within the documents where the target operation was performed is determined. The ratio between the second number and the first number is determined as the co-occurrence frequency of each sample word. This co-occurrence frequency reflects the importance of each sample word.

[0130] The number of characters contained in each sample word is obtained as the word length of each sample word;

[0131] Perform part-of-speech prediction on each sample word to obtain the part-of-speech of each sample word;

[0132] Named entity prediction is performed on each sample word to obtain the word category of each sample word.

[0133] Optionally, the model training unit 503 is specifically used for:

[0134] Based on the initial word classification model, the word features of each sample word in the training sample sentence are classified and predicted to obtain the sample output word level of each sample word in the training sample sentence;

[0135] Based on the difference between the word level of the output word and the word level label of each word in the training sample sentence, the initial word classification model is trained to obtain the target word classification model.

[0136] Optionally, the word processing device 50 further includes a model application unit 504, which is specifically used for:

[0137] Obtain the word features of each word in the multiple words contained in the query statement;

[0138] Based on the target word classification model, the word features of each word in the query statement are classified and predicted to obtain the word level of each word in the query statement;

[0139] The query statement is adjusted based on the word ranking of each word to obtain the search statement. A document search is then performed on the document database based on the search statement. The word adjustment process includes at least one of word sorting or word filtering.

[0140] It should be noted that, Figure 5 For any content not mentioned in the corresponding embodiments, please refer to the description of the method embodiments, which will not be repeated here.

[0141] In this embodiment, training sample sentences are obtained, and word tagging is performed on the training sample sentences based on the relevance between the training sample sentences and documents in the document database to obtain the word level label for each sample word in the training sample sentences. Since the relevance between documents in the document database and the training sample sentences is combined to tag each sample word in the training sample sentences, the accuracy of word tagging can be improved, thereby improving the accuracy of word level label determination. By obtaining the word features of each sample word, which reflect the frequency of occurrence and word category information of each sample word, an initial word classification model is trained based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model. Subsequently, the target word classification model can be used to classify the word level of each word contained in any sentence, improving the efficiency of word level determination. By using the target word classification model to classify words and thus perform word tagging, word tagging efficiency can be improved without the need for manual classification and tagging. Furthermore, by combining information such as word frequency and word category to classify word levels, the accuracy of word classification can be improved, thereby improving the accuracy of determining word levels.

[0142] See Figure 6 , Figure 6 This is a schematic diagram of the structural composition of a computer device provided in an embodiment of this application. For example... Figure 6 As shown, the computer device 60 described above may include: a processor 601, a network interface 604, and a memory 605. Furthermore, the computer device 60 may also include: a user interface 603, and at least one communication bus 602. The communication bus 602 is used to enable communication between these components. The user interface 603 may include a display screen and a keyboard; optionally, the user interface 603 may also include a standard wired interface or a wireless interface. The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 605 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 605 may also be at least one storage device located remotely from the aforementioned processor 601. Figure 6 As shown, the memory 605, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0143] exist Figure 6In the computer device 60 shown, the network interface 604 provides network communication functionality; the user interface 603 is mainly used to provide an input interface for the user; and the processor 601 can be used to call the device control application stored in the memory 605 to achieve:

[0144] Obtain training sample sentences, and based on the relevance between the training sample sentences and documents in the document database, perform word tagging on the training sample sentences to obtain the word level label for each sample word in the training sample sentences.

[0145] The word features of each sample word are obtained, and the word features of each sample word are used to reflect the frequency of occurrence of each sample word and the word category information of each sample word;

[0146] Based on the word level label and word features of each sample word in the training sample sentence, an initial word classification model is trained to obtain a target word classification model. This target word classification model is used to classify the word level of each word contained in the sentence.

[0147] It should be understood that the computer device 60 described in the embodiments of this application can perform the foregoing... Figure 2 and Figure 4 The description of the word processing method in the corresponding embodiments can also be performed as described above. Figure 5 The description of the word processing device in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0148] In this embodiment, training sample sentences are obtained, and word tagging is performed on the training sample sentences based on the relevance between the training sample sentences and documents in the document database to obtain the word level label for each sample word in the training sample sentences. Since the relevance between documents in the document database and the training sample sentences is combined to tag each sample word in the training sample sentences, the accuracy of word tagging can be improved, thereby improving the accuracy of word level label determination. By obtaining the word features of each sample word, which reflect the frequency of occurrence and word category information of each sample word, an initial word classification model is trained based on the word level labels and word features of each sample word in the training sample sentences to obtain a target word classification model. Subsequently, the target word classification model can be used to classify the word level of each word contained in any sentence, improving the efficiency of word level determination. By using the target word classification model to classify words and thus perform word tagging, word tagging efficiency can be improved without the need for manual classification and tagging. Furthermore, by combining information such as word frequency and word category to classify word levels, the accuracy of word classification can be improved, thereby improving the accuracy of determining word levels.

[0149] This application also provides a computer-readable storage medium storing a computer program. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the methods described in the foregoing embodiments. The computer can be part of the aforementioned computer device, such as the processor 601 described above. As an example, the program instructions can be deployed on a single computer device for execution, or deployed on multiple computer devices located in one location for execution, or executed on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can form a blockchain network.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0151] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A word processing method, characterized in that, The method includes: The statement to obtain training samples; Based on the relevance between the training sample statements and documents in the document database, word tagging is performed on the training sample statements to obtain a word level label for each sample word in the training sample statements. This includes: performing word segmentation on the training sample statements to obtain multiple sample words in the training sample statements, where any one of the multiple sample words is represented as a target sample word; performing word removal on the target sample words to obtain unprocessed sample words associated with the target sample words, where the unprocessed sample words associated with the target sample words include sample words other than the target sample words from the multiple sample words; determining the documents corresponding to the unprocessed sample words associated with each sample word from the document database, where each unprocessed sample word associated with each sample word corresponds to at least one document; obtaining the relevance between the training sample statements and the documents corresponding to the unprocessed sample words associated with each sample word, and determining the word level label for each sample word in the multiple sample words based on the relevance; The word features of each sample word are obtained, and the word features of each sample word are used to reflect the frequency of occurrence of each sample word and the word category information of each sample word; Based on the word level label and word features of each sample word in the training sample sentences, an initial word classification model is trained to obtain a target word classification model, which is used to classify the word level of each word contained in the sentence.

2. The method according to claim 1, characterized in that, The step of obtaining the relevance between the training sample sentences and the documents corresponding to the unprocessed sample words associated with each sample word, and determining the word level label of each sample word among the plurality of sample words based on the relevance, includes: Obtain the number of words with the same word in the document corresponding to the sample word to be processed associated with each sample word in the training sample sentence; The number of times the target operation was performed on the document corresponding to the sample word to be processed associated with each sample word when searching the training sample statement in the historical search records; Obtain the sentence category to which the training sample sentence belongs, and the document category to which the document corresponding to the sample word to be processed associated with each sample word belongs, and obtain the category similarity between the sentence category and the document category; Based on the number of words, the number of operations, and the category similarity, the associated documents of each sample word are determined from the documents corresponding to the unprocessed sample words associated with each sample word; The ratio between the number of associated documents for each sample word and the total number of documents corresponding to the unprocessed sample words associated with each sample word is obtained, and the word level label of each sample word among the multiple sample words is determined based on the ratio.

3. The method according to claim 1, characterized in that, The word features of each sample word include the word frequency, the inverse document frequency, the inverse sentence frequency, the co-occurrence frequency, the word length, the part-of-speech, and the word category of each sample word. The step of obtaining the word features of each sample word includes: The frequency of each sample word in the document database and the total number of words in the document database are obtained, and the ratio between the frequency of each word and the total number of words is determined as the word frequency of each sample word. The number of documents containing each sample word in the document database and the total number of documents in the document database are obtained. The ratio between the total number of documents and the number of documents is determined as the inverse document frequency of each sample word. The inverse document frequency is used to reflect the importance of each sample word in the document. The number of sentences containing each sample word in the document database and the total number of sentences in the document database are obtained. The ratio between the total number of sentences and the number of sentences is determined as the inverse sentence frequency of each sample word. The inverse sentence frequency is used to reflect the importance of each sample word in the training sample sentences. The document database contains documents associated with statements containing each sample word. A first number of documents in the associated documents where the target operation is performed is obtained. A second number of documents containing each sample word at a target position within the documents where the target operation is performed is determined. The ratio between the second number and the first number is determined as the co-occurrence frequency of each sample word. The co-occurrence frequency reflects the importance of each sample word. The number of characters contained in each sample word is obtained as the word length of each sample word; Part-of-speech (POS) prediction is performed on each sample word to obtain the POS of each sample word; Named entity prediction is performed on each sample word to obtain the word category of each sample word.

4. The method according to any one of claims 1-3, characterized in that, The process of training an initial word classification model based on the word level label and word features of each sample word in the training sample sentences, and then obtaining the target word classification model, includes: Based on the initial word classification model, the word features of each sample word in the training sample sentence are classified and predicted to obtain the sample output word level of each sample word in the training sample sentence; Based on the difference between the word level of the output sample and the word level label of each sample word in the training sample sentence, the initial word classification model is trained to obtain the target word classification model.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the word features of each word in the multiple words contained in the query statement; Based on the target word classification model, the word features of each word in the query statement are classified and predicted to obtain the word level of each word in the query statement; The query statement is adjusted based on the word level of each word in the query statement to obtain the search statement. A document search is then performed in the document database based on the search statement. The word adjustment process includes at least one of word sorting or word filtering.

6. A word processing device, characterized in that, The device includes: A tag acquisition unit is used to acquire training sample sentences; based on the relevance between the training sample sentences and documents in a document database, the unit performs word tagging processing on the training sample sentences to obtain a word level tag for each sample word in the training sample sentences, including: performing word segmentation processing on the training sample sentences to obtain multiple sample words in the training sample sentences, where any one of the multiple sample words is represented as a target sample word; performing word removal processing on the target sample words to obtain unprocessed sample words associated with the target sample words, where the unprocessed sample words associated with the target sample words include sample words other than the target sample words among the multiple sample words; determining the documents corresponding to the unprocessed sample words associated with each sample word from the document database, where each unprocessed sample word associated with each sample word corresponds to at least one document; acquiring the relevance between the training sample sentences and the documents corresponding to the unprocessed sample words associated with each sample word, and determining the word level tag for each sample word among the multiple sample words based on the relevance; The feature acquisition unit is used to acquire the word features of each sample word, wherein the word features of each sample word are used to reflect the frequency of occurrence of each sample word and the word category information of each sample word; The model training unit is used to train an initial word classification model based on the word level label and word features of each sample word in the training sample sentence, and to obtain a target word classification model. The target word classification model is used to classify the word level of each word contained in the sentence.

7. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store program code, and the processor is used to call the program code so that the computer device executes the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-5.

Images