Word determination and prediction model training method and device, equipment and medium

Abstract

The embodiment of the invention provides a word determination and prediction model training method and device, equipment and a medium, which are used for improving the word prediction accuracy. According to the first feature vector of the first sub-text of the first text and the second feature vector of the second sub-text, the third feature vector of the first text is comprehensively determined instead of directly determining the third feature vector based on the long first text, so that the problem of semantic forgetting of the long text is solved, and when the first prediction probability is determined, the prediction efficiency of the long text is improved. According to the third feature vector, the first word segmentation feature vector and the second word segmentation feature vector, the first prediction probability of each word segmentation in the first text is determined, so that the first word is determined according to the first prediction probability, during prediction, not only is the feature of each word segmentation considered, but also the full-text semantics of the first text is considered, and the prediction efficiency is improved. As long as feature extraction can be accurately carried out, the missing words in the first text can be accurately predicted.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for training a word determination and prediction model. Background Technology

[0002] Enabling electronic devices to hear, speak, understand, and think is a long-term goal of artificial intelligence. Machine reading comprehension refers to enabling machines to read text and then predict or answer questions related to that text. Machine reading comprehension is currently a hot research topic in the field of natural language processing and has received widespread attention. Unlike traditional question-answering systems, reading comprehension tasks focus more on understanding the text itself rather than using world knowledge or common sense to answer questions. Therefore, improving the accuracy of machine reading comprehension has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a method, apparatus, device, and medium for training a word determination and prediction model to improve the accuracy of word prediction.

[0004] Firstly, this application provides a word determination method, the method comprising:

[0005] Determine the first sub-text and the second sub-text of the first text, as well as the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text.

[0006] Based on the first feature vector and the second feature vector, determine the third feature vector of the first text;

[0007] Based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector, determine the first prediction probability of each word in the first text;

[0008] Based on the first predicted probability, a first word is determined, which is used to represent a missing word between the first sub-text and the second sub-text.

[0009] In this embodiment, the third feature vector of the first text is determined by comprehensively considering the first feature vector of the first sub-text and the second feature vector of the second sub-text, rather than directly determining the third feature vector based on the lengthy first text. This solves the problem of semantic forgetting in long texts. Furthermore, when determining the first prediction probability, the first prediction probability of each word in the first text is determined based on the third feature vector, the first segmentation feature vector of the first sub-text, and the second segmentation feature vector of the second sub-text. Thus, the first word is determined based on the first prediction probability. The prediction not only considers the features of each word itself but also the semantics of the entire first text, i.e., it considers the contextual information. There is no need to overly rely on the knowledge learned in advance by the model. As long as the feature extraction can be performed accurately, the missing words in the first text can be predicted accurately.

[0010] Secondly, this application provides a method for training a prediction model, the method comprising:

[0011] Determine the sample text and its first tag;

[0012] The third and fourth sub-texts of the sample text are determined, as well as the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text.

[0013] Based on the fourth and fifth feature vectors, the sixth feature vector of the sample text is determined;

[0014] Based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector, determine the second label of the missing words in the sample text;

[0015] The loss value is determined based on the first label and the second label, and the prediction model is trained based on the loss value.

[0016] In this embodiment, the sixth feature vector of the sample text is determined by comprehensively considering the fourth and fifth feature vectors of the third and fourth sub-texts, rather than directly based on the length of the sample text. This solves the problem of semantic forgetting in long texts. Furthermore, when determining the second label, the second label for missing words in the sample text is determined based on the sixth feature vector, the third segmentation feature vector of the third sub-text, and the fourth segmentation feature vector of the fourth sub-text. The prediction model is then trained based on the loss value between the first and second labels. During model training, not only are the individual features of each segment considered, but also the semantics of the entire sample text, i.e., contextual information. Even if the prediction model does not learn knowledge from a specific domain during training, it can still make accurate predictions as long as it can accurately extract features, thus improving prediction accuracy.

[0017] Thirdly, this application provides a word determination device, the device comprising:

[0018] The determination module is used to determine the first and second subtexts of the first text;

[0019] The feature extraction module is used to determine the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text; and to determine the third feature vector of the first text based on the first feature vector and the second feature vector.

[0020] The prediction module is used to determine the first prediction probability of each word in the first text based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector.

[0021] The determining module is further configured to determine a first word based on the first predicted probability, wherein the first word represents a missing word between the first sub-text and the second sub-text.

[0022] Fourthly, this application provides a training apparatus for a prediction model, the apparatus comprising:

[0023] The determination module is used to determine the sample text, the first tag of the sample text, and the third and fourth sub-texts of the sample text;

[0024] The feature extraction module is used to determine the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text; and to determine the sixth feature vector of the sample text based on the fourth feature vector and the fifth feature vector.

[0025] The prediction module is used to determine the second label of the missing words in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector;

[0026] The training module is used to determine a loss value based on the first label and the second label, and to train the prediction model based on the loss value.

[0027] Fifthly, embodiments of this application provide an electronic device, which includes at least a processor and a memory. The processor is used to execute a computer program stored in the memory to implement the steps of the word determination method described above, or the steps of the prediction model training method described above.

[0028] Sixthly, embodiments of this application provide a computer storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the word determination method described in any of the preceding claims, or the steps of the prediction model training method described in any of the preceding claims.

[0029] In a seventh aspect, embodiments of this application provide a computer program product comprising: computer program code, which, when executed on the electronic device, causes the electronic device to perform the steps of the word determination method described above, or the steps of the prediction model training method described above. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A schematic diagram of a word determination process provided in an embodiment of this application;

[0032] Figure 2 A schematic diagram illustrating an activation function for determining a first prediction probability, provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of a word determination process provided in an embodiment of this application;

[0034] Figure 4 A schematic diagram illustrating the training process of a prediction model provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of a word determination device provided in an embodiment of this application;

[0036] Figure 6 A schematic diagram of the structure of a training device for a prediction model provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0039] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0040] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0041] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0042] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0043] Finally, it should be noted that the embodiments of this application are only used to illustrate the technical solutions of this application, and are not intended to limit them; although this application has been described in detail with reference to the embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the embodiments, or make equivalent substitutions for some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0044] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed. Various modifications and variations can be obtained based on the derivations in each embodiment. The selection and description of the embodiments in this application are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better use the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0045] Machine reading comprehension tasks can be categorized into fill-in-the-blank, extraction, and free-response question-and-answer types. Early methods primarily relied on pre-defined rules or machine learning techniques. However, these methods required significant manual customization of rules or features, resulting in low efficiency, poor generalization ability, and an inability to extract contextual information. With the development of deep learning, deep learning-based machine reading comprehension, also known as neural machine reading comprehension, has shown advantages in capturing contextual information, significantly outperforming earlier methods. However, it still suffers from low efficiency, long processing times, and unpredictable predictions or answers.

[0046] A fill-in-the-blank machine reading comprehension task can be understood as a text in which one or more word positions are missing. That is to say, the text includes not only text but also blank character areas. The task is to select the most suitable word to fill in the missing positions to make the semantics and fluency of the entire text more reasonable.

[0047] With the development of deep learning, using deep learning to solve machine reading comprehension tasks is a common industry solution. This section introduces the BERT (Bidirectional Encoder Representations from Transformers) model, a relevant technology. In fill-in-the-blank machine reading comprehension tasks, the BERT model is generally used to predict the missing target word. The BERT model is a pre-trained language representation model based on the Transformer architecture. A pre-trained BERT model can be fine-tuned with an additional output layer, making it suitable for building state-of-the-art models across a wide range of tasks. The BERT model uses a bidirectional encoder from the Transformer. The BERT model has two main learning tasks: Next Sentence Prediction (NSP) and Masked Language Model (MLM). NSP predicts sentence pairs; given two sentences A and B, it determines whether sentence B is the next sentence after sentence A. MLM randomly selects a portion of the input text, replaces it with a mask, and then trains the model to correctly predict the masked content. The MLM task is particularly well-suited for fill-in-the-blank machine reading comprehension tasks. Specifically, in related technologies, a large amount of text is prepared, and parts of the content in each text are randomly masked. The BERT model is then trained using the masked text to obtain the trained model. When performing a fill-in-the-blank machine reading comprehension task, the first text is input into the trained BERT model, which predicts the words corresponding to the missing words in the text.

[0048] While the aforementioned methods can predict missing words, the BERT model is a pre-trained model, which is slow, inefficient, and time-consuming for recognizing long texts. Furthermore, the texts involved in machine reading comprehension may cover numerous fields, potentially leading to situations where missing words are not learned during model training. Since the model hasn't acquired this knowledge, it cannot accurately determine the missing words. Additionally, reading comprehension texts are typically entire articles, which are often quite long. Models using this technology often suffer from semantic forgetting in long texts. Therefore, to avoid the problem of semantic forgetting in long texts and over-reliance on learned model knowledge affecting the accuracy of missing word identification in fill-in-the-blank machine reading comprehension tasks, this application proposes a training method, apparatus, device, and medium for a word identification and prediction model to improve the accuracy and efficiency of word identification in fill-in-the-blank machine reading comprehension tasks. Specifically, the word determination method proposed in this application involves determining a first sub-text and a second sub-text of a first text, as well as a first feature vector of the first sub-text, a first segmentation feature vector of the first sub-text, a second feature vector of the second sub-text, and a second segmentation feature vector of the second sub-text; determining a third feature vector of the first text based on the first and second feature vectors; determining a first prediction probability for each segment in the first text based on the third feature vector, the first segmentation feature vector, and the second segmentation feature vector; and determining a first word based on the first prediction probability, wherein the first word represents a missing word between the first and second sub-texts.

[0049] The word determination method and prediction model training method provided in this application embodiment can be applied to any scenario that requires a fill-in-the-blank machine reading comprehension task, and the method can be applied to products that perform fill-in-the-blank machine reading comprehension tasks in these scenarios.

[0050] To better understand the word determination method provided in the embodiments of this application, the application environment applicable to the embodiments of this application is described below. The word determination method provided in the embodiments of this application can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. The server can be a standalone physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing.

[0051] The word determination method provided in this application will be described below with reference to various embodiments.

[0052] Figure 1 A schematic diagram of a word determination process provided in an embodiment of this application is shown, the process including:

[0053] S101: Determine the first sub-text and the second sub-text of the first text, as well as the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text.

[0054] The word determination method provided in this application is applied to an electronic device, which may be a server, PC, smart terminal, etc.

[0055] To achieve machine reading comprehension, a first text can be obtained in this embodiment. This first text is text used for machine reading comprehension. The first text can be input by the user of the electronic device, sent by another electronic device, or obtained by the electronic device from its local storage or a database. This embodiment does not limit the method of obtaining the first text.

[0056] Since the first text used for machine reading comprehension is typically a complete article, which may contain hundreds, thousands, or even tens of thousands of words, the acquired first text is usually a long text. Furthermore, current models suffer from semantic forgetting during feature extraction. Therefore, to accurately determine the feature vector of the first text and thus improve the accuracy of word identification, in this embodiment, after acquiring the first text, a first sub-text and a second sub-text can be determined. The first and second sub-texts are determined based on missing words in the first text. For example, the first sub-text is the text segment before the missing word, and the second sub-text is the text segment after the missing word. For instance, in the first text “AAAAASS______WWEECD”, the underscore between “SS” and “WW” represents a missing word in the first text, which can be understood as the original word being missing in the area corresponding to the underscore. “AAAAASS” can be identified as the first sub-text, and “WWEECD” as the second sub-text.

[0057] After determining the first sub-text and the second sub-text, a first feature vector of the first sub-text and a second feature vector of the second sub-text can be determined. The first and second feature vectors can be understood as feature vectors representing the semantic information of the corresponding sub-texts. In this embodiment, a feature extraction model can be used to extract the first feature vector of the first sub-text and the second feature vector of the second sub-text, respectively. In this embodiment, the first and second sub-texts can be input into the feature extraction model sequentially; the order in which they are input is not restricted.

[0058] Since fill-in-the-blank machine reading comprehension involves selecting appropriate words from a first text as the missing words, in this embodiment, the first segmentation feature vector of the first sub-text and the second segmentation feature vector of the second sub-text can also be determined. This facilitates the subsequent determination of the target word that may be the missing word based on the segmentation feature vector of each segment. In other words, in this embodiment, after determining the first and second sub-texts, each sub-text can be segmented, and the segmentation feature vector of each segment can be determined separately.

[0059] In this embodiment of the application, after word segmentation and determining the feature vector of each word, part-of-speech analysis can be performed on each word to delete some words with preset parts of speech, thereby reducing the number of words whose feature vectors need to be determined and improving efficiency. For example, words with the part of speech of modal particles can be deleted.

[0060] S102: Determine the third feature vector of the first text based on the first feature vector and the second feature vector.

[0061] To determine the third feature vector of the first text, in this embodiment, the first feature vector and the second feature vector can be fused to obtain the third feature vector of the first text, which is the feature vector of the entire first text. This third feature vector is used to describe the full semantics of the first text. In this embodiment, the feature vector obtained by adding the first feature vector and the second feature vector can be determined as the third feature vector.

[0062] S103: Determine the first prediction probability of each word in the first text based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector.

[0063] After determining the third feature vector, the first segmentation feature vector of the first sub-text, and the second segmentation feature vector of the second sub-text, the first prediction probability of each segment can be determined based on the third feature vector, the first segmentation feature vector, the second segmentation feature vector, and the activation function.

[0064] To consider the semantic information of the first text's context when determining the words corresponding to missing words, in this embodiment, for each word segment, the word feature vector of that segment is multiplied by a third feature vector to obtain the target word segment feature vector. After determining the target word segment feature vector for each word, the first prediction probability of each word can be determined based on the target word segment feature vector and the activation function. It should be noted that the word feature vector of each word in this embodiment includes a first word segment feature vector and a second word segment feature vector.

[0065] To facilitate understanding, when performing operations on the word segmentation feature vectors of the first and second sub-texts and the third feature vector of the first text to obtain the target word segmentation feature vector for each word, the following formula can be used for calculation:

[0066] t i =v*v i ,1≤i≤n

[0067] t j =v*v j ,1≤j≤m

[0068] Among them, t i t represents the target word segmentation feature vector in the first sub-text; j v represents the target word segmentation feature vector in the second sub-text; v represents the third feature vector of the first text; v i represents the first segmentation feature vector of the i-th segment in the first sub-text; n represents the number of segments included in the first sub-text; v j represents the second segmentation feature vector of the j-th segment in the second sub-text; m represents the number of segments included in the second sub-text.

[0069] In this embodiment, the first prediction probability of each word segment can be determined based on the following activation function:

[0070]

[0071] Among them, Z k Z represents the target segmentation feature vector of segmentation k, which is the vector obtained by multiplying the third feature vector and the segmentation feature vector of segmentation i; p represents the number of segments included in the first text; p represents the target segmentation feature vector of segmentation p; exp() represents an exponential function with the natural constant e as the base; softmax(Z) k ) represents the first predicted probability of word segmentation k.

[0072] Figure 2This is a schematic diagram illustrating an activation function for determining a first prediction probability, as provided in an embodiment of this application. Figure 2 As shown, after determining the target segmentation feature vector z for each word, the softmax activation function can be used as the basis for segmentation. Determine the first predicted probability for each word segment. Figure 2 The diagram shows the target feature vectors for three word segments: word segment 1 has a target feature vector of 2.0, word segment 2 has a target feature vector of 1.0, and word segment 3 has a target feature vector of 0.1. Based on these three target feature vectors, the softmax function determines the first prediction probability for each word segment: word segment 1 has a first prediction probability of 0.7, word segment 2 has a first prediction probability of 0.2, and word segment 3 has a first prediction probability of 0.1.

[0073] S104: Based on the first predicted probability, determine the first word, which is used to represent the missing word between the first sub-text and the second sub-text.

[0074] After obtaining the first predicted probability for each word segment, the first word can be determined based on the magnitude of each first predicted probability. This first word is the missing word between the first and second sub-texts; in other words, it is the word that should fill the blank character area in the first text. In this embodiment, the word segment corresponding to the maximum value among all first predicted probabilities can be determined as the first word.

[0075] In this embodiment, the third feature vector of the first text is determined by comprehensively considering the first feature vector of the first sub-text and the second feature vector of the second sub-text, rather than directly determining the third feature vector based on the lengthy first text. This solves the problem of semantic forgetting in long texts. Furthermore, when determining the first prediction probability, the first prediction probability of each word in the first text is determined based on the third feature vector, the first segmentation feature vector of the first sub-text, and the second segmentation feature vector of the second sub-text. Thus, the first word is determined based on the first prediction probability. The prediction considers not only the features of each word itself but also the semantics of the entire first text, i.e., it considers the contextual information. It does not require excessive reliance on the knowledge learned in advance by the model. As long as the feature extraction is accurate, the missing words in the first text can be accurately predicted, thus improving the accuracy and efficiency of machine reading comprehension.

[0076] To broaden the applicability of word determination, based on the above embodiments, in this application embodiment, if multiple first words exist in the first text, determining the first sub-text and second sub-text of the first text includes:

[0077] Select any one of the plurality of first words as the first target word, and delete the remaining first words in the first text;

[0078] Based on the position information of the first target word in the first text, the first text is divided into a first subtext and a second subtext.

[0079] Since the first text may contain multiple missing words, when multiple first words exist in the obtained first text (i.e., multiple missing words), one first word can be determined each time. For example, if there are 10 first words in the first text, then the word determination method needs to be executed 10 times. Each time the first sub-text and second sub-text of the first text are determined, any one of the multiple first words can be identified as the first target word, and the remaining first words are deleted from the first text. In other words, in each word determination process, only one missing word is retained in the first text. Ensuring that only one first target word exists in the first text, the first text is divided into a first sub-text and a second sub-text based on the position information of the first target word within the first text.

[0080] Specifically, when the first text contains three first words, in the first round of word determination, the first of the three first words can be identified as the first target word, and the remaining two first words can be deleted from the first text, connecting the text before and after each deleted first word. For example, the first text is "AAA[first word 1] BBB[first word 2] CCC[first word 3] DDD". In the first round of word determination, "first word 1" can be identified as the first target word, and "first word 2" and "first word 3" can be deleted. The first text after deleting "first word 2" and "first word 3" is "AAA[first word 1] BBBCCCDDD". Subsequently, based on the position of the first word 1 in the first text, the first text can be divided into a first subtext and a second subtext. Among them, "AAA" can be the first subtext, and "BBBCCCDDD" can be the second subtext.

[0081] After the word corresponding to the first word 1 is determined in the first round, the determined word can be filled into the original first text, that is, the first text received at the beginning. Assume that the word corresponding to the determined first word 1 is "MMMM". Then, the first text filled with the determined word is "AAAMMMMBBB[first word 2]CCC[first word 3]DDD".

[0082] In the second round of word determination, the first word 2 can be identified as the first target word, and "first word 3" can be deleted. The first text after deleting "first word 3" is "AAAMMMMBBB[first word 2]CCCDDD". Subsequently, based on the position information of "first word 2" in this first text, the first text can be divided into a first subtext and a second subtext. Among them, "AAAMMMMBBB" can be the first subtext, and "CCCDDD" can be the second subtext. The principle of subsequent rounds of word determination is similar to the above process, and will not be repeated here.

[0083] To further improve the efficiency of word identification, based on the above embodiments, in this embodiment, determining the first word segmentation feature vector of the first sub-text includes:

[0084] If the first word is found in the first word segmentation set, then the feature vector of the word that matches the first word in the first word segmentation set is determined as the feature vector of the first word segmentation.

[0085] The first word segmentation feature vector has the same dimension as the third feature vector.

[0086] Since the feature vectors of different word segments are determined by the word segments themselves and are independent of the context in which the word segments are located, meaning that the feature vectors of the same word segment are consistent across different first texts, a first word segmentation set can be pre-stored in this embodiment. This first word segmentation set contains different word segments and their corresponding feature vectors. When determining the first word segmentation set, different word segments can be pre-collected, and the feature vector of each word can be determined and stored.

[0087] Specifically, when determining the first word segmentation set, a pre-collected text set can be obtained. Each text in this set is segmented, and all resulting word segments are deduplicated to obtain a word set θ. Then, the trained feature extraction model is used to determine the feature vector corresponding to each word segment, and the resulting word segmentation feature vectors are saved to the first word segmentation set, which can be represented as {k1:v...} k1 ,k2:v k2 ,…}, where k1, k2, … represent word segments in word set θ, v k1 v k2 , ... are the feature vectors corresponding to word segmentation k1, k2, ... respectively.

[0088] It should be noted that, to ensure the accuracy of subsequent word determination, the word segmentation feature vectors stored in the first word segmentation set can be guaranteed to have the same dimension as the third feature vector of the first text. For example, the same feature extraction model can be used to determine the third feature vector and the word segmentation feature vector of any word. Since in this embodiment, the third feature vector is determined by fusing the first feature vector of the first sub-text and the second feature vector of the second sub-text, the same feature extraction model can be used to determine the first feature vector, the second feature vector, and the word segmentation feature vector to ensure consistency of dimensions as much as possible.

[0089] In this embodiment, the process of determining the first segmentation feature vector of a first sub-text is described. If a first segment is found in the first segmentation set, the feature vector of the segment that matches the first segment in the first segmentation set is determined as the first segmentation feature vector. Here, the first segment is any segment in the first sub-text. That is, when determining the first segmentation feature vector of any segment, the system checks whether the feature vector corresponding to that segment is pre-stored in the first segmentation set. If it is indeed stored, no feature extraction step is needed. If the first segment is not included in the first segmentation set, it means that the first segment has not been preprocessed, and the first segment can be processed in real time to determine its first segmentation feature vector.

[0090] It should be noted that in this embodiment, the first word segmentation feature vector and the third feature vector have the same dimension. The purpose of doing so is to avoid vector offset between the first word segmentation feature vector and the third feature vector of the first text as much as possible, that is, to keep the first word segmentation feature vector and the third feature vector in the same vector space, thereby improving the accuracy of word determination.

[0091] This application embodiment reduces the computational load in the word determination process by pre-obtaining the feature vectors of some word segments, thereby improving the efficiency of word determination.

[0092] To further improve the accuracy of word identification, based on the above embodiments, in this embodiment, determining the third feature vector of the first text according to the first feature vector of the first sub-text and the second feature vector of the second sub-text includes:

[0093] The average vector of the first feature vector and the second feature vector is taken as the third feature vector.

[0094] In this embodiment, when determining the third feature vector of the first text, the average vector of the first feature vector of the first sub-text and the second feature vector of the second sub-text can be used as the third feature vector of the first text. For ease of understanding, the following explanation is provided in conjunction with the formula:

[0095]

[0096] Where v represents the third feature vector of the first text; v up v represents the first feature vector of the first subtext; down This represents the second feature vector of the second subtext.

[0097] In determining the third feature vector of the first text, this embodiment does not directly extract features from the lengthy first text. Instead, it extracts feature vectors from each of its sub-texts and then fuses these feature vectors to obtain the third feature vector of the first text. Furthermore, the fusion of feature vectors from all sub-texts does not employ the traditional direct concatenation method, thus shortening the length of the third feature vector. The method for determining the third feature vector of the first text provided by this embodiment effectively solves the problem of semantic forgetting in long texts, improves text representation capabilities, and consequently enhances the accuracy of word identification.

[0098] To further improve the accuracy of word identification, based on the above embodiments, in this embodiment, determining the first word based on the first predicted probability includes:

[0099] Count the number of times each word appears in the first text;

[0100] The first word is determined based on the frequency of occurrence of each word and its corresponding first prediction probability.

[0101] To further improve the accuracy of word identification, when determining the first word based on each first predicted probability, the frequency of each word occurrence in the first text can be counted, and the first word can be determined based on the frequency of each word occurrence and its corresponding first predicted probability. In this embodiment, the target probability of each word can be determined based on its frequency of occurrence and its corresponding first predicted probability. After determining the target probability of each word, the first word can be determined based on the magnitude of each target probability. In this embodiment, the word corresponding to the maximum target probability can be determined as the first word.

[0102] Specifically, suppose the first text contains x segments, namely K1, K2, ..., K... x The first prediction probabilities corresponding to each word segment are p1, p2, ..., p1, p2, ..., p2, p3, p4, p5, p6, p7, p8, p9, p1, p1, p1, p2 ... x And the number of occurrences of each word in the first text are d1, d2, ..., d1, d2, ..., d2, respectively. x Therefore, the target probability of the i-th word segment can be expressed as:

[0103] P i=d i *p i (1≤i≤x)

[0104] Among them, P i d represents the target probability of the i-th word segmentation; i p represents the number of times the i-th word appears in the first text; i represents the first predicted probability of the i-th word; x represents the number of words included in the first text.

[0105] In this embodiment of the application, the first word is not determined directly based on the magnitude of the first predicted probability, but rather the number of times each word appears in the first text is considered, which further improves the accuracy of word determination.

[0106] To further improve the accuracy of word identification, based on the above embodiments, in this embodiment of the application, the method further includes a prediction model, which includes a feature extraction layer and a feature fusion layer;

[0107] Determining the first feature vector of the first sub-text and the second feature vector of the second sub-text includes:

[0108] The first sub-text and the second sub-text are input into the feature extraction layer for feature extraction to obtain the first feature vector and the second feature vector.

[0109] Determining the third feature vector of the first text based on the first feature vector and the second feature vector includes:

[0110] The first feature vector and the second feature vector are input into the feature fusion layer to obtain the third feature vector.

[0111] In this embodiment, a prediction model can also be used in the word determination process. This prediction model includes a feature extraction layer and a fusion layer. The feature extraction layer can be used to determine a first feature vector of the first sub-text and a second feature vector of the second sub-text. In this embodiment, the first and second sub-texts can be input into the feature extraction layer of the prediction model for feature extraction to obtain the first feature vector of the first sub-text and the second feature vector of the second sub-text.

[0112] It should be noted that when determining the first and second feature vectors, the first sub-text can be input separately into the feature extraction layer. For example, the first feature vector of the first sub-text can be extracted first, and then the second feature vector of the second sub-text can be extracted. Alternatively, the first and second sub-texts can be input into the feature extraction layer simultaneously, and the two texts can be distinguished at the input to obtain both the first and second feature vectors.

[0113] Specifically, this feature extraction layer can deploy a feature extraction sub-model, which determines the first and second feature vectors. This feature extraction sub-model can be a Long Short-Term Memory (LSTM) network or ELMO (Embeddings from Language Models). LSTM is a type of recurrent neural network designed to address the long-term dependency problem inherent in general recurrent neural networks, and it performs better on longer sequences.

[0114] After obtaining the first and second feature vectors, when determining the third feature vector of the first text based on the first and second feature vectors, feature fusion can be performed based on the fusion layer of the prediction model to determine the third feature vector. In this embodiment, the first and second feature vectors can be input into the feature fusion layer to obtain the third feature vector. The feature fusion layer can determine the third feature vector by adding the first and second feature vectors, or it can determine the third feature vector by averaging the first and second feature vectors.

[0115] To further improve the accuracy of word identification, based on the above embodiments, in this embodiment of the application, the method further includes a prediction model, which includes a classification layer;

[0116] The step of determining the first predicted probability of each word in the first text based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector includes:

[0117] The third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector are input into the classification layer to obtain the first predicted probability of each word in the first text.

[0118] In this embodiment, a prediction model can also be used in the process of determining words. This prediction model includes a classification layer, which can be used to determine the first predicted probability of each word in the first text. In this embodiment, the determined third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector can be input into the classification layer to obtain the first predicted probability of each word in the first text.

[0119] Specifically, the classification layer can be determined based on the first prediction probability determination process described in the above embodiments. Since the above embodiments have been described in detail, the embodiments of this application will not repeat them.

[0120] The process of word determination is explained below with reference to a specific example. Figure 3 This is a schematic diagram illustrating a word determination process provided in an embodiment of this application. After obtaining the first text, based on the position information of the missing word within the first text, the first text is divided into an upper half and a lower half. The upper half can be referred to as the first sub-text, and the lower half as the second sub-text. In other words, the first sub-text refers to the text preceding the missing word, and the second sub-text refers to the text following the missing word. The first and second sub-texts are then input sequentially into the feature extraction layer of the prediction model. It should be noted that the first and second sub-texts are not input simultaneously. Figure 3 For ease of understanding, they are drawn together.

[0121] Suppose that the first sub-text consists of n words and the second sub-text consists of m words. Figure 3 w1, w2, ..., w n w represents the feature vector of the first, second, ..., last word segment in the first sub-text, respectively. 1 w 2 ... w m These represent the feature vectors of the first, second, ..., last word segment in the second sub-text, respectively. The feature vector of each word segment can be determined using Word2Vec. Word2Vec is a tool for generating word vectors and is a type of language model that learns semantic knowledge from large amounts of text corpora in an unsupervised manner; it is widely used in natural language processing. The feature vector of each word segment in the sub-text is input to the feature extraction layer because the accuracy of directly processing text is not high; it is necessary to convert the text into vectors.

[0122] The feature extraction sub-model deployed in the feature extraction layer of the prediction model can be LSTM or ELOM. It can determine the first feature vector V of the first sub-text based on the order in which each word appears in the first text. up The third feature vector V of the second subtext down Simultaneously, the first segmentation feature vector of each word in the first sub-text and the second segmentation feature vector of each word in the second sub-text are determined. In this embodiment, the first segmentation feature vector of each word in the first sub-text can be represented as v1, v2, ..., v n The second segmentation feature vector of each segment in the second sub-text can be represented as v 1 v 2 ..., v mBefore being input into the feature extraction layer, the feature vector of each word segment has already been determined. Subsequent feature extraction layers further determine either the first or second feature vector for each word segment to ensure that the dimensions of the first and second feature vectors are similar to the first feature vector V. up Second eigenvector V down They have the same dimensions.

[0123] After obtaining the first feature vector of the first sub-text and the second feature vector of the second sub-text, the first feature vector and the second feature vector are input into the feature fusion layer. The feature fusion layer fuses the first feature vector and the second feature vector to obtain the third feature vector v of the first text.

[0124] The third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector are then input into the classification layer. The classification layer multiplies the feature vector of each word segmentation (including the first and second word segmentation feature vectors) with the third feature vector v to obtain the target word segmentation feature vectors t1, t2, ..., t for each word segmentation. n , t 1 , t 2 , ..., t m Where t1 represents the target word segmentation feature vector of the first word segmentation in the first sub-text, t2 represents the target word segmentation feature vector of the second word segmentation in the first sub-text, and t n t represents the target word segmentation feature vector of the nth word in the first sub-text. 1 t represents the target word segmentation feature vector of the first word in the second sub-text. 2 t represents the target word segmentation feature vector of the second word in the second sub-text. m This represents the target word segmentation feature vector of the m-th word in the second sub-text.

[0125] After obtaining the target word segmentation feature vector for each word, the first predicted probability of each word can be determined based on the softmax activation function. Then, the first word is determined according to the magnitude of each first predicted probability.

[0126] Based on the above embodiments, this application also provides a method for training a prediction model. Figure 4 This application provides a schematic diagram of a training process for a prediction model, which includes:

[0127] S401: Determine the sample text and the first tag of the sample text.

[0128] The prediction model training method provided in this application is applied to an electronic device, such as a server or PC. The electronic device used to train the prediction model can be the same as the electronic device used for word determination, or it can be a different electronic device.

[0129] To obtain a prediction model with higher semantic matching accuracy, in this embodiment, a sample text and a first label for the sample text can be determined. The first label is used to identify the standard word corresponding to a missing word in the sample text; that is, the standard word is the word that should be filled in the missing word position in the sample text.

[0130] To facilitate subsequent training of the prediction model, a training sample set is pre-configured in this embodiment. This training sample set includes multiple sample texts, and the prediction model can be trained based on each sample text in the training sample set. Each sample text includes missing words, and each sample text corresponds to a first label. The missing words in the sample text can be one or more.

[0131] When determining the sample text and its first label, a sample text can be randomly selected from the training sample set, and the first label saved for that sample text in the training sample set can be obtained.

[0132] S402: Determine the third and fourth sub-texts of the sample text, the fourth feature vector of the third sub-text, the third segmentation feature vector of the word segmentation of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth segmentation feature vector of the word segmentation of the fourth sub-text.

[0133] In order to train the initial prediction model, in this embodiment of the application, the determined sample text and the corresponding first label can be input into the prediction model, and the prediction model can predict the missing words.

[0134] In the embodiments of this application, the third and fourth sub-texts of the sample text can be determined, as well as the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text.

[0135] It should be noted that the method for determining the third sub-text, the fourth sub-text, the fourth feature vector, the fifth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector is similar in principle to the method for determining the first sub-text, the second sub-text, the first feature vector, the second feature vector, the first word segmentation feature vector, and the second word segmentation feature vector described in the above embodiments. Since the above embodiments have been described in detail, the embodiments of this application will not repeat them. Those skilled in the art can refer to the above embodiments.

[0136] S403: Determine the sixth feature vector of the sample text based on the fourth feature vector and the fifth feature vector.

[0137] After determining the fourth and fifth feature vectors, the sixth feature vector of the sample text can be determined by fusing the fourth and fifth feature vectors.

[0138] It should be noted that the principle of determining the sixth feature vector of the sample text is similar to the principle of determining the third feature vector of the first text. This application will not elaborate further on this, but those skilled in the art can refer to the above embodiments.

[0139] S404: Determine the second label of the missing words in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector.

[0140] To determine the predicted word corresponding to a missing word in the sample text, in this embodiment, a second prediction probability for each word in the sample text can be determined based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector. Therefore, a second label for the missing word in the sample text is determined based on the second prediction probability of each word segmentation. This second label is used to identify the predicted word corresponding to the missing word in the sample text.

[0141] It should be noted that the principle for determining the second prediction probability is similar to the principle for determining the first prediction probability described in the above embodiments, and the principle for determining the second label is similar to the principle for determining the first word in the above embodiments. Since the above embodiments have been described in detail, the embodiments of this application will not be repeated. Those skilled in the art can refer to the above embodiments.

[0142] S405: Determine the loss value based on the first label and the second label, and train the prediction model based on the loss value.

[0143] Since the standard words corresponding to the missing words in the sample text are known, i.e., the first label of the sample text is known, after determining the predicted words of the sample text, i.e. the second label, the loss value corresponding to the sample text can be determined based on the first and second labels of the sample text. Based on the determined loss value, the parameters of the prediction model can be adjusted to obtain the latest prediction model.

[0144] In this embodiment, a convergence condition is preset. This convergence condition may be that the number of times the first label and the second label of the sample text in the training sample set match is greater than a set number, or that the number of iterations of training the prediction model reaches a set maximum number of iterations, etc. Specific embodiments of this application do not limit this. When the convergence condition is met, the training of the prediction model can be considered complete, and a trained prediction model is obtained. This prediction model can be used for word determination.

[0145] In this embodiment, the sixth feature vector of the sample text is determined by comprehensively considering the fourth and fifth feature vectors of the third and fourth sub-texts, rather than directly based on the length of the sample text. This solves the problem of semantic forgetting in long texts. Furthermore, when determining the second label, the second label for missing words in the sample text is determined based on the sixth feature vector, the third segmentation feature vector of the third sub-text, and the fourth segmentation feature vector of the fourth sub-text. The prediction model is then trained based on the loss value between the first and second labels. During model training, not only are the individual features of each segment considered, but also the semantics of the entire sample text, i.e., contextual information. Even if the prediction model does not learn knowledge from a specific domain during training, it can still make accurate predictions as long as it can accurately extract features, thus improving prediction accuracy.

[0146] To improve the efficiency of model training, based on the above embodiments, in this embodiment, determining the sample text and the first label of the sample text includes:

[0147] Get the initial text;

[0148] By masking some words in the initial text, a sample text is obtained;

[0149] The masked words are identified as standard words, and the standard words are marked to obtain the first tag of the sample text.

[0150] Because training the model requires an enormous number of sample texts, it would be extremely time-consuming and inefficient to have each sample text's regions to be filled and labels determined by annotators. Therefore, in this embodiment, the sample text and its labels can be automatically determined by an electronic device. In this embodiment, an initial text can be obtained that does not contain missing words; that is, the initial text is complete and does not contain any blank character regions.

[0151] In this embodiment of the application, the initial text may be entered by the user of the electronic device, or it may be related text retrieved from the Internet based on the received keywords. The retrieved related text may be journals, news, etc.

[0152] After obtaining the initial text, the system automatically masks certain words in each initial text according to pre-saved masking rules to obtain sample text, and identifies the masked words as standard words. Simultaneously, these standard words are marked as the first tag of the sample text.

[0153] Specifically, the initial text can be segmented to obtain each word in the initial text, and the position of any word can be masked to obtain the sample text.

[0154] To further improve the accuracy of fill-in-the-blank semantic matching, based on the above embodiments, in this embodiment, the process of masking some words in the initial text to obtain sample text includes:

[0155] The first candidate word segment appears more than M times in the initial text; a second word segment is selected from the first candidate word segment and then masked to obtain the sample text, where M is a positive integer; or,

[0156] The second candidate word appears more than P times in the initial text. The third word is selected from the second candidate word. The similarity between the third word and each word in the initial text is determined, and the number of words with similarity greater than a preset threshold is counted. If the number is greater than the preset threshold, the fourth word is masked to obtain the sample text, where P is a positive integer.

[0157] In one possible implementation, during the process of masking some words in the initial text to obtain the sample text, the first candidate word segment that appears more than M times in the initial text can be counted. For example, M can be any positive integer greater than 1, such as M = 2 or 3.

[0158] After determining each first candidate word segment, one or more second words can be randomly selected from all the first candidate words, and one or more positions where the second word appears in the initial text can be masked to obtain the sample text. Masking can be understood as replacing one or more positions where the second word appears with whitespace.

[0159] Specifically, we can count the first candidate word that appears twice in the initial text, randomly select one of the first candidate word as the second word, and mask one of the two positions in which the second word appears in the initial text. The masked initial text is the sample text, and the masked second word is the standard word.

[0160] Alternatively, you can count the first candidate word that appears more than twice in the initial text, randomly select one of the second word from the first candidate word, and mask one of the multiple positions where the second word appears in the initial text. The masked initial text is the sample text, and the masked second word is the standard word.

[0161] In one possible implementation, during the process of masking some words in the initial text to obtain the sample text, the second candidate word segment that appears more than P times in the initial text can be counted, where P can be any positive integer greater than 1, for example, P = 2 or 3, and P can be the same as or different from M.

[0162] After determining each second candidate word, one or more second candidate words can be randomly selected from all the second candidate words as the third word.

[0163] After selecting the third word segment, the similarity between each third word segment and each word segment in the initial text can be determined. After determining each similarity, the number of word segments with a similarity greater than a preset similarity threshold is counted. This preset similarity threshold can be any decimal greater than 0 and less than 1, for example, a preset similarity threshold of 0.8.

[0164] After counting the number of word segments with similarity greater than a preset similarity threshold, we can determine whether this number exceeds the preset threshold. If so, it indicates that there are a large number of other similar word segments in the initial text. Masking this third word segment can increase the difficulty of model training and further improve the prediction performance of the trained prediction model. Therefore, at least one position of the third word segment in the initial text can be masked to obtain the sample text.

[0165] If it is determined that the number is not greater than the preset number threshold, the second target word can be discarded, and a third word can be reselected from each second candidate word until the sample text is obtained.

[0166] Specifically, we can count the second candidate word segments that appear two or more times in the initial text, randomly select one of these second candidate word segments as the third word segment, and determine the similarity between this third word segment and each word segment included in the initial text. We count the number of word segments with similarity greater than a preset threshold. If this number exceeds the preset threshold, we can mask one of the two or more positions where the selected third word segment appears in the initial text. The masked initial text is the sample text, and the masked third word segment is the standard word. If the count does not exceed the preset threshold, we can randomly select another second candidate word segment as the third word, and so on, until the sample text is determined.

[0167] The following describes the process of determining the sample text and its first tag using a specific example.

[0168] First, obtain N initial texts. After obtaining the N initial texts, the following three steps can be performed:

[0169] Step 1: Select N / 3 initial texts and process each initial text as follows: count the word segments that appear only twice in the initial text, randomly select one of the word segments, and mask one of the two positions of the word segment in the initial text to obtain the sample text. The masked word segment can be identified as the standard word.

[0170] The second step involves selecting N / 3 initial texts, ensuring that the selected initial texts are not duplicates of those obtained in the first step. For each initial text, the following processing is performed: count the word segments that appear more than twice in the initial text, randomly select one of these word segments, and mask one of its multiple positions in the initial text to obtain the sample text. The masked word segment can then be identified as the standard word.

[0171] The third step involves selecting N / 3 initial texts, ensuring that these selected texts are distinct from those obtained in the first and second steps. For each initial text, the following processing is performed: Segmentations appearing two or more times in the initial text are counted. One segment is randomly selected, and its position in one of two or more locations within the initial text is masked, resulting in a sample text. The masked segment is then identified as the standard word. The randomly selected segment must meet the following condition: at least K distinct segmentations in the initial text have a similarity greater than a threshold γ to this segment. If this condition is not met, another randomly selected segment is chosen, and the similarity is recalculated, and so on. For example, K = 3; γ = 0.9.

[0172] The above three steps complete the construction of the sample text. In this embodiment, the set of sample texts can be called the training sample set. The initial texts obtained in the above three steps are mutually exclusive. This embodiment constructs the training sample set in a more reasonable way, increasing the difficulty of model training and effectively improving the accuracy of model prediction.

[0173] To further improve the accuracy of the prediction model, based on the above embodiments, in this embodiment, the prediction model includes a feature extraction layer and a feature fusion layer.

[0174] Determining the fourth feature vector and the fifth feature vector of the third sub-text includes:

[0175] The third sub-text and the fourth sub-text are input into the feature extraction layer for feature extraction to obtain the fourth feature vector and the fifth feature vector;

[0176] Determining the sixth feature vector of the sample text based on the fourth and fifth feature vectors includes:

[0177] The fourth and fifth feature vectors are input into the feature fusion layer to obtain the sixth feature vector.

[0178] In this embodiment, the prediction model includes a feature extraction layer and a feature fusion layer. The feature extraction layer is used to extract features from the received third and fourth sub-texts, thereby obtaining a fourth feature vector of the third sub-text and a fifth feature vector of the fourth sub-text.

[0179] The feature fusion layer is used to fuse the received fourth and fifth feature vectors to obtain the sixth feature vector.

[0180] To further improve the accuracy of the prediction model, based on the above embodiments, in this embodiment, the prediction model includes a classification layer.

[0181] The step of determining the second label of each word in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector includes:

[0182] The sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector are input into the classification layer to obtain the second label of the missing words in the sample text.

[0183] In this embodiment, the prediction model further includes a classification layer. This classification layer is used to classify the received sixth feature vector, third word segmentation feature vector, and fourth word segmentation feature vector to obtain the second label of the missing words in the sample text.

[0184] In one possible implementation, if the sample text contains multiple second words, then determining the third and fourth sub-texts of the sample text includes:

[0185] Select any one of the plurality of second words as the second target word, and delete the remaining second words in the sample text;

[0186] Based on the position information of the second target word in the sample text, the sample text is divided into a third sub-text and a fourth sub-text.

[0187] In one possible implementation, determining the sixth feature vector of the sample text based on the fourth feature vector and the fifth feature vector includes:

[0188] The average vector of the fourth feature vector and the fifth feature vector is taken as the sixth feature vector.

[0189] In one possible implementation, determining the second label of the missing words in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector includes:

[0190] Based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector, determine the second prediction probability of each word in the sample text;

[0191] Based on the second prediction probability of each word segment, the second label of the missing word in the sample text is determined.

[0192] In one possible implementation, determining the second label of the missing word in the sample text based on the second prediction probability of each word segmentation includes:

[0193] Count the number of times each word appears in the sample text;

[0194] The second tag is determined based on the occurrence frequency of each word and its corresponding second prediction probability.

[0195] Since the processing principle of the sample text described above is similar to the processing principle of the first text in the above embodiments, and has been described in detail in the above embodiments, it will not be repeated in the embodiments of this application. The sample text can be processed by referring to the above embodiments.

[0196] Based on the same technical concept and the above embodiments, this application provides a word determination device. Figure 5 This is a schematic diagram of the structure of a word determination device provided in an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0197] The determination module 501 is used to determine the first subtext and the second subtext of the first text;

[0198] The feature extraction module 502 is used to determine the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text; and to determine the third feature vector of the first text based on the first feature vector and the second feature vector.

[0199] Prediction module 503 is used to determine the first prediction probability of each word in the first text based on the third feature vector, the first word segmentation feature vector and the second word segmentation feature vector;

[0200] The determining module 501 is further configured to determine a first word based on the first predicted probability, wherein the first word is used to represent a missing word between the first sub-text and the second sub-text.

[0201] In one possible implementation, if there are multiple first words in the first text, the determining module 501 is specifically used to determine any one of the multiple first words as the first target word, and delete the remaining first words in the first text; according to the position information of the first target word in the first text, the first text is divided into a first subtext and a second subtext.

[0202] In one possible implementation, the feature extraction module 502 is specifically used to determine the feature vector of the word that matches the first word in the first word set as the first word feature vector if the first word is found in the first word set; wherein the first word feature vector has the same dimension as the third feature vector.

[0203] In one possible implementation, the feature extraction module 502 is specifically used to take the average vector of the first feature vector and the second feature vector as the third feature vector.

[0204] In one possible implementation, the determining module 501 is specifically used to count the number of times each word appears in the first text; and to determine the first word based on the number of times each word appears and the corresponding first predicted probability.

[0205] In one possible implementation, the apparatus further includes a prediction model, which includes a feature extraction layer and a feature fusion layer;

[0206] The feature extraction module 502 is specifically used to input the first sub-text and the second sub-text into the feature extraction layer for feature extraction to obtain the first feature vector and the second feature vector; and to input the first feature vector and the second feature vector into the feature fusion layer to obtain the third feature vector.

[0207] In one possible implementation, the apparatus further includes a prediction model, the prediction model including a classification layer;

[0208] The prediction module 503 is specifically used to input the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector into the classification layer to obtain the first prediction probability of each word in the first text.

[0209] Based on the same technical concept and the above embodiments, this application provides a training apparatus for a prediction model. Figure 6 This is a schematic diagram of the structure of a training device for a prediction model provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0210] The determining module 601 is used to determine the sample text, the first tag of the sample text, and the third and fourth sub-texts of the sample text;

[0211] The feature extraction module 602 is used to determine the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text; and to determine the sixth feature vector of the sample text based on the fourth feature vector and the fifth feature vector.

[0212] The prediction module 603 is used to determine the second label of the missing words in the sample text based on the sixth feature vector, the third word segmentation feature vector and the fourth word segmentation feature vector;

[0213] The training module 604 is used to determine a loss value based on the first label and the second label, and to train the prediction model based on the loss value.

[0214] In one possible implementation, the determining module 601 is specifically used to acquire initial text; mask some words in the initial text to obtain sample text; determine the masked words as standard words, and mark the standard words to obtain the first tag of the sample text.

[0215] In one possible implementation, the determining module 601 is specifically used to count the first candidate word segments that appear more than M times in the initial text; select a second word segment from the first candidate word segments and mask the second word segment to obtain sample text, where M is a positive integer; or, count the second candidate word segments that appear more than P times in the initial text, select a third word segment from the second candidate word segments; determine the similarity between the third word segment and each word segment in the initial text, and count the number of word segments with similarity greater than a preset threshold; if the number is greater than a preset threshold, then mask the fourth word segment to obtain sample text, where P is a positive integer.

[0216] In one possible implementation, the prediction model includes a feature extraction layer and a feature fusion layer. The feature extraction module 602 is specifically used to input the third sub-text and the fourth sub-text into the feature extraction layer for feature extraction to obtain the fourth feature vector and the fifth feature vector; and to input the fourth feature vector and the fifth feature vector into the feature fusion layer to obtain the sixth feature vector.

[0217] In one possible implementation, the prediction model includes a classification layer, and the prediction module 603 is specifically used to input the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector into the classification layer to obtain the second label of the missing words in the sample text.

[0218] In one possible implementation, the first tag is used to identify the standard word corresponding to the missing word in the sample text, and the second tag is used to identify the predicted word corresponding to the missing word in the sample text.

[0219] In one possible implementation, if there are multiple second words in the sample text, the determining module 601 is specifically used to determine any one of the multiple second words as the second target word, and delete the remaining second words in the sample text; according to the position information of the second target word in the sample text, the sample text is divided into a third sub-text and a fourth sub-text.

[0220] In one possible implementation, the feature extraction module 602 is specifically used to take the average vector of the fourth feature vector and the fifth feature vector as the sixth feature vector.

[0221] In one possible implementation, the prediction module 603 is specifically used to determine the second prediction probability of each word in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector; and to determine the second label of the missing words in the sample text based on the second prediction probability of each word segmentation.

[0222] In one possible implementation, the prediction module 603 is specifically used to count the number of times each word appears in the sample text; and to determine the second tag based on the number of times each word appears and the corresponding second prediction probability.

[0223] Based on the same technical concept, this application also provides an electronic device. Figure 7 This application provides a schematic diagram of an electronic device structure, such as... Figure 7 As shown, it includes: processor 701, communication interface 702, memory 703 and communication bus 704, wherein processor 701, communication interface 702 and memory 703 communicate with each other through communication bus 704.

[0224] The memory 703 stores a computer program. When the program is executed by the processor 701, the processor 701 performs the following steps:

[0225] Determine the first sub-text and the second sub-text of the first text, as well as the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text.

[0226] Based on the first feature vector and the second feature vector, determine the third feature vector of the first text;

[0227] Based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector, determine the first prediction probability of each word in the first text;

[0228] Based on the first predicted probability, a first word is determined, which is used to represent a missing word between the first sub-text and the second sub-text.

[0229] In one possible implementation, if there are multiple first words in the first text, the processor 701 is further configured to determine any one of the multiple first words as the first target word and delete the remaining first words in the first text;

[0230] Based on the position information of the first target word in the first text, the first text is divided into a first subtext and a second subtext.

[0231] In one possible implementation, the processor 701 is further configured to, if a first segment is found in the first segment set, determine the feature vector of the segment that matches the first segment in the first segment set as the feature vector of the first segment;

[0232] The first word segmentation feature vector has the same dimension as the third feature vector.

[0233] In one possible implementation, the processor 701 is further configured to use the average vector of the first feature vector and the second feature vector as the third feature vector.

[0234] In one possible implementation, the processor 701 is further configured to count the number of times each word appears in the first text;

[0235] The first word is determined based on the frequency of occurrence of each word and its corresponding first prediction probability.

[0236] In one possible implementation, the method further includes a prediction model, which includes a feature extraction layer and a feature fusion layer; the processor 701 is further configured to input the first sub-text and the second sub-text into the feature extraction layer for feature extraction to obtain the first feature vector and the second feature vector;

[0237] The first feature vector and the second feature vector are input into the feature fusion layer to obtain the third feature vector.

[0238] In one possible implementation, the method further includes a prediction model, the prediction model including a classification layer;

[0239] The processor 701 is further configured to input the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector into the classification layer to obtain the first predicted probability of each word in the first text.

[0240] Since the principle of the above-mentioned electronic device in solving the problem is similar to that of the word determination method, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated details will not be described again.

[0241] Processor 701 can also perform the following steps:

[0242] Determine the sample text and its first tag;

[0243] The third and fourth sub-texts of the sample text are determined, as well as the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text.

[0244] Based on the fourth and fifth feature vectors, the sixth feature vector of the sample text is determined;

[0245] Based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector, determine the second label of the missing words in the sample text;

[0246] The loss value is determined based on the first label and the second label, and the prediction model is trained based on the loss value.

[0247] In one possible implementation, the processor 701 is further configured to acquire initial text;

[0248] By masking some words in the initial text, a sample text is obtained;

[0249] The masked words are identified as standard words, and the standard words are marked to obtain the first tag of the sample text.

[0250] In one possible implementation, the processor 701 is further configured to: count first candidate word segments that appear more than M times in the initial text; select a second word segment from the first candidate word segments and mask the second word segment to obtain sample text, where M is a positive integer; or,

[0251] The second candidate word appears more than P times in the initial text. The third word is selected from the second candidate word. The similarity between the third word and each word in the initial text is determined, and the number of words with similarity greater than a preset threshold is counted. If the number is greater than the preset threshold, the fourth word is masked to obtain the sample text, where P is a positive integer.

[0252] In one possible implementation, the prediction model includes a feature extraction layer and a feature fusion layer. The processor 701 is further configured to input the third sub-text and the fourth sub-text into the feature extraction layer for feature extraction to obtain the fourth feature vector and the fifth feature vector; and input the fourth feature vector and the fifth feature vector into the feature fusion layer to obtain the sixth feature vector.

[0253] In one possible implementation, the prediction model includes a classification layer, and the processor 701 is further configured to input the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector into the classification layer to obtain a second label for the missing words in the sample text.

[0254] In one possible implementation, if there are multiple second words in the sample text, the processor 701 is further configured to determine any one of the multiple second words as the second target word and delete the remaining second words in the sample text;

[0255] Based on the position information of the second target word in the sample text, the sample text is divided into a third sub-text and a fourth sub-text.

[0256] In one possible implementation, the processor 701 is further configured to use the average vector of the fourth feature vector and the fifth feature vector as the sixth feature vector.

[0257] In one possible implementation, the processor 701 is further configured to determine a second prediction probability for each word in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector;

[0258] Based on the second prediction probability of each word segment, the second label of the missing word in the sample text is determined.

[0259] In one possible implementation, the processor 701 is further configured to count the number of times each word appears in the sample text;

[0260] The second tag is determined based on the occurrence frequency of each word and its corresponding second prediction probability.

[0261] Since the principle of the above-mentioned electronic device in solving the problem is similar to the principle of the training method of the prediction model, the implementation of the above-mentioned electronic device can be found in the embodiments of the method, and repeated parts will not be described again.

[0262] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 702 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0263] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0264] Based on the same technical concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, it causes the electronic device to implement the word determination method or the prediction model training method described in any of the above embodiments.

[0265] The aforementioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in an electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, MO (magneto-optical disks), optical storage such as CDs, DVDs, BDs, HVDs, etc., and semiconductor storage such as ROMs, EPROMs, EEPROMs, NAND flash (non-volatile memory), SSDs (solid-state drives), etc.

[0266] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0267] Based on the same technical concept, this application provides a computer program product, which includes: computer program code, which, when executed on an electronic device, causes the electronic device to implement the word determination method described in any of the above embodiments, or the training method of the prediction model described in any of the above embodiments.

[0268] Computer programs used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing status information from the computer-readable program instructions to implement various aspects of this disclosure.

[0269] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0270] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0271] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0272] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0273] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0274] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for determining words, characterized in that, The method includes: Determine the first sub-text and the second sub-text of the first text, as well as the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text. Based on the first feature vector and the second feature vector, determine the third feature vector of the first text; Based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector, determine the first prediction probability of each word in the first text; Based on the first predicted probability, a first word is determined, which is used to represent a missing word between the first sub-text and the second sub-text.

2. The method according to claim 1, characterized in that, If the first text contains multiple first words, determining the first sub-text and the second sub-text of the first text includes: Select any one of the plurality of first words as the first target word, and delete the remaining first words in the first text; Based on the position information of the first target word in the first text, the first text is divided into a first subtext and a second subtext.

3. The method according to any one of claims 1, characterized in that, The step of determining the first segmentation feature vector of the first sub-text includes: If the first word is found in the first word segmentation set, then the feature vector of the word that matches the first word in the first word segmentation set is determined as the feature vector of the first word segmentation. The first word segmentation feature vector has the same dimension as the third feature vector.

4. The method according to claim 1, characterized in that, Determining the third feature vector of the first text based on the first feature vector of the first sub-text and the second feature vector of the second sub-text includes: The average vector of the first feature vector and the second feature vector is taken as the third feature vector.

5. The method according to claim 1, characterized in that, The step of determining the first word based on the first predicted probability includes: Count the number of times each word appears in the first text; The first word is determined based on the frequency of occurrence of each word and its corresponding first prediction probability.

6. The method according to claim 1, characterized in that: The method also includes a prediction model, which comprises a feature extraction layer and a feature fusion layer; Determining the first feature vector of the first sub-text and the second feature vector of the second sub-text includes: The first sub-text and the second sub-text are input into the feature extraction layer for feature extraction to obtain the first feature vector and the second feature vector. Determining the third feature vector of the first text based on the first feature vector and the second feature vector includes: The first feature vector and the second feature vector are input into the feature fusion layer to obtain the third feature vector.

7. The method according to claim 1, characterized in that: The method also includes a prediction model, which includes a classification layer; The step of determining the first predicted probability of each word in the first text based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector includes: The third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector are input into the classification layer to obtain the first predicted probability of each word in the first text.

8. A method for training a prediction model, characterized in that, The method includes: Determine the sample text and its first tag; The third and fourth sub-texts of the sample text are determined, as well as the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text. Based on the fourth and fifth feature vectors, the sixth feature vector of the sample text is determined; Based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector, determine the second label of the missing words in the sample text; The loss value is determined based on the first label and the second label, and the prediction model is trained based on the loss value.

9. The method according to claim 8, characterized in that, The determination of the sample text and the first tag of the sample text includes: Get the initial text; By masking some words in the initial text, a sample text is obtained; The masked words are identified as standard words, and the standard words are marked to obtain the first tag of the sample text.

10. The method according to claim 9, characterized in that, The process of masking certain words in the initial text to obtain sample text includes: The first candidate word segment appears more than M times in the initial text; a second word segment is selected from the first candidate word segment and then masked to obtain the sample text, where M is a positive integer; or, The second candidate word appears more than P times in the initial text. The third word is selected from the second candidate word. The similarity between the third word and each word in the initial text is determined, and the number of words with similarity greater than a preset threshold is counted. If the number is greater than the preset threshold, the fourth word is masked to obtain the sample text, where P is a positive integer.

11. The method according to claim 8, characterized in that, The prediction model includes a feature extraction layer and a feature fusion layer. Determining the fourth feature vector and the fifth feature vector of the third sub-text includes: The third sub-text and the fourth sub-text are input into the feature extraction layer for feature extraction to obtain the fourth feature vector and the fifth feature vector; Determining the sixth feature vector of the sample text based on the fourth and fifth feature vectors includes: The fourth and fifth feature vectors are input into the feature fusion layer to obtain the sixth feature vector.

12. The method according to claim 8, characterized in that, The prediction model includes a classification layer. The step of determining the second label of each word in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector includes: The sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector are input into the classification layer to obtain the second label of the missing words in the sample text.

13. The method according to claim 8, characterized in that, The first label is used to identify the standard word corresponding to the missing word in the sample text, and the second label is used to identify the predicted word corresponding to the missing word in the sample text.

14. A word determination device, characterized in that, The device includes: The determination module is used to determine the first and second subtexts of the first text; The feature extraction module is used to determine the first feature vector of the first sub-text, the first segmentation feature vector of the first sub-text, the second feature vector of the second sub-text, and the second segmentation feature vector of the second sub-text; and to determine the third feature vector of the first text based on the first feature vector and the second feature vector. The prediction module is used to determine the first prediction probability of each word in the first text based on the third feature vector, the first word segmentation feature vector, and the second word segmentation feature vector. The determining module is further configured to determine a first word based on the first predicted probability, wherein the first word represents a missing word between the first sub-text and the second sub-text.

15. A training device for a prediction model, characterized in that, The device includes: The determination module is used to determine the sample text, the first tag of the sample text, and the third and fourth sub-texts of the sample text; The feature extraction module is used to determine the fourth feature vector of the third sub-text, the third word segmentation feature vector of the third sub-text, the fifth feature vector of the fourth sub-text, and the fourth word segmentation feature vector of the fourth sub-text; and to determine the sixth feature vector of the sample text based on the fourth feature vector and the fifth feature vector. The prediction module is used to determine the second label of the missing words in the sample text based on the sixth feature vector, the third word segmentation feature vector, and the fourth word segmentation feature vector; The training module is used to determine a loss value based on the first label and the second label, and to train the prediction model based on the loss value.

16. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the steps of the word determination method as described in any one of claims 1-7, or the steps of the training method for the prediction model as described in any one of claims 8-13.

17. A computer storage medium, characterized in that, It stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the word determination method according to any one of claims 1-7, or the steps of the training method for the prediction model according to any one of claims 8-13.

18. A computer program product, characterized in that, The computer program product includes: computer program code, which, when executed on the electronic device, causes the electronic device to perform the steps of the word determination method according to any one of claims 1-7, or the steps of the training method for the prediction model according to any one of claims 8-13.

Images