Variant word recognition method and device

Abstract

The invention relates to the technical field of natural language processing, and provides a variant word recognition method and device, and the method comprises the steps: carrying out the word segmentation of a to-be-recognized text, and obtaining at least one text segmented word; performing text image generation on each text segmented word to obtain a segmented word image corresponding to each text segmented word; and obtaining a variant word recognition result of each text segmented word based on the coding feature of each text segmented word and the coding feature of the corresponding segmented word image. According to the method, the encoding features of the text segmented words and the encoding features of the corresponding segmented word images are combined, text semantics and visual detail information can be fused, the text segmented words in the text mode and the segmented word images in the image mode are aligned to the same semantic space, the variant words are comprehensively understood and recognized, and the accuracy of variant word recognition is effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a variant word recognition method and device. Background Art

[0002] With the rapid development of the Internet, text data has exploded, and text data sometimes contains variant words, which bring huge challenges to natural language processing tasks such as text analysis or information retrieval.

[0003] In the prior art, when identifying variant words, the identification is usually performed based on the type of each word segment in the text to be identified in combination with a variant word library. However, this method relies too much on the variant word library, resulting in limited recognition ability, making it difficult to cope with the emergence of new variant words, and the recognition effect is poor for variant words that are not in the word library. Summary of the invention

[0004] The present invention provides a variant word recognition method and device to solve the defects in the prior art.

[0005] The present invention provides a variant word recognition method, comprising the following steps: Segment the text to be recognized to obtain at least one text segmentation; Generate text images for each text segmentation to obtain segmentation images corresponding to each text segmentation; Based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained; The variant word recognition results of each text segmentation are obtained based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, including: Selecting at least one expert model from a plurality of expert models as a text expert model based on the encoding features of each text segmentation word, and selecting at least one expert model from the plurality of expert models as an image expert model based on the encoding features of each segmentation image; Based on each text expert model, the encoding features of each text word segmentation are applied to perform variant word recognition to obtain the text recognition results output by each text expert model; Based on each image expert model, the encoding features of each word segmentation image are used to perform variant word recognition, and the image recognition results output by each image expert model are obtained; Based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, the variant word recognition results of each text segmentation are determined.

[0006] According to a variant word recognition method provided by the present invention, the variant word recognition results of each text segmentation are determined based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, including: Based on a first weight, the text recognition results output by each text expert model are fused to obtain a first fusion result, wherein the first weight is determined based on the matching degree between the encoding features of each text segmentation and each text expert model; Based on the second weight, the image recognition results output by each image expert model are fused to obtain a second fusion result, wherein the second weight is determined based on the matching degree between the encoding features of each word segmentation image and each image expert model; Based on the first fusion result and the second fusion result, variant word recognition results of each text segmentation are determined.

[0007] According to a variant word recognition method provided by the present invention, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, a variant word recognition result of each text segmentation is obtained, including: Based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition result of each text segmentation is obtained; the variant word recognition model is trained based on a sample text and the variant word label of the sample text, and the sample text is a fusion of a general sample and a domain sample.

[0008] According to a variant word recognition method provided by the present invention, the variant word recognition model is trained based on the following steps: Get the pre-trained model; Performing low-rank decomposition on the weights of the pre-trained model to obtain pre-trained weights and incremental weights; The pre-training weights are frozen, and based on the sample text and the variant word labels of the sample text, the incremental weights are updated to obtain the variant word recognition model.

[0009] According to a variant word recognition method provided by the present invention, based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, a variant word recognition result of each text segmentation is obtained, including: Constructing a prompt text for representing variant word recognition requirements; Based on the variant word recognition model, the prompt text is applied to segment the text to be recognized to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition result of each text segmentation is obtained.

[0010] The present invention also provides a variant word recognition device, comprising the following modules: A word segmentation unit, used for segmenting the text to be recognized to obtain at least one text segmentation; A generating unit, used for generating a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; A recognition unit, used for obtaining variant word recognition results of each text segmentation based on the coding features of each text segmentation and the coding features of the corresponding segmentation image; The variant word recognition results of each text segmentation are obtained based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, including: Selecting at least one expert model from a plurality of expert models as a text expert model based on the encoding features of each text segmentation word, and selecting at least one expert model from the plurality of expert models as an image expert model based on the encoding features of each segmentation image; Based on each text expert model, the encoding features of each text word segmentation are applied to perform variant word recognition to obtain the text recognition results output by each text expert model; Based on each image expert model, the encoding features of each word segmentation image are used to perform variant word recognition, and the image recognition results output by each image expert model are obtained; Based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, the variant word recognition results of each text segmentation are determined.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the variant word recognition methods described above is implemented.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the variant word recognition method as described in any one of the above is implemented.

[0013] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for identifying variant words as described above is implemented.

[0014] The variant word recognition method and device provided by the present invention use the encoding features of each text segmentation to represent the semantic information of each text segmentation, and the encoding features of the segmentation image corresponding to each text segmentation to represent the visual information, contextual association and implicit meaning of the segmentation image. Then, by combining the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the text semantics and visual detail information can be integrated, and the text segmentation in the text modality and the segmentation image in the image modality can be aligned to the same semantic space, so as to more comprehensively understand and recognize the variant words and effectively improve the accuracy of variant word recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flow chart of the variant word recognition method provided by the present invention.

[0017] Figure 2 It is a structural diagram of the variant word recognition model provided by the present invention.

[0018] Figure 3 It is a structural schematic diagram of the variant word recognition device provided by the present invention.

[0019] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] At present, when identifying variant words, the text to be identified is first converted into a standard format, and the converted text is segmented. Then, according to the type of segmentation (such as full Chinese characters, Chinese characters and character combinations, full letters), the corresponding variant word library is selected, and a specific recognition algorithm is applied for preliminary recognition. The preliminary recognition result is then associated with the associated word library for analysis to obtain the final recognition result. However, this method is overly dependent on the variant word library, resulting in limited recognition capabilities, making it difficult to cope with the emergence of new variant words, and the recognition effect of variant words that are not in the word library is poor.

[0022] In this regard, the present invention provides a variant word recognition method, which can be applied to any scenario where variant word recognition is required, such as information retrieval, large language model input and output content review, etc. It should be noted that the variant words involved in the present invention refer to similar characters, such as characters obtained by disassembling a certain character or replacing radicals or components, and are not morphological characters that appear in images in optical character recognition (OCR) scenarios and are formed due to distortion or deformation of the image. The executor of the present invention can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed variant word recognition device, or it can be a variant word recognition device set in the electronic device, and the variant word recognition device can be implemented by software, hardware, or a combination of both.

[0023] Figure 1 is a flow chart of the variant word recognition method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0024] Step 110: Segment the text to be recognized to obtain at least one text segmentation.

[0025] Specifically, the text to be recognized can be understood as text that needs to be checked for variant words, such as input text or output text of a large language model, etc. The text to be recognized includes text directly input by a user or text obtained through audio conversion.

[0026] After obtaining the text to be recognized, a word segmentation tool may be used to perform word segmentation on the text to be recognized to obtain at least one text word segmentation, wherein the text word segmentation may include a word or a single character.

[0027] Step 120: Generate a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation.

[0028] Step 130: Based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, obtain the variant word recognition results of each text segmentation.

[0029] Specifically, text tokenization is used to represent the semantic information of the tokenization itself. If variant word recognition is only based on the features of text tokenization, the recognition accuracy may be relatively low due to the lack or ambiguity of semantic information. Considering that image information can intuitively reflect the visual features of text tokenization, and the tokenization images corresponding to text tokenization can be obtained through text-to-image synthesis technology, there are rich visual details, context associations, and possible implicit meanings in the tokenization images corresponding to text tokenization. Therefore, by combining this visual information and context associations (or implicit meanings), the semantic understanding deviation in variant word recognition based only on the features of text tokenization can be corrected, thereby enhancing the accuracy and robustness of variant word recognition.

[0030] Among them, the tokenization images corresponding to each text tokenization not only contain each text tokenization, but also visually describe the semantic information of each text tokenization. For example, for the text tokenization "invisible person", in the online context, it usually refers to those who are inactive and unnoticed in online communities. Then, the generated tokenization image includes both the text corresponding to "invisible person" and the visual elements used to describe this inactive and unnoticed state, such as an image showing a small person hiding in a corner or being ignored. That is to say, the tokenization image contains visual information describing the online meaning of "invisible person". By combining this rich visual information and context associations (or implicit meanings), the deep semantics of text tokenization can be more effectively understood, and the semantic understanding deviation that may occur when recognizing variant words based only on the features of text tokenization can be corrected. Thus, not only can the accuracy of variant word recognition be improved, but its robustness can also be enhanced because even if the text tokenization itself is ambiguous or polysemous, through the intuitive visual clues provided by the tokenization image, the system can more accurately identify the correct meaning of the variant word.

[0031] On this basis, the encoding features of each text tokenization are used to represent the semantic information of each text tokenization, and the encoding features of the tokenization images corresponding to each text tokenization are used to represent the visual information, context associations, and implicit meanings of the tokenization images. Therefore, by combining the encoding features of each text tokenization and the encoding features of the corresponding tokenization images, text semantics and visual detail information can be fused to more comprehensively understand and identify text tokenization. Especially when dealing with variant words, the deviation that may occur in recognition based on the features of text tokenization can be corrected, improving the accuracy and robustness of recognition.

[0032] Exemplarily, based on the above embodiments, the variant word recognition results of each text tokenization can be used to represent whether each text tokenization is a variant word and the original word corresponding to the variant word. For example, the variant word is "V letter", and its corresponding original word is "WeChat".

[0033] In the variant word recognition method provided by the embodiment of the present invention, the encoding features of each text segmentation are used to represent the semantic information of each text segmentation, and the encoding features of the segmentation image corresponding to each text segmentation are used to represent the visual information, contextual association and implicit meaning of the segmentation image. Then, by combining the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the text semantics and visual detail information can be integrated, and the text segmentation in the text modality and the segmentation image in the image modality can be aligned to the same semantic space, so as to more comprehensively understand and recognize the variant words and effectively improve the accuracy of variant word recognition.

[0034] Based on the above embodiment, based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained, including: Selecting at least one expert model from a plurality of expert models as a text expert model based on the encoding features of each text segmentation word, and selecting at least one expert model from a plurality of expert models as an image expert model based on the encoding features of each segmentation image; Based on each text expert model, the encoding features of each text word segmentation are applied to perform variant word recognition to obtain the text recognition results output by each text expert model; Based on each image expert model, the encoding features of each word segmentation image are used to perform variant word recognition, and the image recognition results output by each image expert model are obtained; Based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, the variant word recognition results of each text segmentation are determined.

[0035] Taking into account the differences in form and semantics between text segmentation in text modality and segmentation images in image modality, such as text segmentation in text modality is divided into vocabulary units through spaces, punctuation marks or algorithms, while segmentation images in image modality are divided into image blocks or regions with specific semantics through image processing technology. Based on these differences, if the same network model is used for variant word recognition, there may be problems such as decreased recognition accuracy and poor adaptability.

[0036] In this regard, the embodiment of the present invention matches each text segmentation and the corresponding segmentation image with a corresponding expert model, that is, the text segmentation matches the text expert model, and the segmentation image matches the image expert model. On this basis, the matched text expert model focuses on processing and analyzing the encoding features of the text segmentation, and the image expert model focuses on processing and analyzing the encoding features of the segmentation image. Furthermore, the text expert model can accurately identify variant words based on the encoding features of the text segmentation to obtain text recognition results, and the image expert model can accurately identify variant words based on the encoding features of the segmentation image to obtain image recognition results.

[0037] Exemplarily, the matching degree between the encoding features of each text segmentation and each expert model can be determined based on a gating network (i.e., routing). The matching degree can be understood as the degree of fit or applicability between each text segmentation and each expert model. The higher the matching degree, the more suitable the corresponding expert model is for processing the text segmentation. For example, a threshold can be set, and an expert model with a matching degree greater than the threshold is used as the text expert model of the corresponding text segmentation, so that the text expert model can focus on processing the encoding features of the text segmentation, accurately identify variant words for the text segmentation based on the encoding features of the text segmentation, and obtain a text recognition result.

[0038] Similarly, the matching degree between the encoding features of each word segmentation image and each expert model can also be determined based on the gated network. The matching degree can be understood as the degree of fit or applicability between each word segmentation image and each expert model. The higher the matching degree, the more suitable the corresponding expert model is for processing the word segmentation image. For example, a threshold can be set, and the expert model with a matching degree greater than the threshold is used as the image expert model of the corresponding word segmentation image, so that the image expert model can focus on processing the encoding features of the word segmentation image, accurately perform variant word recognition on the word segmentation image based on the encoding features of the word segmentation image, and obtain an image recognition result.

[0039] On this basis, by combining the text recognition results and the image recognition results, the text features and the image features can be fused to accurately obtain the variant word recognition results. This fusion method can make full use of the information of the two modalities of text and image to improve the accuracy and robustness of variant word recognition. Among them, the above-mentioned expert model can be a MOE (Mixture of Experts) model, which can select the most suitable expert for processing according to the different characteristics of the input data, thereby improving the recognition performance of the model.

[0040] Based on any of the above embodiments, based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, determining the variant word recognition results of each text segmentation includes: Based on a first weight, the text recognition results output by each text expert model are fused to obtain a first fusion result, wherein the first weight is determined based on the matching degree between the encoding features of each text segmentation and each text expert model; Based on the second weight, the image recognition results output by each image expert model are fused to obtain a second fusion result, wherein the second weight is determined based on the matching degree between the encoding features of each word segmentation image and each image expert model; Based on the first fusion result and the second fusion result, the variant word recognition result of each text segmentation is determined.

[0041] Specifically, the first weight is determined based on the matching degree between the encoding features of each text segmentation and each text expert model, that is, the first weight is used to characterize the confidence or importance of the text expert model corresponding to each text segmentation. On this basis, the text recognition results output by each text expert model can be fused based on the first weight to obtain a first fusion result. For example, based on the first weight, the weights of each text recognition result are added, and the text recognition result with a larger weight contributes more to the final fusion result, thereby obtaining the first fusion result.

[0042] The second weight is determined based on the matching degree between the encoding features of each word segmentation image and each image expert model, that is, the second weight is used to characterize the confidence or importance of each word segmentation image corresponding to the image expert model. On this basis, the image recognition results output by each image expert model can be fused based on the second weight to obtain a second fusion result. Similarly, the image recognition results can be weighted based on the second weight, and the image recognition results with larger weights have a higher contribution to the final fusion result, thereby obtaining a second fusion result.

[0043] In addition, considering that the first fusion result may have problems of misrecognition or incomplete recognition due to the noise of the text data or the limitations of the text expert model, and the second fusion result may have problems of misrecognition or inaccurate recognition due to the ambiguity of the image data or the limitations of the image expert model, the embodiment of the present invention combines the first fusion result and the second fusion result to determine the variant word recognition result, which can avoid the recognition errors that may be caused by single modality data or a single model and accurately obtain the variant word recognition result.

[0044] Exemplarily, the variant word recognition result can be determined based on a weighted average or voting mechanism, in combination with the first fusion result and the second fusion result. For example, different weights can be assigned to the first fusion result and the second fusion result according to their confidence or reliability, and then weighted average can be performed to obtain the final variant word recognition result. Alternatively, a voting mechanism can be used to perform majority voting or weighted voting based on the recognition of the variant words in each fusion result to determine the final variant word recognition result.

[0045] Based on any of the above embodiments, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, variant word recognition results of each text segmentation are obtained, including: Based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained; the variant word recognition model is trained based on sample text and the variant word labels of the sample text, and the sample text is a fusion of general samples and domain samples.

[0046] Specifically, the variant word recognition model is trained based on sample text and variant word labels of the sample text. The sample text is integrated with general samples and domain samples. General samples refer to samples that are widely applicable to multiple fields or scenarios, and domain samples refer to samples within a specific field or professional scope. The integration ratio of general samples and domain samples can be 2:1, or can be set according to actual conditions, which is not specifically limited in the embodiments of the present invention.

[0047] If the variant word recognition model is obtained based only on domain sample training, the model may over-focus on the specific knowledge of the domain during training and forget general knowledge, which will affect the generalization ability of the model and further affect the accuracy of the variant word recognition results.

[0048] On this basis, the embodiment of the present invention is based on a variant word recognition model and can utilize the learned cross-modal fusion knowledge to integrate and understand information from different modalities, segment the text to be recognized, and obtain at least one text segmentation; generate a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, accurately obtain the variant word recognition results of each text segmentation.

[0049] Based on any of the above embodiments, the variant word recognition model is trained based on the following steps: Get the pre-trained model; Perform low-rank decomposition on the weights of the pre-trained model to obtain pre-trained weights and incremental weights; Freeze the pre-trained weights, and based on the sample text and the variant word labels of the sample text, update the incremental weights to obtain the variant word recognition model.

[0050] Specifically, the pre-trained model can be understood as a model framework with a broad knowledge base, which is usually obtained through training based on large-scale general samples. Therefore, the pre-trained model has the ability to efficiently extract and represent general features, but lacks the ability to perform refined recognition for specific fields or tasks. In order to enable the pre-trained model to have the ability to recognize variant words for specific fields or tasks, it is necessary to further train the pre-trained model based on sample texts containing general samples and domain samples. However, considering that the number of parameters of the pre-trained model is usually large, if all weights are updated directly, in order to make the model converge quickly, it is often necessary to obtain a large amount of sample text. However, these large amounts of sample texts are not only difficult to obtain, but also the cost of manual annotation is high.

[0051] To address this problem, an embodiment of the present invention proposes an innovative solution: low-rank decomposition of the weights of the pre-trained model is performed into two parts: pre-trained weights and incremental weights. Among them, the pre-trained weights mainly carry the extensive knowledge learned by the model on general samples, while the incremental weights are used to capture new features in specific fields or tasks. During the training process, the embodiment of the present invention chooses to freeze the pre-trained weights, that is, this part of the weights is not updated, but only the incremental weights are updated based on the sample text and the variant word labels of the sample text. This method can not only ensure that the trained variant word recognition model has excellent performance, but also can significantly improve the training efficiency and save training costs.

[0052] Optionally, the weights of the pre-trained model can be calculated by referring to the following formula: Perform low-rank decomposition: in, represents the pre-training weights, Indicates incremental weights, freezing pre-trained weights during training , only update the incremental weight , that is, only need to save ,when When , the number of parameters saved is small.

[0053] Compared with full-parameter incremental pre-training, the incremental pre-training based on low-rank decomposition in the embodiment of the present invention is not only more efficient, but also effectively alleviates the problem that the model is difficult to converge under small sample learning.

[0054] Based on any of the above embodiments, based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, a variant word recognition result of each text segmentation is obtained, including: Construct prompt text for characterizing variant word recognition requirements; Based on the variant word recognition model, apply the prompt text to segment the text to be recognized, obtaining at least one text segment; generate text images for each text segment, obtaining the segmented images corresponding to each text segment; based on the encoding features of each text segment and the encoding features of the corresponding segmented images, obtain the variant word recognition results for each text segment.

[0055] Specifically, although the above-mentioned variant word recognition model has learned rich knowledge in the field of variant words, however, due to the possible limitations of the model in dealing with some complex or ambiguous situations, in order to further improve its recognition accuracy, the embodiments of the present invention also design corresponding prompt text. The prompt text aims to fully stimulate the potential recognition ability and context understanding ability of the variant word recognition model. By providing additional information or clues, it helps the model make more accurate judgments when facing complex or ambiguous situations, so as to accurately obtain the variant word recognition results.

[0056] Among them, this prompt text can be understood as Prompt text, which is used to characterize the variant word recognition requirements. For example, the prompt text can assign a role to the model, clearly define the recognition requirements, and give small sample examples.

[0057] An example of the prompt text is as follows: {You are a Chinese expert. Please carefully identify whether there are variant words in

text to be recognized

[0058] The variant words and the original words may be related or similar in pronunciation, spelling, and semantics.

[0059]

Text or picture to be recognized

[0060] Requirement: Output the set of all variant words in the text in json format, without too much explanation and no extra output.

[0061] Example of the output format: ''' [{ "Variant word": "V letter", "Original word": "WeChat" }] The above prompt words assign a role to the model: Chinese expert, and clearly define the variant words; at the same time, it also gives sample examples in the form of Few-Shot; in addition, it also restricts the output format, and based on the instruction-following ability of the model, the expected output can be obtained.

[0062] Figure 2is a schematic diagram of the structure of the variant word recognition model provided by the present invention, such as Figure 2 As shown in the figure, the variant word recognition model replaces the fully connected layer in the transformer with a sparse hybrid expert network, which contains 4 expert models (FFN1, FFN2, FFN3 and FFN4). In model training, token is the basic unit for the model to process and understand text or images. Token can represent words, characters, phrases, or even data such as images or sounds. The gating network (i.e., routing) in the hybrid expert determines which tokens are sent to which expert model. For example, the token corresponding to "picture" may be sent to the second expert model, while the token corresponding to "text" is sent to the first expert model. A token can even be sent to multiple expert models.

[0063] The gating network is responsible for selecting a sparse expert combination for each input token (the input token includes the encoding features of the text segmentation and the encoding features of the segmentation image). is the weight of the i-th expert model, is the output of the i-th expert model, then for the current input (i.e. the current input token), output That is the weighted sum of all corresponding expert models: In the above formula, if G (the output of the gated network) is 0, there is no need to calculate the corresponding E(x), so computing resources can be saved. A typical gated network is usually a network with a softmax function. As shown below: in, represents the output of the gating network, represents the input of the gating network, Represents the weight matrix of the gating network.

[0064] In addition, the encoding features of the word segmentation image can be extracted using the visual encoder (VIT), and the encoding features of the text segmentation can be extracted using the text encoder. Among them, ViT divides the input word segmentation image into multiple patches (16x16), and then projects each patch into a vector of fixed length and sends it to the Transformer. The subsequent encoder operation is exactly the same as the standard Transformer.

[0065] Among them, the encoding process of VIT is as follows: Patch embedding: If the input word segmentation image size is 224×224, the image is divided into fixed-size patches, and the patch size is 16×16, then each image will generate 224×224 / 16×16=196 patches, that is, the input sequence length is 196, each patch dimension is 16×16×3=768, and the dimension of the linear projection layer is 768×N (N=768), so the dimension of the input after passing through the linear projection layer is still 196×768, that is, there are a total of 196 tokens, and the dimension of each token is 768. Here, a special character cls needs to be added in front, so the final dimension is 197×768.

[0066] Positional encoding: ViT also needs to add positional encoding. Positional encoding can be understood as a table with N rows in total. The size of N is the same as the length of the input sequence. Each row represents a vector. The dimension of the vector is the same as the dimension of the input sequence embedding (768). Note that the operation of positional encoding is sum, not concat. After adding positional encoding information, the dimension is still 197×768.

[0067] Through the above steps, a visual problem can be converted into a seq2seq problem. Subsequent learning and training are similar to the standard transformer.

[0068] The encoding process of the text encoder is as follows: Based on the learning and representation capabilities of the large model itself, an efficient text encoder can be constructed. In order to make the learning of the large model more stable, Layer Normalization is optimized and trained using DeepNorm. The expression is as follows: in, Indicates transformer layers, represents the hyperparameter, Indicates The input of the layer, Indicates +1 layer input, Indicates The variable of the layer, LN means normalization operation.

[0069] The variant word recognition device provided by the present invention is described below. The variant word recognition device described below and the variant word recognition method described above can be referenced to each other.

[0070] Based on any of the above embodiments, Figure 3 The invention provides a variant word recognition device, comprising: A word segmentation unit 310, used to segment the text to be recognized to obtain at least one text segmentation; A generating unit 320 is used to generate a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; The recognition unit 330 is used to obtain variant word recognition results of each text segmentation based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image.

[0071] Based on any of the above embodiments, based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained, including: Selecting at least one expert model from a plurality of expert models as a text expert model based on the encoding features of each text segmentation word, and selecting at least one expert model from a plurality of expert models as an image expert model based on the encoding features of each segmentation image; Based on each text expert model, the encoding features of each text word segmentation are applied to perform variant word recognition to obtain the text recognition results output by each text expert model; Based on each image expert model, the encoding features of each word segmentation image are used to perform variant word recognition, and the image recognition results output by each image expert model are obtained; Based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, the variant word recognition results of each text segmentation are determined.

[0072] Based on any of the above embodiments, based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, determining the variant word recognition results of each text segmentation includes: Based on a first weight, the text recognition results output by each text expert model are fused to obtain a first fusion result, wherein the first weight is determined based on the matching degree between the encoding features of each text segmentation and each text expert model; Based on the second weight, the image recognition results output by each image expert model are fused to obtain a second fusion result, wherein the second weight is determined based on the matching degree between the encoding features of each word segmentation image and each image expert model; Based on the first fusion result and the second fusion result, the variant word recognition result of each text segmentation is determined.

[0073] Based on any of the above embodiments, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, variant word recognition results of each text segmentation are obtained, including: Based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained; the variant word recognition model is trained based on sample text and the variant word labels of the sample text, and the sample text is a fusion of general samples and domain samples.

[0074] Based on any of the above embodiments, the variant word recognition model is trained based on the following steps: Get the pre-trained model; Perform low-rank decomposition on the weights of the pre-trained model to obtain pre-trained weights and incremental weights; Freeze the pre-trained weights, and based on the sample text and the variant word labels of the sample text, update the incremental weights to obtain the variant word recognition model.

[0075] Based on any of the above embodiments, based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, a variant word recognition result of each text segmentation is obtained, including: Constructing a prompt text for representing variant word recognition requirements; Based on the variant word recognition model, the prompt text is applied to segment the text to be recognized to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained.

[0076] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the variant word recognition method, which includes: segmenting the text to be recognized to obtain at least one text segmentation; generating a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; and obtaining a variant word recognition result of each text segmentation based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image.

[0077] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0078] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the variant word recognition method provided by the above methods, which includes: segmenting the text to be recognized to obtain at least one text segmentation; generating a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; and obtaining a variant word recognition result for each text segmentation based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image.

[0079] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the variant word recognition method provided by the above-mentioned methods, the method comprising: segmenting the text to be recognized to obtain at least one text segmentation; generating a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; and obtaining a variant word recognition result for each text segmentation based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image.

[0080] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A variant word recognition method, characterized in that: include: Segment the text to be recognized to obtain at least one text segmentation; Generate text images for each text segmentation to obtain segmentation images corresponding to each text segmentation; Based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, the variant word recognition results of each text segmentation are obtained; The variant word recognition results of each text segmentation are obtained based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, including: Selecting at least one expert model from a plurality of expert models as a text expert model based on the encoding features of each text segmentation word, and selecting at least one expert model from the plurality of expert models as an image expert model based on the encoding features of each segmentation image; Based on each text expert model, the encoding features of each text word segmentation are applied to perform variant word recognition to obtain the text recognition results output by each text expert model; Based on each image expert model, the encoding features of each word segmentation image are used to perform variant word recognition, and the image recognition results output by each image expert model are obtained; Based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, the variant word recognition results of each text segmentation are determined.

2. The variant word recognition method according to claim 1, characterized in that: The method of determining variant word recognition results of each text segmentation based on the text recognition results output by each text expert model and the image recognition results output by each image expert model includes: Based on a first weight, the text recognition results output by each text expert model are fused to obtain a first fusion result, wherein the first weight is determined based on the matching degree between the encoding features of each text segmentation and each text expert model; Based on the second weight, the image recognition results output by each image expert model are fused to obtain a second fusion result, wherein the second weight is determined based on the matching degree between the encoding features of each word segmentation image and each image expert model; Based on the first fusion result and the second fusion result, variant word recognition results of each text segmentation are determined.

3. The variant word recognition method according to claim 1 or 2, characterized in that: The text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, a variant word recognition result of each text segmentation is obtained, including: Based on the variant word recognition model, the text to be recognized is segmented to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition result of each text segmentation is obtained; the variant word recognition model is trained based on a sample text and the variant word labels of the sample text, and the sample text is a fusion of a general sample and a domain sample.

4. The variant word recognition method according to claim 3, characterized in that: The variant word recognition model is trained based on the following steps: Get the pre-trained model; Performing low-rank decomposition on the weights of the pre-trained model to obtain pre-trained weights and incremental weights; The pre-training weights are frozen, and based on the sample text and the variant word labels of the sample text, the incremental weights are updated to obtain the variant word recognition model.

5. The variant word recognition method according to claim 3, characterized in that: The variant word recognition model is based on segmenting the text to be recognized to obtain at least one text segmentation; generating a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; and obtaining variant word recognition results of each text segmentation based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, including: Constructing a prompt text for representing variant word recognition requirements; Based on the variant word recognition model, the prompt text is applied to segment the text to be recognized to obtain at least one text segmentation; a text image is generated for each text segmentation to obtain a segmentation image corresponding to each text segmentation; based on the encoding features of each text segmentation and the encoding features of the corresponding segmentation image, the variant word recognition result of each text segmentation is obtained.

6. A variant word recognition device, characterized in that: include: A word segmentation unit, used for segmenting the text to be recognized to obtain at least one text segmentation; A generating unit, used for generating a text image for each text segmentation to obtain a segmentation image corresponding to each text segmentation; A recognition unit, used for obtaining variant word recognition results of each text segmentation based on the coding features of each text segmentation and the coding features of the corresponding segmentation image; The variant word recognition results of each text segmentation are obtained based on the coding features of each text segmentation and the coding features of the corresponding segmentation image, including: Selecting at least one expert model from a plurality of expert models as a text expert model based on the encoding features of each text segmentation word, and selecting at least one expert model from the plurality of expert models as an image expert model based on the encoding features of each segmentation image; Based on each text expert model, the encoding features of each text word segmentation are applied to perform variant word recognition to obtain the text recognition results output by each text expert model; Based on each image expert model, the encoding features of each word segmentation image are used to perform variant word recognition, and the image recognition results output by each image expert model are obtained; Based on the text recognition results output by each text expert model and the image recognition results output by each image expert model, the variant word recognition results of each text segmentation are determined.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the variant word identification method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the variant word identification method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the variant word identification method according to any one of claims 1 to 5 is implemented.

Images