Variant word recognition method, device, equipment and storage medium

Abstract

The present invention provides a variant word recognition method, device, equipment and storage medium, which relates to the field of artificial intelligence technology. The method includes: obtaining a text to be recognized; segmenting the text to be recognized to obtain at least one vocabulary; for each of the vocabulary, inputting a target image containing the vocabulary and at least one target vocabulary in a vocabulary list into a variant word recognition model to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to characterize whether the vocabulary is a variant word; wherein the variant word recognition model is obtained by training an initial variant word recognition model based on each vocabulary in the vocabulary list and a sample image corresponding to each vocabulary, wherein the sample image includes the variant word corresponding to the vocabulary. The present invention can improve the accuracy of variant word recognition results.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a variant word recognition method, device, equipment and storage medium. Background Art

[0002] With the rapid development of the Internet, text data has exploded, and text data sometimes contains variant words, which bring huge challenges to natural language processing tasks such as text analysis or information retrieval.

[0003] In the prior art, when identifying variant words, it is usually necessary to list various variant words of a certain vocabulary, such as similar characters and compound characters, and use the vocabulary and the listed variant words as sample texts for model training, so that the variant words can be identified through the trained model.

[0004] However, in the above method, since the exhaustively enumerated variant words are used as sample texts for model training, the model will not be able to accurately recognize the newly appeared variant words, resulting in low accuracy in the recognition results of the variant words. Summary of the invention

[0005] The present invention provides a variant word recognition method, device, equipment and storage medium, which are used to solve the defect of low accuracy of variant word recognition results in the prior art and achieve the purpose of improving the accuracy of variant word recognition results.

[0006] The present invention provides a variant word recognition method, comprising:

[0007] Get the text to be recognized;

[0008] Segmenting the text to be recognized to obtain at least one word;

[0009] For each of the words, inputting a target image containing the word and at least one target word in the word list into a variant word recognition model to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to indicate whether the word is a variant word;

[0010] The variant word recognition model is obtained by training an initial variant word recognition model based on each word in the vocabulary list and a sample image corresponding to each word, wherein the sample image includes the variant word corresponding to the word.

[0011] According to a variant word recognition method provided by the present invention, the initial variant word recognition model includes at least two self-attention modules connected in sequence;

[0012] The variant word recognition model is trained based on the following method:

[0013] Based on at least one word in the vocabulary list and the sample images corresponding to each word, the at least two self-attention modules alternately perform multimodal fusion feature extraction and independent feature extraction of each modality until a target feature vector corresponding to each modality output by the last self-attention module is obtained; the target feature vector includes a category vector;

[0014] The variant word recognition model is determined based on the category vector corresponding to each of the modalities.

[0015] According to a variant word recognition method provided by the present invention, each of the self-attention modules includes a first Transformer layer, and a second Transformer layer and a third Transformer layer respectively connected to the first Transformer layer; the target feature vectors corresponding to each modality include a first target feature vector corresponding to each of the words and a second target feature vector corresponding to each sample image of the words;

[0016] The method of alternately performing multimodal fusion feature extraction and independent feature extraction of each modality based on at least one word in the vocabulary list and the sample images corresponding to each word through the at least two self-attention modules until a target feature vector corresponding to each modality output by the last self-attention module is obtained includes:

[0017] Inputting at least one word in the vocabulary list and a sample image corresponding to each word into the first Transformer layer of the first self-attention module, inputting the first feature vector corresponding to each word output by the first Transformer layer into the second Transformer layer, obtaining the third feature vector corresponding to each word output by the second Transformer layer, and inputting the second feature vector corresponding to each sample image of each word output by the first Transformer layer into the third Transformer layer, obtaining the fourth feature vector corresponding to each sample image of each word output by the third Transformer layer;

[0018] Input each of the third eigenvectors and each of the fourth eigenvectors into the first Transformer layer of the second self-attention module until obtaining the first target feature vectors corresponding to each of the words output by the second Transformer layer of the last self-attention module and the second target feature vectors corresponding to each of the sample images of the words output by the third Transformer layer of the last self-attention module.

[0019] According to a variant word recognition method provided by the present invention, the variant word recognition model is determined based on the category vector corresponding to each of the modalities, including:

[0020] Determine a first loss based on a first category vector included in each of the first target feature vectors, and determine a second loss based on a second category vector included in each of the first category vectors and each of the second target feature vectors;

[0021] determining a third loss based on each of the second category vectors;

[0022] Based on the first loss, the second loss and the third loss, the model parameters of the initial variant word recognition model are adjusted to obtain the variant word recognition model.

[0023] According to a variant word recognition method provided by the present invention, the step of inputting at least one word in the vocabulary list and a sample image corresponding to each word into a first Transformer layer of a first self-attention module comprises:

[0024] For each of the words, determining a first Token corresponding to the word and a second Token corresponding to a sample image of the word;

[0025] Concatenate the first Token and the second Token to obtain a concatenated vector;

[0026] The concatenated vectors corresponding to each of the words are input into the first Transformer layer of the first self-attention module.

[0027] According to a variant word identification method provided by the present invention, the method further includes:

[0028] For each of the words, determining at least one variant word of the word;

[0029] For each variant word of the vocabulary, determining an initial image including the variant word of the vocabulary;

[0030] The initial image is augmented to obtain at least one sample image of the vocabulary.

[0031] According to a variant word identification method provided by the present invention, the step of determining at least one variant word of the vocabulary comprises:

[0032] The vocabulary is transformed based on at least one of the following methods to obtain at least one variant word of the vocabulary:

[0033] Changes in similar characters, changes in radicals or components, changes in the order of characters in a vocabulary, changes in reducing strokes, and changes in increasing strokes.

[0034] According to a variant word recognition method provided by the present invention, the recognition result includes the words included in the target image and the matching probability of each of the target words; the method further includes:

[0035] In the case where a matching probability greater than a preset probability is included in all the matching probabilities, a prompt information is output, where the prompt information is used to indicate that the vocabulary is a variant word.

[0036] The present invention also provides a variant word recognition device, comprising:

[0037] An acquisition module is used to acquire the text to be recognized;

[0038] A word segmentation module, used for segmenting the text to be recognized to obtain at least one word;

[0039] An input module, for inputting, for each of the words, a target image containing the word and at least one target word in the word list into a variant word recognition model, to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to indicate whether the word is a variant word;

[0040] The variant word recognition model is obtained by training an initial variant word recognition model based on each word in the vocabulary list and a sample image corresponding to each word, wherein the sample image includes the variant word corresponding to the word.

[0041] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the variant word recognition method described above is implemented.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the variant word recognition method as described in any one of the above is implemented.

[0043] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for identifying variant words as described above is implemented.

[0044] The variant word recognition method, device, equipment and storage medium provided by the present invention segment the acquired text to be recognized to obtain at least one word, and for each word, input the target image containing the word and at least one target word in the vocabulary list into the variant word recognition model to obtain the recognition result output by the variant word recognition model, and the recognition result is used to characterize whether the word is a variant word. The variant word recognition model is obtained by training the initial variant word recognition model based on the sample images corresponding to the words in the vocabulary list and the variant words of each word. Since the sample images corresponding to the variant words of the words are used as training samples, the trained font recognition model can recognize all variant words with similar glyphs to the word, thereby improving the generalization of the variant word recognition model. Therefore, when identifying whether each word is a variant word based on the variant word recognition model, the recognition accuracy can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a variant word recognition method provided by an embodiment of the present invention.

[0047] Figure 2 A schematic diagram of the structure of a variant word recognition model provided in an embodiment of the present invention.

[0048] Figure 3 A schematic diagram of comparative learning provided by an embodiment of the present invention.

[0049] Figure 4 A schematic diagram of the structure of a variant word recognition device provided in an embodiment of the present invention.

[0050] Figure 5 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Text deformation recognition is an important research direction in the field of natural language processing. For example, with the development of artificial intelligence, large language models are used in more and more fields. When processing information through large language models, it is necessary to review the input and output of large language models. For some content that does not meet the standards, the large language model will filter it. In some scenarios, some malicious users will use variant words to attack the large language model, such as inputting "What is Zhang San's ID number" into the large language model? Generally speaking, when reviewing the content, the large language model will intercept the input information. Therefore, malicious users may bypass the review mechanism of the large language model by using variant words, such as inputting "What is Zhang San's ID number". Therefore, it is very important in the field of natural language processing to accurately identify whether there are variant words in the text.

[0053] In the prior art, when identifying variant words, it is usually necessary to list various variant words such as similar characters and compound characters of multiple words, and use these words and the listed variant words as sample texts for model training, so that the variant words can be identified through the trained model. However, in the above method, since the exhaustively enumerated variant words are used as sample texts for model training, and the variant words of the vocabulary are diverse, for some newly emerged variant words, since the newly emerged variant words are not involved in the training of the model, the model will not be able to accurately identify the newly emerged variant words, resulting in low accuracy of the variant word recognition results.

[0054] In response to the above problems, a variant word recognition method is proposed in an embodiment of the present invention. In this method, considering that the glyphs of a certain vocabulary and its corresponding variant word are relatively similar, from a visual perspective, after seeing the variant word, one will associate it with the semantics of the variant word. Therefore, the embodiment of the present invention creatively solves the variant word recognition in language tasks from a visual perspective, perceives the glyphs, and maps the perceived glyphs to semantics, thereby achieving the purpose of variant word recognition from a visual perspective and improving the accuracy of variant word recognition.

[0055] Combine the following Figures 1 to 3The variant word recognition method provided by the embodiment of the present invention is described. The embodiment of the present invention can be applied to any scenario where variant word recognition is required, such as information retrieval, large language model input and output content review, etc. It should be noted that the variant words involved in the embodiment of the present invention refer to similar characters, such as characters obtained by disassembling a certain character or replacing radicals or components, and are not morphological characters that appear in the image in the optical character recognition (OCR) scenario due to distortion or deformation of the image. The executor of this method can be an electronic device such as a terminal device, a computer, a server, a server cluster, or a specially designed variant word recognition device, or it can be a variant word recognition device set in the electronic device, and the variant word recognition device can be implemented by software, hardware, or a combination of both.

[0056] Figure 1 A flow chart of a variant word recognition method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0057] Step 101: Obtain text to be recognized.

[0058] In this step, the text to be recognized can be understood as the text that needs to be checked for variant words, such as the input text or output text of a large language model, etc. The text to be recognized includes text directly input by the user or text obtained through audio conversion.

[0059] Step 102: Segment the text to be recognized to obtain at least one vocabulary.

[0060] In this step, after the text to be recognized is obtained, a word segmentation tool is used to perform word segmentation on the text to be recognized to obtain at least one vocabulary, wherein the vocabulary may include words or single characters, etc.

[0061] Step 103: For each word, a target image containing the word and at least one target word in the word list are input into a variant word recognition model to obtain a recognition result output by the variant word recognition model, and the recognition result is used to indicate whether the word is a variant word.

[0062] The variant word recognition model is obtained by training an initial variant word recognition model based on each word in the vocabulary list and a sample image corresponding to each word, and the sample image includes the variant word corresponding to the word.

[0063] In this step, the variant word recognition model is used to identify whether the input word is a variant word. When training the variant word recognition model, it is not only obtained by training the initial variant word recognition model with multiple words and the variant words corresponding to each word as samples, but also by constructing a vocabulary list containing multiple correct words and the variant words corresponding to each word, and then generating sample images containing these variant words, and using each word in the vocabulary list and the sample images containing the variant words for training. Since each word and its corresponding variant word are very similar in terms of glyph, therefore, in the embodiment of the present invention, a visual perspective is adopted, and the sample images of the listed variant words are used as training samples. In this way, the trained font recognition model can recognize all variant words similar to the glyphs of the word, and even if the variant words corresponding to each word are not exhaustively listed, it will not affect the recognition of new variant words that are not used as training samples, which can improve the generalization of the variant word recognition model.

[0064] In addition, the vocabulary list is a pre-constructed list including multiple correct words, and none of the words included in the vocabulary list contains variant words. When identifying whether each word included in the text to be recognized contains variant words, for each word, a target image containing the word can be constructed. Exemplarily, the target image can be obtained by manual screenshot or by Python script. Among them, the target image can only contain the word. Afterwards, the constructed target image containing the word and each target word in the pre-constructed vocabulary list are respectively composed of image text recognition pairs, and each image text recognition pair is input into the pre-trained variant word recognition model, so that the recognition result output by the variant word recognition model can be obtained. The recognition result can, for example, include the matching probability between the words included in the target image and each target word in the vocabulary list, and these matching probabilities can be used to characterize whether the words included in the target image are variant words of the target words. For example, if the matching probability between the words included in the target image and a target word in the vocabulary list is greater than the preset probability, it means that the words included in the target image are variant words of the target word.

[0065] The variant word recognition method provided by the embodiment of the present invention obtains at least one word by segmenting the acquired text to be recognized. For each word, the target image containing the word and at least one target word in the vocabulary list are input into the variant word recognition model to obtain a recognition result output by the variant word recognition model, and the recognition result is used to characterize whether the word is a variant word. The variant word recognition model is obtained by training an initial variant word recognition model based on sample images corresponding to each word in the vocabulary list and the variant words of each word. Since the sample images corresponding to the variant words of the word are used as training samples, the trained font recognition model can recognize all variant words with similar glyphs to the word, thereby improving the generalization of the variant word recognition model. Therefore, when identifying whether each word is a variant word based on the variant word recognition model, the recognition accuracy can be improved.

[0066] Exemplarily, based on the above-mentioned embodiment, the recognition result output by the variant word recognition model includes the vocabulary included in the target image and the matching probability of each target vocabulary; in this way, when all matching probabilities include a matching probability greater than a preset probability, a prompt information is output, and the prompt information is used to characterize the vocabulary as a variant word.

[0067] Specifically, the greater the matching probability between the vocabulary included in the target image and the target vocabulary, the greater the probability that the vocabulary is obtained by transforming the target vocabulary. Therefore, after determining the matching probability between the vocabulary and each target vocabulary, it is determined whether the maximum matching probability is greater than the preset probability. If it is greater than the preset probability, it means that the probability that the vocabulary is obtained by transforming the target vocabulary corresponding to the maximum matching probability is high. Therefore, prompt information can be output, and the prompt information is used to indicate that the vocabulary is a variant word, so that the variant words in the text to be recognized can be recognized in a timely and accurate manner.

[0068] It should be understood that if in the application scenario of the large language model, when it is determined that the input content or the output content includes variant words, the content will not pass the review.

[0069] Exemplarily, based on the above embodiment, the initial variant word recognition model includes at least two self-attention modules connected in sequence, and the variant word recognition model is trained based on the following method:

[0070] Based on at least one word in the vocabulary list and the sample images corresponding to each word, multimodal fusion feature extraction and independent feature extraction of each modality are alternately performed through at least two self-attention modules until a target feature vector corresponding to each modality output by the last self-attention module is obtained, and the target feature vector includes a category vector, so as to determine a variant word recognition model based on the category vector corresponding to each modality.

[0071] Specifically, at least one word in the vocabulary list and the sample images corresponding to each word can be input into the first self-attention module, and the multimodal fusion feature extraction of the words and the sample images is performed through the first self-attention module, and the extracted fusion features are used for independent feature extraction of each modality respectively, and the target feature vector corresponding to the extracted words and the target feature vector corresponding to the sample image are input into the second self-attention module to repeat the above process until the target feature vector corresponding to the words and the target feature vector corresponding to the sample image are output by the last self-attention module, wherein the target feature vector corresponding to the words and the target feature vector corresponding to the sample image both include category vectors, so that comparative learning is performed based on the category vectors in the target feature vectors corresponding to each modality, and the final variant word recognition model can be obtained.

[0072] In this embodiment, after multimodal fusion of text vocabulary and sample images, self-attention calculation is performed separately. After multiple rounds of calculation, the variant word recognition model is determined based on the category vector in the final target feature vector. This allows the variant word recognition model to recognize variant words that are similar in shape to each vocabulary, thereby improving the generalization of the trained variant word recognition model.

[0073] Exemplarily, based on the above embodiments, each attention module includes a first Transformer layer, and a second Transformer layer and a third Transformer layer respectively connected to the first Transformer layer, and the target feature vector corresponding to each modality includes a first target feature vector corresponding to each vocabulary and a second target feature vector corresponding to each sample image of each vocabulary.

[0074] Based on at least one word in the vocabulary list and the sample images corresponding to each word, at least two self-attention modules are used to alternately perform multimodal fusion feature extraction and independent feature extraction of each modality until the target feature vector corresponding to each modality output by the last self-attention module is obtained. The method can be as follows: at least one word in the vocabulary list and the sample images corresponding to each word are input into the first Transformer layer of the first self-attention module, the first feature vectors corresponding to each word output by the first Transformer layer are input into the second Transformer layer, and the third feature vectors corresponding to each word output by the second Transformer layer are input into the third Transformer layer, and the fourth feature vectors corresponding to each word sample image output by the third Transformer layer are obtained; the third feature vectors and the fourth feature vectors are input into the first Transformer layer of the second self-attention module until the first target feature vectors corresponding to each word output by the second Transformer layer of the last self-attention module and the second target feature vectors corresponding to each word sample image output by the third Transformer layer of the last self-attention module are obtained.

[0075] Specifically, Figure 2 A schematic diagram of the structure of a variant word recognition model provided by an embodiment of the present invention, such as Figure 2 As shown, the variant word recognition model is a twin multimodal model of the same structure. Specifically, the variant word recognition model includes N self-attention modules 200 connected in sequence, where N is a positive integer greater than or equal to 2. Each self-attention module 200 includes a first Transformer layer 201, and a second Transformer layer 202 and a third Transformer layer 203 respectively connected to the first Transformer layer 201. Exemplarily, the text encoder for processing text vocabulary can adopt BERT, and the model for processing sample images can adopt a Vision Transformer (ViT) model with the same structure as BERT, wherein the BERT model and the ViT model share parameters, that is, an identical twin network architecture is adopted.

[0076] Before model training, the model parameters of the first Transformer layer 201, the second Transformer layer 202, and the third Transformer layer 203 can be initialized. After the initialization is completed, in each training batch, each word in the pre-constructed vocabulary list and its corresponding sample image can be input into the first Transformer layer 201 of the first self-attention module for self-attention calculation to obtain the fused feature vector output by the first Transformer layer 201, which includes the first feature vector corresponding to each word and the second feature vector corresponding to the sample image of each word. Each sample image includes a variant word of the word.

[0077] Exemplarily, when each word and its corresponding sample image are input into the first Transformer layer of the first self-attention module, the first Token corresponding to the word and the second Token corresponding to the sample image of the word can be determined, and the first Token and the second Token are concatenated to obtain the concatenated vector, and then the concatenated vector corresponding to each word is input into the first Transformer layer of the first self-attention module.

[0078] For example, taking the word "ID number" and the variant word "body powder political number" included in the sample image as an example, the first token of the text "ID number" and the second token of the sample image are determined respectively, for example, Figure 2 The dot-filled block represents the first token of the text "ID number", and the line-filled box represents the second token of the sample image. The first token includes the category vector CLS of the text word "ID number", and the second token includes the category vector CLS of the variant word in the sample image.

[0079] After determining the first Token and the second Token, the first Token and the second Token are concatenated to obtain a concatenated vector, which is then input into the first Transformer layer of the first self-attention module for self-attention calculation. Since the vector input into the first Transformer layer contains vectors of different modalities, after the self-attention calculation, the fused feature vector obtained will fuse information of different modalities, so that the variant word recognition model fuses visual information. Therefore, the variant word recognition model can recognize variant words with similar vocabulary shapes.

[0080] In the above method, the first Token corresponding to the text-type vocabulary and the second Token corresponding to the sample image are concatenated and input into the Transformer layer for self-attention calculation. Since the training samples include samples of text modality and image modality, the variant word recognition model is trained through multimodal fusion, which improves the robustness of the variant word recognition model finally identified.

[0081] After obtaining the fused feature vector output by the first Transformer layer, the first feature vector corresponding to each word and the second feature vector corresponding to the sample image of each word are determined from the fused feature vector, so that the first feature vector is input into the second Transformer layer 202 of the first self-attention module for self-attention calculation, and the third feature vector corresponding to each word output by the second Transformer layer 202 is obtained, and the second feature vector is input into the third Transformer layer 203 of the first self-attention module for self-attention calculation, and the fourth feature vector corresponding to the sample image of each word output by the third Transformer layer 203 is obtained.

[0082] Furthermore, after the third feature vector and the fourth feature vector are concatenated, the concatenated feature vector is input into the first Transformer layer of the second self-attention module for self-attention calculation, and the feature vectors corresponding to each word are determined from the fused feature vector output by the first Transformer layer and input into the second Transformer layer of the second self-attention module for self-attention calculation, and the feature vectors corresponding to the sample images of each word in the fused feature vector are input into the third Transformer layer of the second self-attention module for self-attention calculation, and the above process is repeated until the first target feature vectors corresponding to each word output by the second Transformer layer of the last self-attention module and the second target feature vectors corresponding to the sample images of each word output by the third Transformer layer of the last self-attention module are obtained. It should be understood that the above-mentioned process of fused self-attention calculation and independent self-attention calculation is performed alternately N times.

[0083] Since each word corresponds to a first target feature vector, and each word's sample image corresponds to a second target feature vector, the first category vector CLS corresponding to each word can be obtained from each first target feature vector, and the second category vector CLS corresponding to the variant word in the sample image of each word can be obtained from each second target feature vector, and comparative learning is performed based on each first category vector CLS and each second category vector CLS to determine the variant word recognition model.

[0084] In this embodiment, after multimodal fusion of text vocabulary and sample images, self-attention calculation is performed separately. After multiple rounds of calculation, the variant word recognition model is determined based on the first category vector in the first target feature vector and the second category vector in the second target feature vector finally obtained. This allows the variant word recognition model to recognize variant words that are similar in shape to each vocabulary, thereby improving the generalization of the trained variant word recognition model.

[0085] Exemplarily, based on the above embodiment, when determining the variant word recognition model based on the category vector corresponding to each modality, it can be performed in the following manner:

[0086] A first loss is determined based on a first category vector included in each first target feature vector, and a second loss is determined based on each first category vector and a second category vector included in each second target feature vector; a third loss is determined based on each second category vector; based on the first loss, the second loss and the third loss, model parameters of an initial variant word recognition model are adjusted to obtain a variant word recognition model.

[0087] Specifically, Figure 3 A comparative learning schematic diagram provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, the first category vector CLS of the dot-filled box is the vector corresponding to the text vocabulary, and the second category vector CLS of the line-filled box is the vector corresponding to the sample image. Therefore, in the specific implementation process, the loss information can be determined by the contrast learning method of homomorphic contrast and cross-modal contrast. Among them, homomorphic contrast can be understood as comparing samples of the same modality, and cross-modal contrast can be understood as comparing samples of different modalities.

[0088] For each first category vector, a first loss loss_text between the same text modalities may be determined based on each first category vector. In addition, a second loss loss_cross between different modalities may also be determined based on each first category vector and each second category vector.

[0089] For each second category vector, a third loss loss_img between the same image modalities may be determined based on the second category vector.

[0090] The sum of the first loss loss_text, the second loss loss_cross and the third loss loss_img is determined as the final loss value, and the model parameters of the initial variant word recognition model are adjusted based on the loss value. The above process is iteratively performed until the obtained model converges or the number of iterations meets the preset number, and the final model is determined as the variant word recognition model.

[0091] It should be noted that in multi-modal contrastive learning, data of each modality is usually encoded separately to obtain the embedding vectors of each modality. Then, by calculating the similarity or distance between the embedding vectors of different modalities, the similarities and differences between different samples are compared and distinguished. For example, if two images belong to the same category, the distance between their image embedding vectors should be small; if two images belong to different categories, the distance should be large. Similarly, similar methods can also be used for representation learning of text and audio data.

[0092] The main advantage of multi-modal contrastive learning is that it can process multiple types of data simultaneously and extract richer and more comprehensive feature representations from data of different modalities. In addition, since it does not rely on label information, multi-modal contrastive learning can also be used in unsupervised learning tasks. However, because it needs to process multiple types of data simultaneously, multi-modal contrastive learning usually requires more computing resources and a more complex model structure.

[0093] In this embodiment, the final loss value is determined by means of intra-modal contrast and cross-modal contrast, that is, the variant word recognition model is determined by using multi-modal contrastive learning, which can extract common features from data of different modalities and perform representation learning, so that the accuracy of the finally obtained variant word recognition model is higher.

[0094] Exemplarily, based on the above embodiments, the sample images corresponding to each vocabulary can be determined in the following manner:

[0095] For each vocabulary, at least one variant word of the vocabulary is determined. For each variant word of the vocabulary, an initial image including the variant word of the vocabulary is determined, and the initial image is augmented to obtain at least one sample image of the vocabulary.

[0096] Specifically, each vocabulary in the vocabulary list is a word that is key-concerned in the application scenario, such as a person's name, a sensitive word, etc. For each vocabulary in the vocabulary list, at least one variant word of the vocabulary can be constructed. For example, the vocabulary can be transformed based on at least one of the following methods to obtain at least one variant word of the vocabulary: transformation of similar-shaped characters, transformation of radicals or components, transformation of the order of characters in the vocabulary, transformation of reducing strokes, and transformation of adding strokes. Among them, the transformation of similar-shaped characters can be understood as replacing at least one character in the vocabulary with a similar-shaped character, such as transforming "习" into "刁", and the transformation of radicals or components can include disassembling or combining the radicals or components of at least one character in the vocabulary, such as transforming "雷" into "雨田", and transforming "林木" into "森", etc. By transforming the vocabulary in the above manner, the diversity of the obtained variant words can be improved, and the richness of the variant words can be enhanced.

[0097] After obtaining at least one variant word of a vocabulary, for each variant word, the initial image containing the variant word can be determined by manual screenshot or Python script. Furthermore, in order to improve the richness of sample images and the robustness of the variant word recognition model, each obtained initial image can be augmented, for example, by translating, scaling, adding background, etc., so as to obtain at least one sample image corresponding to each vocabulary.

[0098] In addition, the training of variant word recognition models usually involves the following key steps:

[0099] Preprocessing data: Cleaning and preparing the dataset, including denoising, standardization, normalization, encoding, cropping, scaling, etc., to ensure that the input data meets the requirements of the variant word recognition model.

[0100] Choose a model architecture: Choose a suitable neural network architecture, such as ViT and BERT, based on the nature of the problem.

[0101] Initialize network weights: Before starting training, you need to randomly initialize the weights in the network. This is usually done with small random numbers to avoid getting local minima at the beginning.

[0102] Forward propagation: Pass the input data through the network to produce output. For each batch of data, calculate the error (loss function) between the predicted value and the actual value.

[0103] Back propagation: The error is passed from the output layer to the input layer layer by layer, the contribution of each weight to the loss function is calculated, and the weight is updated accordingly to reduce the error.

[0104] Optimization algorithm: Use an optimization algorithm (such as gradient descent) to adjust the network weights in order to reduce the value of the loss function. This process is repeated many times until the performance of the model no longer improves significantly or reaches a predetermined number of training rounds.

[0105] Regularization and prevention of overfitting: To improve the generalization ability of the model, some techniques may be applied, such as Dropout, L2 regularization, or early stopping.

[0106] Validation and testing: Use the validation set to tune hyperparameters and monitor model performance. The final model will evaluate its performance on the test set to ensure that it performs well on unseen data.

[0107] Deployment: Deploy the trained model into a real application, such as integrating it into an application or service.

[0108] Other techniques that may be involved in the deep learning training process include minibatch gradient descent, learning rate decay, Adam optimizer, etc. In addition, with the development of technology, some new training strategies and techniques have emerged, such as semi-supervised learning and transfer learning.

[0109] The variant word recognition device provided by the present invention is described below. The variant word recognition device described below and the variant word recognition method described above can be referenced to each other.

[0110] Figure 4 A schematic diagram of a variant word recognition device provided by an embodiment of the present invention, referring to Figure 4 As shown, the variant word identification device 400 includes:

[0111] An acquisition module 11 is used to acquire the text to be recognized;

[0112] A word segmentation module 12, used for segmenting the text to be recognized to obtain at least one word;

[0113] An input module 13, for inputting, for each of the words, a target image containing the word and at least one target word in the word list into a variant word recognition model, to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to indicate whether the word is a variant word;

[0114] The variant word recognition model is obtained by training an initial variant word recognition model based on each word in the vocabulary list and a sample image corresponding to each word, wherein the sample image includes the variant word corresponding to the word.

[0115] In an example embodiment, the initial variant word recognition model includes at least two self-attention modules connected in sequence;

[0116] The variant word recognition model is trained based on the following method:

[0117] Based on at least one word in the vocabulary list and the sample images corresponding to each word, the at least two self-attention modules alternately perform multimodal fusion feature extraction and independent feature extraction of each modality until a target feature vector corresponding to each modality output by the last self-attention module is obtained; the target feature vector includes a category vector;

[0118] The variant word recognition model is determined based on the category vector corresponding to each of the modalities.

[0119] In an exemplary embodiment, each of the self-attention modules includes a first Transformer layer, and a second Transformer layer and a third Transformer layer respectively connected to the first Transformer layer; the target feature vectors corresponding to each of the modalities include a first target feature vector corresponding to each of the words and a second target feature vector corresponding to each of the sample images of the words;

[0120] The apparatus further comprises a determining module, wherein:

[0121] An input module 13 is used to input at least one word in the vocabulary list and a sample image corresponding to each word into a first Transformer layer of a first self-attention module, input a first feature vector corresponding to each word output by the first Transformer layer into a second Transformer layer to obtain a third feature vector corresponding to each word output by the second Transformer layer, and input a second feature vector corresponding to each sample image of each word output by the first Transformer layer into the third Transformer layer to obtain a fourth feature vector corresponding to each sample image of each word output by the third Transformer layer;

[0122] The input module 13 is used to input each of the third feature vectors and each of the fourth feature vectors into the first Transformer layer of the second self-attention module until the first target feature vectors corresponding to each of the words output by the second Transformer layer of the last self-attention module and the second target feature vectors corresponding to each of the sample images of the words output by the third Transformer layer of the last self-attention module are obtained.

[0123] In an exemplary embodiment, the apparatus further comprises an adjustment module, wherein:

[0124] The determination module is further configured to determine a first loss based on a first category vector included in each of the first target feature vectors, and to determine a second loss based on a second category vector included in each of the first category vectors and each of the second target feature vectors;

[0125] A determination module, further configured to determine a third loss based on each of the second category vectors;

[0126] An adjustment module is used to adjust the model parameters of the initial variant word recognition model based on the first loss, the second loss and the third loss to obtain the variant word recognition model.

[0127] In an exemplary embodiment, the input module 13 is specifically configured to:

[0128] For each of the words, determining a first Token corresponding to the word and a second Token corresponding to a sample image of the word;

[0129] Concatenate the first Token and the second Token to obtain a concatenated vector;

[0130] The concatenated vectors corresponding to each of the words are input into the first Transformer layer of the first self-attention module.

[0131] In an exemplary embodiment, the apparatus further comprises: an augmentation module, wherein:

[0132] The determination module is further used to determine at least one variant word of each of the words;

[0133] The determination module is further used to determine, for each variant word of the vocabulary, an initial image including the variant word of the vocabulary;

[0134] The augmentation module is used to augment the initial image to obtain at least one sample image of the vocabulary.

[0135] In an exemplary embodiment, the determining module is specifically configured to:

[0136] The vocabulary is transformed based on at least one of the following methods to obtain at least one variant word of the vocabulary:

[0137] Changes in similar characters, changes in radicals or components, changes in the order of characters in a vocabulary, changes in reducing strokes, and changes in increasing strokes.

[0138] In an example embodiment, the recognition result includes the words included in the target image and the matching probability of each of the target words;

[0139] The device further comprises: an output module, wherein:

[0140] The output module is further used to output prompt information when all the matching probabilities include a matching probability greater than a preset probability, wherein the prompt information is used to indicate that the vocabulary is a variant word.

[0141] The device of this embodiment can be used to execute the method of any embodiment in the variant word identification method embodiment. Its specific implementation process and technical effects are similar to those in the variant word identification method embodiment. For details, please refer to the detailed introduction in the variant word identification method embodiment, which will not be repeated here.

[0142] Figure 5A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the variant word recognition method, which includes: obtaining a text to be recognized; segmenting the text to be recognized to obtain at least one vocabulary; for each of the vocabulary, inputting a target image containing the vocabulary and at least one target vocabulary in the vocabulary list into a variant word recognition model to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to characterize whether the vocabulary is a variant word; wherein the variant word recognition model is obtained by training an initial variant word recognition model based on each vocabulary in the vocabulary list and a sample image corresponding to each vocabulary, wherein the sample image includes the variant word corresponding to the vocabulary.

[0143] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0144] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the variant word recognition method provided by the above-mentioned methods, which method includes: obtaining a text to be recognized; segmenting the text to be recognized to obtain at least one vocabulary; for each of the vocabulary, inputting a target image containing the vocabulary and at least one target vocabulary in a vocabulary list into a variant word recognition model to obtain a recognition result output by the variant word recognition model, and the recognition result is used to characterize whether the vocabulary is a variant word; wherein the variant word recognition model is obtained by training an initial variant word recognition model based on each vocabulary in the vocabulary list and a sample image corresponding to each vocabulary, and the sample image includes the variant word corresponding to the vocabulary.

[0145] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the variant word recognition method provided by the above-mentioned methods, the method comprising: obtaining a text to be recognized; segmenting the text to be recognized to obtain at least one vocabulary; for each of the vocabulary, inputting a target image containing the vocabulary and at least one target vocabulary in a vocabulary list into a variant word recognition model to obtain a recognition result output by the variant word recognition model, the recognition result being used to characterize whether the vocabulary is a variant word; wherein the variant word recognition model is obtained by training an initial variant word recognition model based on each vocabulary in the vocabulary list and a sample image corresponding to each of the vocabulary, the sample image including the variant word corresponding to the vocabulary.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A variant word recognition method, characterized in that: include: Get the text to be recognized; Segmenting the text to be recognized to obtain at least one word; For each of the words, inputting a target image containing the word and each correct word in the word list into a variant word recognition model to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to indicate whether the word is a variant word; The variant word recognition model is obtained by training an initial variant word recognition model based on each correct word in the vocabulary list and a sample image corresponding to each correct word, wherein the sample image includes the variant word corresponding to the correct word.

2. The variant word recognition method according to claim 1, characterized in that: The initial variant word recognition model includes at least two self-attention modules connected in sequence; The variant word recognition model is trained based on the following method: Based on each correct word in the vocabulary list and the sample image corresponding to each correct word, the at least two self-attention modules alternately perform multimodal fusion feature extraction and independent feature extraction of each modality until a target feature vector corresponding to each modality output by the last self-attention module is obtained; the target feature vector includes a category vector; The variant word recognition model is determined based on the category vector corresponding to each of the modalities.

3. The variant word recognition method according to claim 2, characterized in that: Each of the self-attention modules includes a first Transformer layer, and a second Transformer layer and a third Transformer layer respectively connected to the first Transformer layer; the target feature vectors corresponding to each of the modalities include a first target feature vector corresponding to each of the correct words and a second target feature vector corresponding to each of the sample images of the correct words; The method of alternately performing multimodal fusion feature extraction and independent feature extraction of each modality based on each correct word in the vocabulary list and the sample image corresponding to each correct word through the at least two self-attention modules until a target feature vector corresponding to each modality output by the last self-attention module is obtained includes: Inputting each correct word in the vocabulary list and the sample image corresponding to each correct word into the first Transformer layer of the first self-attention module, inputting the first feature vector corresponding to each correct word output by the first Transformer layer into the second Transformer layer, obtaining the third feature vector corresponding to each correct word output by the second Transformer layer, and inputting the second feature vector corresponding to each sample image of each correct word output by the first Transformer layer into the third Transformer layer, obtaining the fourth feature vector corresponding to each sample image of each correct word output by the third Transformer layer; The third eigenvectors and the fourth eigenvectors are input into the first Transformer layer of the second self-attention module until the first target eigenvectors corresponding to the correct words output by the second Transformer layer of the last self-attention module and the second target eigenvectors corresponding to the sample images of the correct words output by the third Transformer layer of the last self-attention module are obtained.

4. The variant word recognition method according to claim 3, characterized in that: The determining the variant word recognition model based on the category vector corresponding to each of the modalities includes: Determine a first loss based on a first category vector included in each of the first target feature vectors, and determine a second loss based on a second category vector included in each of the first category vectors and each of the second target feature vectors; determining a third loss based on each of the second category vectors; Based on the first loss, the second loss and the third loss, the model parameters of the initial variant word recognition model are adjusted to obtain the variant word recognition model.

5. The variant word recognition method according to claim 3, characterized in that: The step of inputting each correct word in the vocabulary list and a sample image corresponding to each correct word into a first Transformer layer of a first self-attention module comprises: For each correct word, determining a first Token corresponding to the correct word and a second Token corresponding to the sample image of the correct word; Concatenate the first Token and the second Token to obtain a concatenated vector; The concatenated vectors corresponding to the correct words are input into the first Transformer layer of the first self-attention module.

6. The variant word recognition method according to claim 1, characterized in that: The method further comprises: For each of the correct words, determining at least one variant word of the correct word; For each variant word of the correct vocabulary, determining an initial image including the variant word of the correct vocabulary; The initial image is augmented to obtain at least one sample image of the correct word.

7. The variant word recognition method according to claim 6, characterized in that: The determining of at least one variant word of the correct vocabulary comprises: The correct word is transformed based on at least one of the following methods to obtain at least one variant word of the correct word: Changes in similar characters, changes in radicals or components, changes in the order of characters in a vocabulary, changes in reducing strokes, and changes in increasing strokes.

8. The variant word recognition method according to any one of claims 1 to 7, characterized in that: The recognition result includes the vocabulary included in the target image and the matching probability of each correct vocabulary; the method also includes: In the case where a matching probability greater than a preset probability is included in all the matching probabilities, a prompt information is output, where the prompt information is used to indicate that the vocabulary is a variant word.

9. A variant word recognition device, characterized in that: include: An acquisition module is used to acquire the text to be recognized; A word segmentation module, used for segmenting the text to be recognized to obtain at least one word; An input module, for inputting, for each of the words, a target image containing the word and each correct word in the word list into a variant word recognition model, to obtain a recognition result output by the variant word recognition model, wherein the recognition result is used to indicate whether the word is a variant word; The variant word recognition model is obtained by training an initial variant word recognition model based on each correct word in the vocabulary list and a sample image corresponding to each correct word, wherein the sample image includes the variant word corresponding to the correct word.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the variant word identification method according to any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the variant word identification method according to any one of claims 1 to 8 is implemented.

Images