Model training, image and text recognition method, device, electronic device and storage medium

Abstract

The present invention provides a model training, image and text recognition method, device, electronic device and storage medium, wherein the method includes: extracting visual features and semantic features of sample images respectively based on the visual branch and semantic branch of the initial recognition model; determining visual constraint loss based on the difference between the visual features of multiple sample images with consistent characters but different orderings; determining semantic constraint loss based on the difference between the semantic features of multiple sample images with consistent semantics and / or the difference between the semantic features of multiple sample images with similar semantics; iterating parameters of the visual branch based on the visual constraint loss, and iterating parameters of the semantic branch based on the semantic constraint loss, and determining a trained image and text recognition model based on the initial recognition model after the parameter iteration. The method, device, electronic device and storage medium provided by the present invention ensure the semantic rationality and recognition accuracy of image and text recognition in complex scenes.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model training, image and text recognition method, device, electronic equipment and storage medium. Background Art

[0002] Image and text recognition technology is an important part of human-computer interaction and is widely used in medical, education, finance and other fields. However, in complex natural scenes (such as characters are occluded, blurred, and the background is complex), existing image and text recognition technology still faces significant challenges. Traditional methods mainly rely on deep learning models to extract visual features, but usually lack supervision at the semantic level, resulting in errors in the recognition results that do not conform to language habits or contextual logic. For example, the model may not be able to accurately distinguish homographs or infer the reasonable content of occluded characters.

[0003] This technical path lacking semantic supervision also exposes the limitations of visual feature extraction methods. Because the model focuses too much on local visual features and lacks effective modeling of the relationship between characters and the global characteristics of the text and image content, it is easy to cause the recognition results to be inaccurate or lack logic. Especially in scenes with complex backgrounds or large interference, the model has difficulty extracting key information, which further limits the scope of application and performance of text and image recognition technology. Summary of the invention

[0004] The present invention provides a model training, image and text recognition method, device, electronic equipment and storage medium, which are used to solve the defect of poor image and text recognition effect in related technologies.

[0005] The present invention provides a model training method, comprising the following steps: Based on the visual branch and semantic branch of the initial recognition model, the visual features and semantic features of the sample image are extracted respectively; Determine a visual constraint loss based on the difference between visual features of multiple sample images with consistent characters but different orderings; Determine a semantic constraint loss based on differences between semantic features of a plurality of sample images with consistent semantics and / or differences between semantic features of a plurality of sample images with similar semantics; Based on the visual constraint loss, parameters of the visual branch are iterated, and based on the semantic constraint loss, parameters of the semantic branch are iterated, and based on the initial recognition model after the parameter iteration, a trained image and text recognition model is determined.

[0006] A model training method provided by the present invention also includes: Extracting reference semantic features of the sample image based on a large language model; Determining a distillation loss based on a difference between the reference semantic feature and the semantic feature of the sample image; Based on the distillation loss, parameters of the semantic branch are iterated.

[0007] According to a model training method provided by the present invention, the determining of the distillation loss based on the difference between the reference semantic feature and the semantic feature of the sample image includes: Extracting character semantic features of each character block of the sample image based on the semantic branch of the initial recognition model; The distillation loss is determined based on the difference between the reference semantic feature and the semantic feature of the sample image, and the difference between the semantic feature of each character in the reference semantic feature and the character semantic feature of each character block of the sample image.

[0008] According to a model training method provided by the present invention, the visual constraint loss is determined based on the difference between the visual features of multiple sample images with consistent characters but different orderings, including: Determining the visual constraint loss based on differences between visual features of a plurality of sample images having consistent characters but different orderings, and differences between the visual features and class center features of the sample images; The class center feature of the sample image is determined based on the classification features of all characters contained in the sample image.

[0009] A model training method provided by the present invention also includes: Based on the visual branch and semantic branch of the encoder, the pre-trained visual features and pre-trained semantic features of the pre-trained image are extracted respectively; The encoder is pre-trained based on the orthogonal constraint between the pre-trained visual features and the pre-trained semantic features, and the initial recognition model is determined based on the pre-trained encoder.

[0010] A model training method provided by the present invention also includes: Predicting the order of characters in the pre-trained image based on pre-trained semantic features of the pre-trained image with disordered characters; Based on the character sequence, a semantic branch of the encoder is pre-trained.

[0011] The present invention provides a method for image and text recognition, comprising the following steps: Determine an image to be identified; Based on the image-text recognition model, performing image-text recognition on the image to be recognized; The image and text recognition model is obtained based on the model training method.

[0012] The present invention also provides a model training device, comprising the following modules: A feature extraction unit, used for extracting visual features and semantic features of the sample image based on the visual branch and the semantic branch of the initial recognition model; A visual constraint unit, for determining a visual constraint loss based on differences between visual features of a plurality of sample images having the same characters but in different orderings; A semantic constraint unit, configured to determine a semantic constraint loss based on a difference between semantic features of a plurality of sample images with consistent semantics and / or a difference between semantic features of a plurality of sample images with similar semantics; A parameter iteration unit is used to perform parameter iteration on the visual branch based on the visual constraint loss, and to perform parameter iteration on the semantic branch based on the semantic constraint loss, and to determine a trained image and text recognition model based on an initial recognition model after parameter iteration.

[0013] The present invention also provides a graphic recognition device, comprising the following modules: An image determination unit, used for determining an image to be identified; An image and text recognition unit, used for performing image and text recognition on the image to be recognized based on an image and text recognition model; The image and text recognition model is obtained based on the model training method.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the processor implements any one of the above-mentioned model training methods or image and text recognition methods.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described model training methods or image-text recognition methods.

[0016] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned model training methods or image-text recognition methods.

[0017] The model training, image and text recognition method, device, electronic device and storage medium provided by the present invention solve the problem that the recognition results do not conform to language habits or contextual logic due to the lack of semantic-level supervision by decoupling visual features and semantic features and performing constraint optimization on both. The visual branch's ability to extract visual features is strengthened through visual constraints, and the semantic branch's ability to understand global and local semantic features is enhanced through semantic constraints, effectively reducing the interference of visual and semantic information. The present invention significantly improves the semantic rationality and recognition accuracy of image and text recognition in complex scenarios, and has stronger applicability and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flow chart of the model training method provided by the present invention.

[0020] Figure 2 It is a flow chart of the image and text recognition method provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the model training device provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the image and text recognition device provided by the present invention.

[0023] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Image and text recognition technology is an important part of human-computer interaction and is widely used in the fields of medical care, education, finance, etc. It plays a key role in practical scenarios such as text extraction from electronic medical records, automatic marking of examination papers, bill and contract management, and billboard text recognition.

[0026] However, as application scenarios become increasingly complex, existing image and text recognition technologies still face significant challenges in some natural scenarios. For example, in the case of complex backgrounds, character occlusion, lighting changes, or blurred text, the results of image and text recognition are often unsatisfactory. This phenomenon not only limits the widespread application of technology, but also puts higher demands on the adaptability and generalization ability of related algorithms.

[0027] The current mainstream image and text recognition methods usually rely on the extraction of visual features by deep learning models. Although the ability to extract visual features has been greatly improved with the development of deep learning technology, the existing technology still has great limitations when dealing with complex scenarios due to the lack of effective supervision at the semantic level. This lack of semantic supervision often leads to the recognition results of the model not conforming to language habits or contextual logic. For example, when identifying homographs, the model may make incorrect judgments due to insufficient semantic understanding; when characters are occluded, it is difficult for the model to reasonably infer the missing content. These problems not only affect the accuracy of recognition, but also reduce the usability of the results.

[0028] In addition, this technical path that relies on visual features exposes more of its limitations. Existing models usually focus too much on the extraction of local visual features, and lack the ability to model the relationship between characters and the global characteristics of graphic content. This limitation makes the model particularly weak in scenes with complex backgrounds or large interference. For example, in the case of complex backgrounds and disordered character arrangement, the model may not be able to effectively distinguish between key text and noise, resulting in the final output results that are neither accurate nor logically reasonable. Worse, when the characters are arranged irregularly or some information is lost, the model's recognition ability may completely collapse.

[0029] In view of the above problems, an embodiment of the present invention provides a model training method, wherein the model training method is specifically used for training a picture and text recognition model. Figure 1 It is a flow chart of the model training method provided by the present invention, such as Figure 1 As shown, the method includes: Step 110 , based on the visual branch and the semantic branch of the initial recognition model, respectively extract the visual features and the semantic features of the sample image.

[0030] Specifically, the initial recognition model is an initial model used for image and text recognition, and may be a pre-trained model or a model obtained by parameter initialization.

[0031] The initial recognition model includes a visual branch and a semantic branch, and the visual branch and the semantic branch are independent of each other. The visual branch is used to extract the visual features of the sample image, and the semantic branch is used to extract the semantic features of the sample image.

[0032] Among them, the visual branch is mainly used to capture the appearance features of characters in the image, including information such as the shape, size, texture, and position of the characters. In the feature extraction process, the visual branch only focuses on the pixel features and geometric structure of the characters in the image, and does not rely on their semantic content. For example, in the process of training the visual branch, you can generate images with multiple arrangements by randomly swapping the positions of the characters in the sample image. By constraining the visual branch to extract the visual features of these images as similar as possible, it ensures that the visual features can focus on the appearance attributes of the characters, regardless of the semantics formed by the character arrangement.

[0033] The semantic branch is mainly used to extract the semantic features of text in images, that is, to understand the order of characters and their contextual meanings. For example, in the process of training the semantic branch, the model can be trained to effectively predict the correct order of text in complex scenarios by segmenting and shuffling the order of characters, thereby capturing the semantic information of the text. For another example, you can consider using the feature distillation technology of the language model, using the pre-trained large language model as the teacher model, and guiding the semantic branch to learn the semantic features of characters, so that the model can better understand the global semantic relationship of text.

[0034] In the process of extracting features, the initial recognition model processes the sample image separately through the visual branch and the semantic branch. The visual branch extracts the visual features of the characters in the sample image, and the semantic branch extracts the semantic features of the characters in the sample image.

[0035] Step 120 , determining a visual constraint loss based on the difference between visual features of a plurality of sample images having the same characters but in different order.

[0036] Specifically, in order to ensure that the visual branch focuses on the appearance information of the characters in the image and is not affected by the order of character arrangement, the model needs to train the robustness of the visual branch through a set of sample images with the same character content but different character orders. First, the position of each character is marked from the original sample image using the character position detection module. Then, by randomly swapping the positions of these characters, multiple new images with different arrangement orders are generated. For example, for a sample image containing the character sequence "ABC", images with different arrangements such as "BCA" and "CAB" can be generated. These newly generated images constitute the same set of positive example pairs because their character content is consistent but the order of characters is different. This way of data generation can effectively break the semantic association of characters, so that the training of the visual branch focuses only on the appearance characteristics of the characters.

[0037] Next, the sample images of these positive example pairs can be input into the visual branch through step 110, and the visual features of each sample image can be extracted through the visual branch. Thus, the visual features of each of these sample images based on the output of the visual branch can be obtained. Ideally, these visual features should be highly similar, because the character content in the sample image has not changed, only the character order has changed, and the extraction of visual features should be independent of the semantics reflected by the character order. In order to quantify the consistency of the representation of these images by the visual branch, the visual constraint loss can be calculated.

[0038] Specifically, the visual constraint loss can be determined by calculating the feature similarity between the positive example pairs. It can be understood that, for the visual features of multiple sample images with the same characters but different orderings, the smaller the difference between these visual features, the smaller the visual constraint loss, and conversely, the greater the difference between these visual features, the greater the visual constraint loss.

[0039] In addition, the visual constraint loss can also be calculated by contrastive learning. That is, in the calculation of the visual constraint loss, not only the difference between the visual features of multiple sample images with consistent characters but different orderings is considered, but also the difference between these visual features and the visual features of sample images with inconsistent characters is considered, thereby optimizing the feature extraction ability of the visual branch. It can be understood that the greater the difference between the visual features of sample images with inconsistent characters, the smaller the visual constraint loss, and conversely, the smaller the difference between these visual features, the greater the visual constraint loss.

[0040] Furthermore, the infoNCE loss function, mean square error loss function and other methods can be used to calculate the visual constraint loss.

[0041] The visual constraint loss can effectively strengthen the consistency of the visual branch in extracting character appearance features, making it highly robust and stable in scenes where the character order changes. This constraint mechanism ensures that the visual branch focuses on the visual information of the image without being disturbed by character arrangement or other semantic-related factors.

[0042] Step 130 : determining a semantic constraint loss based on differences between semantic features of a plurality of sample images with consistent semantics and / or differences between semantic features of a plurality of sample images with similar semantics.

[0043] Specifically, the core goal of the semantic branch is to accurately extract the global and local semantic information of the sample image, while maintaining the consistency of semantic features for semantically consistent sample images and reflecting the difference of appropriate semantic features for semantically similar sample images. By applying constraints to semantically consistent samples and semantically similar samples separately, or combining the two to constrain, the semantic branch can better learn semantic features.

[0044] For semantically consistent sample images, the text content of these images is exactly the same, but the appearance may change due to data augmentation, such as font, background color, noise, etc. For example, a sample containing the text "HELLO WORLD" can generate multiple variants by changing the font style, adjusting the brightness, etc. The semantic branch needs to extract stable and consistent semantic features between these samples to ensure that the model can focus on the content itself without being disturbed by appearance changes.

[0045] For sample images with similar semantics, these images have certain similarities in content, but have local differences. For example, replacing "HELLO WORLD" with "HELLO WORLDS" or "HELLO WLRLD" generates new samples. The semantic branch needs to be able to extract the overall semantic similarity of these samples while reflecting the subtle differences in local content. By imposing constraints on the feature distribution of semantically similar samples, the model can learn a refined semantic hierarchy and has the ability to distinguish similar semantic content.

[0046] In specific applications, the semantic consistency constraints obtained from semantically consistent sample images and the semantic distinction constraints obtained from semantically similar sample images can be used independently or in combination according to needs. When the semantic branch needs to process diverse representations of the same semantic content in the sample, the semantic consistency constraints can be used as the focus; when it is necessary to distinguish subtle changes in the semantic content in the sample, it is more appropriate to use semantic distinction constraints. In complex scenarios, such as when it is necessary to distinguish words with similar spellings and to process semantically equivalent sentences, semantic consistency and semantic distinction constraints can be introduced at the same time, combining the advantages of the two to comprehensively improve the model's ability to understand semantic features.

[0047] It can be understood that, for multiple sample images with consistent semantics, the smaller the difference between the semantic features of these sample images, the smaller the semantic constraint loss, and conversely, the greater the difference between the semantic features of these sample images, the greater the semantic constraint loss; and for multiple sample images with similar semantics, the smaller the difference between the semantic features of these sample images, the greater the semantic constraint loss, and conversely, the greater the difference between the semantic features of these sample images, the smaller the semantic constraint loss.

[0048] Through the above constraint mechanism, the semantic branch can better extract and represent the semantic features of samples, and show higher generalization and differentiation capabilities when dealing with semantically complex scenes. This flexible constraint design also provides strong support for efficient model training and semantic feature learning.

[0049] Step 140, based on the visual constraint loss, perform parameter iteration on the visual branch, and based on the semantic constraint loss, perform parameter iteration on the semantic branch, and determine a trained image and text recognition model based on the initial recognition model after the parameter iteration.

[0050] Specifically, the training process of the model is guided by the specific constraints of the visual branch and the semantic branch. By gradually optimizing the parameters of the two parts, an image and text recognition model that can adapt to complex scenes is finally obtained.

[0051] During the training of the visual branch, the model adjusts the parameters of the visual branch according to the visual constraint loss to enhance its ability to represent the appearance features of characters. By continuously minimizing the visual constraint loss, the model can more stably extract visual features such as the shape, size, and texture of characters while avoiding interference from semantic information. After each iteration, the parameters of the visual branch are gradually adjusted to a state where visual information can be captured efficiently, thereby ensuring that the model can still accurately recognize the visual features of characters in different fonts and complex backgrounds.

[0052] The training of the semantic branch is based on parameter optimization of semantic constraint loss, aiming to enhance the model's global and local semantic understanding of text content. During the training process, the model constrains the distribution of semantic features obtained from semantically consistent sample images and semantically similar sample images, so that the semantic branch gradually learns to distinguish between global semantic consistency and local semantic differences. Parameter iteration of the semantic branch can further improve the model's adaptability in complex semantic scenarios. For example, after multiple iterations, the semantic branch can more accurately identify sample images with similar semantics but different contents, while maintaining a stable feature representation of semantically consistent sample images.

[0053] After multiple rounds of parameter iterations, the visual branch and the semantic branch have respectively completed the optimization of their specific constraints, and finally formed an initial recognition model with updated parameters. At this stage, the feature extraction capabilities of the visual branch and the semantic branch have achieved a high degree of robustness and generalization. At this point, the model integrates the features of the two branches of vision and semantics, and combines their representation capabilities to determine the recognition result corresponding to the image, that is, to achieve recognition of the text contained in the image. The resulting model can achieve efficient image and text recognition in complex natural scenes. It can not only accurately extract the appearance features of characters, but also understand the semantic logic of text content, thereby adapting to diverse practical application needs.

[0054] The model training method provided in the embodiment of the present invention solves the problem that the recognition results do not conform to language habits or contextual logic due to the lack of supervision at the semantic level by decoupling visual features and semantic features and performing constraint optimization on both. The visual branch's ability to extract visual features is strengthened through visual constraints, and the semantic branch's ability to understand global and local semantic features is enhanced through semantic constraints, effectively reducing the interference of visual and semantic information. The method provided in the embodiment of the present invention significantly improves the semantic rationality and recognition accuracy of image and text recognition in complex scenarios, and has stronger applicability and generalization capabilities.

[0055] Based on the above embodiment, the model training method further includes: Extracting reference semantic features of the sample image based on a large language model; Determining a distillation loss based on a difference between the reference semantic feature and the semantic feature of the sample image; Based on the distillation loss, parameters of the semantic branch are iterated.

[0056] Specifically, a large language model (LLM) is a natural language processing (NLP) model with a large number of parameters. The number of model parameters and / or the complexity of the model structure exceeds the preset threshold. The model processes large amounts of text data during training and has the ability to understand and generate natural language. For example, large language models can include the Spark large model.

[0057] The large language model has a powerful ability to extract semantic features. By analyzing the sample image, the large language model can extract high-quality semantic features. Compared with extracting semantic features from images based on semantic branches, the semantic features output by the large language model are of better quality and can provide an effective reference for the optimization of semantic branches. Therefore, in the embodiment of the present invention, the semantic features output by the large language model are recorded as reference semantic features.

[0058] On this basis, the semantic branch calculates the distillation loss by comparing the difference between the semantic features extracted by itself and the reference semantic features. It is understandable that the greater the difference between the two, the greater the distillation loss. The design of this loss is intended to guide the semantic branch to gradually approach the expression effect of the reference features and improve its ability to model text semantics and contextual logic. By quantifying the difference, the distillation loss helps the model clarify the optimization goal, ensuring that the semantic branch is constantly adjusted during the training process to optimize the semantic expression effect of the extracted features.

[0059] Subsequently, the semantic branch is iterated based on the distillation loss, so that it can not only accurately capture the semantic information of a single character, but also establish reasonable contextual associations between characters. In this process, the semantic branch gradually has a higher semantic extraction capability and a deeper understanding of the language logic relationship of the text content. For example, for common homonyms or phrases, the model can reasonably judge their specific semantic meanings based on the context, avoiding semantic confusion or logical inconsistency in the recognition results.

[0060] By introducing reference features from large language models and optimizing distillation loss, the semantic branch significantly improves its ability to adapt to semantic information in complex scenes. Combined with the appearance feature extraction capabilities of the visual branch, the model can ultimately perform well in terms of recognition accuracy, semantic rationality, and contextual consistency.

[0061] Based on the above embodiment, determining the distillation loss based on the difference between the reference semantic feature and the semantic feature of the sample image includes: Extracting character semantic features of each character block of the sample image based on the semantic branch of the initial recognition model; The distillation loss is determined based on the difference between the reference semantic feature and the semantic feature of the sample image, and the difference between the semantic feature of each character in the reference semantic feature and the character semantic feature of each character block of the sample image.

[0062] Specifically, the semantic branch of the initial recognition model is first used to extract the character semantic features of each character block from the sample image. It can be understood that the character semantic features here reflect the semantic features of the characters in a single character block. The semantic features of the sample image extracted by the semantic branch reflect the global semantic features of the text composed of all characters contained in the sample image as a whole.

[0063] By calculating the difference between the reference semantic features and the semantic features of the sample image, the ability of the semantic branch to model the global semantic content can be evaluated. The reference semantic features are generated by a large language model and have rich semantic expressions and language logic. In addition, the difference comparison is not only performed on the global semantic features, but also includes a fine-grained comparison between the semantic features of each character in the reference semantic features and the semantic features of the corresponding characters in the sample image. This fine-grained comparison can evaluate the ability of the semantic branch to model local semantics. In this way, the model can optimize the semantic branch at both the global and local levels, so that it can not only accurately capture the semantic relationship between characters, but also ensure the logical consistency of the overall text semantics.

[0064] Through the design of the above distillation loss, the semantic branch has been significantly optimized both globally and locally. On the one hand, character-level alignment strengthens the semantic branch's ability to capture the semantic information of a single character, enabling it to better understand the independent meaning of the character; on the other hand, global semantic alignment ensures that the features extracted by the model have good language logic and contextual consistency, thereby improving the ability to understand the entire text. This multi-level optimization design enables the semantic branch to more comprehensively model the semantic information in the sample image, with higher expression accuracy and generalization ability, providing a strong guarantee for image and text recognition in complex scenarios.

[0065] Based on the above embodiment, in step 120, determining the visual constraint loss based on the difference between the visual features of a plurality of sample images having the same characters but different orderings includes: Determining the visual constraint loss based on differences between visual features of a plurality of sample images having consistent characters but different orderings, and differences between the visual features and class center features of the sample images; The class center feature of the sample image is determined based on the classification features of all characters contained in the sample image.

[0066] Specifically, the class center feature is the feature center calculated by counting the classification features of all characters contained in the sample image, that is, the average representation of each category of characters in the classification feature space. The class center feature has nothing to do with the character order. For example, the mean of the classification features of all characters contained in the sample image can be used as the class center feature of the sample image. For example, character images in a large number of images can be classified in advance, and visual features for classification corresponding to the characters extracted from the character images during the classification process can be obtained. The visual features of characters belonging to the same category can be aggregated and averaged to form stable and standardized classification features for the category to which the character belongs, which can be understood as the class center features of the category to which the character belongs. Here, a class of characters can be the same character in various shapes, sizes, and fonts. The classification features can be regarded as the core expression of the character category in the visual feature space, reflecting the common visual attributes of the character category under different fonts, backgrounds, lighting, and noise conditions. For example, for an image containing the characters "ABC", the visual branch can extract the visual features of each character. These features represent the appearance information of the characters, such as shape, size, and texture. On this basis, the visual features of characters of the same category (such as all "C" characters) are aggregated, and the classification features of the "C" characters are obtained by averaging. The classification features obtained in this way can be regarded as the ideal state of the visual features of the same character category under various changing conditions, providing the goal and reference standard for feature optimization.

[0067] On this basis, the class center feature of the sample image determined based on the classification features of all characters contained in the sample image can reflect the ideal state of visual features of all characters in the sample image without considering various changing conditions.

[0068] In addition to determining the visual constraint loss based on the difference between the visual features of multiple sample images with consistent characters but different orderings, the visual constraint loss can also be determined by applying the difference between the visual features of the sample image and the class center features.

[0069] Specifically, the model first generates sample images with the same character content but different character arrangement orders. For example, the character order in the original image is "ABC", and new images such as "BCA" or "CAB" are generated by random swapping. Although the character order of these images is different, the character content remains consistent. Ideally, their visual features should be highly similar because the visual branch should only focus on the appearance information of the characters rather than their arrangement order. By comparing the differences between these visual features, the model is able to constrain the visual branch to focus on the appearance features of the characters while ignoring the impact of the character order on feature expression. This constraint design breaks the interference of semantic level information on the visual branch, ensuring that the model can maintain the robustness and stability of the visual features of the characters in complex scenes.

[0070] In addition, in order to further optimize the feature extraction capability of the visual branch, the model introduces class center features as constraint criteria. During the training process, the model calculates the difference between the visual features extracted from the sample image and the corresponding class center features to ensure that the visual features of each character gradually move closer to the class center features. This process not only strengthens the aggregation of the features extracted by the visual branch, but also makes the features more consistent and generalizable in the face of appearance changes (such as fonts, noise, lighting, etc.). For example, even if the same character "C" presents different visual effects under different backgrounds or styles, the visual branch can still extract a stable expression that is close to the class center feature.

[0071] Through the above method, the design of visual constraint loss achieves dual optimization of the visual branch: on the one hand, by comparing the visual features of samples with the same characters but different orders, the visual branch's ability to focus on the appearance information of the characters is strengthened, eliminating the interference of semantic information; on the other hand, by guiding the visual features to move closer to the class center features, the stability and aggregation of the visual features are further enhanced. This design effectively improves the performance of the visual branch in complex scenes, enables the model to accurately extract the core appearance information of the characters, ensures the efficient extraction and expression of visual features in the image and text recognition task, and lays a solid foundation for the performance optimization of the overall model.

[0072] Based on the above embodiment, the model training method further includes before step 110: Based on the visual branch and semantic branch of the encoder, the pre-trained visual features and pre-trained semantic features of the pre-trained image are extracted respectively; The encoder is pre-trained based on the orthogonal constraint between the pre-trained visual features and the pre-trained semantic features, and the initial recognition model is determined based on the pre-trained encoder.

[0073] Specifically, the initial recognition model may be a pre-trained model.

[0074] During the pre-training process, the visual branch is responsible for extracting the visual features of characters from the pre-training images, including the feature representations of the character's shape, size, texture, and geometric structure. The visual features extracted from the pre-training images are referred to as pre-trained visual features. The semantic branch extracts the semantic features of the text in the pre-training images through context modeling to capture the semantic associations and logical structures between characters. The semantic features extracted from the pre-training images are referred to as pre-trained semantic features. In order to effectively separate the pre-trained visual features and the pre-trained semantic features, the model introduces orthogonal constraints during the pre-training process so that the two do not interfere with each other in the feature space and maintain independence.

[0075] The orthogonal constraint mathematically forces the inner product of the pre-trained visual features and the pre-trained semantic features to be close to zero, ensuring that the visual branch and the semantic branch focus on their respective learning goals during training. For example, when the character content is the same but the order changes, the visual branch can still stably extract the appearance features of the characters, while the semantic branch can ignore visual interference and understand the semantic logic of the text. In this way, the model forms a clear feature decoupling in the pre-training stage, effectively avoiding the confusion of visual information and semantic information.

[0076] The pre-training process continuously optimizes the encoder including the visual branch and the semantic branch through this orthogonal constraint, so that it gradually learns the decoupled high-quality feature representation. During the iterative training process, the visual branch and the semantic branch of the encoder are independently optimized according to their respective goals, and the independence of visual features and semantic features is gradually enhanced, and finally the effective separation of the two in the feature space is achieved. The visual branch strengthens the robust representation of the character appearance, while the semantic branch has stronger semantic modeling capabilities.

[0077] The pre-trained encoder is used to build the initial recognition model, providing a stable and high-quality feature extraction basis for subsequent image and text recognition tasks. Feature decoupling achieved through orthogonal constraints not only avoids mutual interference between visual information and semantic information, but also improves the generalization ability and adaptability of the model in complex scenes. Ultimately, the encoder optimized through pre-training ensures that the visual branch and the semantic branch perform their respective duties, significantly improving the overall performance of the model.

[0078] Based on the above embodiment, the model training method further includes: Predicting the order of characters in the pre-trained image based on pre-trained semantic features of the pre-trained image with disordered characters; Based on the character sequence, a semantic branch of the encoder is pre-trained.

[0079] Specifically, the core idea of ​​pre-training the semantic branch based on the pre-training images with disordered characters is to construct a task scenario that challenges the modeling ability of the semantic branch by disrupting the order of characters in the image. For the input pre-training image, the character detection model is first used to locate the position of each character in the image, and then the order of characters is randomly disrupted to generate pre-training samples with disordered character order, that is, pre-training images with disordered characters are obtained. For example, the original character order "HELLO" is disrupted into "LEHLO" or "OLEHL", resulting in an unnatural arrangement of the characters in spatial position. Although the appearance information of the characters in these pre-training images has not changed, the semantic logic has been disrupted. The semantic branch must learn to restore the correct arrangement of the text from the disordered character order through training, and then understand the true semantic relationship between characters.

[0080] In this process, the semantic branch is required to extract pre-trained semantic features of the input pre-trained image, and then these extracted pre-trained semantic features are used to predict the original order of characters. The design of this task forces the semantic branch to reconstruct the semantic order of the text through contextual logic and association information between characters without relying on visual appearance. Specifically, the model needs to start from the semantic relationship between local characters and gradually infer the reasonable order of the entire text. For example, by learning the contextual connection between "E", "L" and "H", the most reasonable character arrangement is inferred to be "HELLO".

[0081] In order to further optimize the modeling ability of the semantic branch, the semantic branch is pre-trained based on the predicted character order, so that it can continuously improve its ability to understand the relationship between character order and text semantics. This pre-training task can not only help the semantic branch better capture the sequential dependencies between characters, but also improve the model's ability to model the global semantics of the context. For example, when a character is partially occluded or blurred, the semantic branch can reasonably infer the location and content of the missing character based on the context information.

[0082] Through such character disorder pre-training, the semantic branch can gradually learn to ignore the chaotic interference of characters in spatial position, focus on understanding and reconstructing text semantics, and have strong global and local semantic modeling capabilities. This pre-training process effectively improves the model's adaptability to complex text scenes, especially when characters are disordered, some characters are missing, or the context logic is complex, the semantic branch can still generate accurate semantic representations. In addition, this task design can also enhance the semantic branch's ability to generalize text semantic structures, making it more stable and accurate in subsequent image and text recognition tasks.

[0083] Based on the above embodiment, the model training method further includes: The visual branch of the encoder is pre-trained based on the difference between pre-trained visual features of multiple pre-training images with consistent characters but different orderings.

[0084] Specifically, during the pre-training stage, the order of characters in the pre-training image can be disrupted to obtain pre-trained visual features of pre-training images with consistent characters but different ordering. On this basis, the parameters of the visual branch in the encoder can be iterated with the goal of making such pre-trained visual features as consistent as possible, thereby improving the robustness of the visual branch during the pre-training stage.

[0085] An embodiment of the present invention provides a method for image and text recognition. Figure 2 It is a schematic diagram of the flow chart of the image and text recognition method provided by the present invention, such as Figure 2 As shown, the method includes: Step 210: determine the image to be recognized.

[0086] Determining the image to be recognized means obtaining and preparing the target image for image and text recognition. The image can come from a scanned document, a photo, or a real-time captured screen. To ensure the accuracy of subsequent recognition, you can consider preprocessing the image, including denoising, brightness adjustment, rotation correction and cropping, extracting the text area and removing irrelevant background information. In addition, the image format and size can be standardized to meet the model input requirements, thereby providing a high-quality data input foundation for visual feature extraction and semantic modeling.

[0087] Step 220, performing image and text recognition on the image to be recognized based on the image and text recognition model; The image and text recognition model is obtained based on the model training method.

[0088] Specifically, the image and text recognition model first extracts the appearance information of the characters through the visual branch, including visual features such as shape, size, and texture, to ensure the accurate extraction of basic visual information. At the same time, the semantic branch extracts the global and local semantic information of the text by modeling the contextual relationship between characters, and understands the logical structure and actual meaning of the text. Since the model is enhanced at the semantic level during the training phase, even if the text in the image is occluded, blurred, or partially missing, the semantic branch can still make reasonable inferences about the missing or blurred characters based on the contextual information, ensuring that the recognition results conform to language habits and semantic logic.

[0089] After the visual features and semantic features are processed separately, they are combined through the fusion mechanism inside the image and text recognition model to generate the final recognition result. The fused features have both the character appearance information of the visual branch and the context understanding ability of the semantic branch, which can effectively improve the accuracy and language rationality of the recognition results and avoid erroneous output that does not conform to semantic logic. The image and text recognition result obtained in this way is the text content contained in the image to be recognized.

[0090] Based on the model training method, the image and text recognition model shows significant advantages in complex scenarios, especially when there are interference factors such as blur, occlusion or missing text, it can still perform accurate recognition, ensuring that the final result reaches a high level in terms of semantic rationality and visual accuracy.

[0091] The image and text recognition method provided by the present invention is based on the decoupling design of visual features and semantic features, so that the model pays attention to the appearance information and contextual semantic information of the characters at the same time. Through the enhancement at the semantic level, the method can combine the context to infer the character content when the image text is occluded, blurred or missing, and output the recognition result that conforms to the language habits and logic. In addition, the effective fusion of visual and semantic features improves the robustness and adaptability of the model, so that it still has high recognition accuracy and stability under interference conditions such as complex backgrounds.

[0092] The model training device and the image-text recognition device provided by the present invention are described below. The model training device and the image-text recognition device described below can correspond to the model training method and the image-text recognition method described above.

[0093] Figure 3 It is a structural schematic diagram of the model training device provided by the present invention, such as Figure 3 As shown, the device comprises: A feature extraction unit 310, used to extract visual features and semantic features of the sample image based on the visual branch and the semantic branch of the initial recognition model; A visual constraint unit 320, configured to determine a visual constraint loss based on a difference between visual features of a plurality of sample images having the same characters but in different order; A semantic constraint unit 330, configured to determine a semantic constraint loss based on a difference between semantic features of a plurality of sample images with consistent semantics and / or a difference between semantic features of a plurality of sample images with similar semantics; The parameter iteration unit 340 is used to perform parameter iteration on the visual branch based on the visual constraint loss, and to perform parameter iteration on the semantic branch based on the semantic constraint loss, and to determine a trained image and text recognition model based on the initial recognition model after the parameter iteration is completed.

[0094] The model training device provided by the present invention solves the problem that the recognition results do not conform to language habits or contextual logic due to the lack of semantic-level supervision by decoupling visual features and semantic features and performing constraint optimization on both. The visual branch's ability to extract visual features is strengthened through visual constraints, and the semantic branch's ability to understand global and local semantic features is enhanced through semantic constraints, effectively reducing the interference of visual and semantic information. The present invention significantly improves the semantic rationality and recognition accuracy of image and text recognition in complex scenarios, and has stronger applicability and generalization capabilities.

[0095] Based on any of the above embodiments, the semantic constraint unit is specifically used for: Extracting reference semantic features of the sample image based on a large language model; Determining a distillation loss based on a difference between the reference semantic feature and the semantic feature of the sample image; Based on the distillation loss, parameters of the semantic branch are iterated.

[0096] Extracting character semantic features of each character block of the sample image based on the semantic branch of the initial recognition model; The distillation loss is determined based on the difference between the reference semantic feature and the semantic feature of the sample image, and the difference between the semantic feature of each character in the reference semantic feature and the character semantic feature of each character block of the sample image.

[0097] Based on any of the above embodiments, the visual constraint unit is specifically used for: Determining the visual constraint loss based on differences between visual features of a plurality of sample images having consistent characters but different orderings, and differences between the visual features and class center features of the sample images; The class center feature of the sample image is determined based on the classification features of all characters contained in the sample image.

[0098] Based on any of the above embodiments, the feature extraction unit is specifically used for: Based on the visual branch and semantic branch of the encoder, the pre-trained visual features and pre-trained semantic features of the pre-trained image are extracted respectively; The encoder is pre-trained based on the orthogonal constraint between the pre-trained visual features and the pre-trained semantic features, and the initial recognition model is determined based on the pre-trained encoder.

[0099] Predicting the order of characters in the pre-trained image based on pre-trained semantic features of the pre-trained image with disordered characters; Based on the character sequence, a semantic branch of the encoder is pre-trained.

[0100] Figure 4 is a schematic diagram of the structure of the image and text recognition device provided by the present invention, such as Figure 4 As shown, the device comprises: An image determination unit 410 is used to determine an image to be identified; An image and text recognition unit 420 is used to perform image and text recognition on the image to be recognized based on an image and text recognition model; The image and text recognition model is obtained based on the model training method.

[0101] The image and text recognition device provided by the present invention is based on the decoupling design of visual features and semantic features, so that the model pays attention to the appearance information and contextual semantic information of the characters at the same time. Through the enhancement at the semantic level, the method can combine the context to infer the character content when the image text is occluded, blurred or missing, and output the recognition result that conforms to the language habits and logic. In addition, the effective fusion of visual and semantic features improves the robustness and adaptability of the model, so that it still has high recognition accuracy and stability under interference conditions such as complex backgrounds.

[0102] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the model training method or the image and text recognition method. The model training method includes: extracting the visual features and semantic features of the sample image based on the visual branch and the semantic branch of the initial recognition model; Determine a visual constraint loss based on the difference between visual features of multiple sample images with consistent characters but different orderings; Determine a semantic constraint loss based on differences between semantic features of a plurality of sample images with consistent semantics and / or differences between semantic features of a plurality of sample images with similar semantics; Based on the visual constraint loss, parameters of the visual branch are iterated, and based on the semantic constraint loss, parameters of the semantic branch are iterated, and based on the initial recognition model after the parameter iteration, a trained image and text recognition model is determined.

[0103] The image and text recognition methods include: An image determination unit, used for determining an image to be identified; An image and text recognition unit, used for performing image and text recognition on the image to be recognized based on an image and text recognition model; The image and text recognition model is obtained based on the model training method.

[0104] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0105] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the model training method or the image and text recognition method provided by the above methods. The model training method includes: extracting visual features and semantic features of the sample image based on the visual branch and semantic branch of the initial recognition model; Determine a visual constraint loss based on the difference between visual features of multiple sample images with consistent characters but different orderings; Determine a semantic constraint loss based on differences between semantic features of a plurality of sample images with consistent semantics and / or differences between semantic features of a plurality of sample images with similar semantics; Based on the visual constraint loss, parameters of the visual branch are iterated, and based on the semantic constraint loss, parameters of the semantic branch are iterated, and based on the initial recognition model after the parameter iteration, a trained image and text recognition model is determined.

[0106] The image and text recognition methods include: An image determination unit, used for determining an image to be identified; An image and text recognition unit, used for performing image and text recognition on the image to be recognized based on an image and text recognition model; The image and text recognition model is obtained based on the model training method.

[0107] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the model training method or the image and text recognition method provided by the above methods. The model training method includes: extracting visual features and semantic features of the sample image based on the visual branch and semantic branch of the initial recognition model; Determine a visual constraint loss based on the difference between visual features of multiple sample images with consistent characters but different orderings; Determine a semantic constraint loss based on differences between semantic features of a plurality of sample images with consistent semantics and / or differences between semantic features of a plurality of sample images with similar semantics; Based on the visual constraint loss, parameters of the visual branch are iterated, and based on the semantic constraint loss, parameters of the semantic branch are iterated, and based on the initial recognition model after the parameter iteration, a trained image and text recognition model is determined.

[0108] The image and text recognition methods include: An image determination unit, used for determining an image to be identified; An image and text recognition unit, used for performing image and text recognition on the image to be recognized based on an image and text recognition model; The image and text recognition model is obtained based on the model training method.

[0109] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0110] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method, characterized in that: include: Based on the visual branch and semantic branch of the initial recognition model, the visual features and semantic features of the sample image are extracted respectively; Determine a visual constraint loss based on the difference between visual features of multiple sample images with consistent characters but different orderings; Determine a semantic constraint loss based on differences between semantic features of a plurality of sample images with consistent semantics and / or differences between semantic features of a plurality of sample images with similar semantics; Based on the visual constraint loss, parameters of the visual branch are iterated, and based on the semantic constraint loss, parameters of the semantic branch are iterated, and based on the initial recognition model after the parameter iteration, a trained image and text recognition model is determined.

2. The model training method according to claim 1, characterized in that: Also includes: Extracting reference semantic features of the sample image based on a large language model; Determining a distillation loss based on a difference between the reference semantic feature and the semantic feature of the sample image; Based on the distillation loss, parameters of the semantic branch are iterated.

3. The model training method according to claim 2, characterized in that: The determining of the distillation loss based on the difference between the reference semantic feature and the semantic feature of the sample image includes: Extracting character semantic features of each character block of the sample image based on the semantic branch of the initial recognition model; The distillation loss is determined based on the difference between the reference semantic feature and the semantic feature of the sample image, and the difference between the semantic feature of each character in the reference semantic feature and the character semantic feature of each character block of the sample image.

4. The model training method according to claim 1, characterized in that: The determining of the visual constraint loss based on the difference between the visual features of a plurality of sample images having the same characters but different orderings includes: Determining the visual constraint loss based on differences between visual features of a plurality of sample images having consistent characters but different orderings, and differences between the visual features and class center features of the sample images; The class center feature of the sample image is determined based on the classification features of all characters contained in the sample image.

5. The model training method according to any one of claims 1 to 4, characterized in that: Also includes: Based on the visual branch and semantic branch of the encoder, the pre-trained visual features and pre-trained semantic features of the pre-trained image are extracted respectively; The encoder is pre-trained based on the orthogonal constraint between the pre-trained visual features and the pre-trained semantic features, and the initial recognition model is determined based on the pre-trained encoder.

6. The model training method according to claim 5, characterized in that: Also includes: Predicting the order of characters in the pre-trained image based on pre-trained semantic features of the pre-trained image with disordered characters; Based on the character sequence, a semantic branch of the encoder is pre-trained.

7. A method for image and text recognition, characterized in that: include: Determine an image to be identified; Based on the image-text recognition model, performing image-text recognition on the image to be recognized; The image and text recognition model is obtained based on the model training method according to any one of claims 1 to 6.

8. A model training device, characterized in that: include: A feature extraction unit, used for extracting visual features and semantic features of the sample image based on the visual branch and the semantic branch of the initial recognition model; A visual constraint unit, configured to determine a visual constraint loss based on differences between visual features of a plurality of sample images having the same characters but in different orderings; A semantic constraint unit, configured to determine a semantic constraint loss based on a difference between semantic features of a plurality of sample images with consistent semantics and / or a difference between semantic features of a plurality of sample images with similar semantics; A parameter iteration unit is used to perform parameter iteration on the visual branch based on the visual constraint loss, and to perform parameter iteration on the semantic branch based on the semantic constraint loss, and to determine a trained image and text recognition model based on an initial recognition model after parameter iteration.

9. A graphic recognition device, characterized in that: include: An image determination unit, used for determining an image to be identified; An image and text recognition unit, used for performing image and text recognition on the image to be recognized based on an image and text recognition model; The image and text recognition model is obtained based on the model training method according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the model training method according to any one of claims 1 to 6 or the image and text recognition method according to claim 7.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the model training method according to any one of claims 1 to 6 or the image and text recognition method according to claim 7.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method according to any one of claims 1 to 6 or the image and text recognition method according to claim 7.

Images