Text recognition method, electronic device and computer readable storage medium

Abstract

The application discloses a text recognition method, an electronic device and a computer readable storage medium. The method comprises: acquiring a plurality of visual tokens corresponding to a to-be-recognized image, and performing feature extraction on all visual tokens by using a visual encoder to obtain foreground features. The visual encoder comprises at least one group of visual feature networks and a mixed expert network. The mixed expert comprises a foreground expert and a background expert. The visual feature network extracts features of all visual tokens. The foreground expert selects part of the visual tokens according to the adaptability to the visual tokens and extracts features. The background expert extracts features of the visual tokens that are not selected by the foreground expert. The foreground features are determined based on the features extracted by the foreground expert. A text token corresponding to a prompt text matched with the to-be-recognized image is acquired. The text token is spliced with the foreground features to obtain to-be-decoded features. The to-be-decoded features are input into a large language model to obtain a text recognition result. The above scheme can save the computing power consumption and guarantee the accuracy of text recognition.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a text recognition method, electronic device, and computer-readable storage medium. Background Technology

[0002] Text recognition is a technique that extracts text from images to facilitate further analysis of the text content. It is commonly used to identify images that match a target scene. These matching images can be actual scenes captured by cameras or portable imaging devices, or downloaded or cropped images from the internet. Common recognition techniques are based on deep learning methods, including multi-stage and one-stage approaches. Multi-stage methods first crop the text and then recognize the characters, which is prone to cropping errors or introducing noise. One-stage methods use the network's perception capabilities to recognize text based on global features, but require significant computational resources. Therefore, current technologies struggle to balance computational overhead with text recognition accuracy. In light of this, how to conserve computational resources while ensuring text recognition accuracy has become a pressing issue. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a text recognition method, electronic device, and computer-readable storage medium that can save computing power while ensuring the accuracy of text recognition.

[0004] To address the aforementioned technical problems, this application provides a text recognition method, comprising: acquiring multiple visual tokens corresponding to an image to be recognized; extracting features from all the visual tokens using a visual encoder to obtain foreground features of the image to be recognized; wherein the visual encoder includes at least one set of visual feature networks and a hybrid expert network, the hybrid experts including foreground experts and background experts; the visual feature network extracts features from all the visual tokens; the foreground experts select some visual tokens according to their fit with the visual tokens and extract features; the background experts extract features from visual tokens not selected by the foreground experts; and the foreground features are determined based on the features extracted by the foreground experts; acquiring a text token corresponding to a prompt text matched by the image to be recognized; concatenating the text token with the foreground features to obtain features to be decoded; and inputting the features to be decoded into a large language model to obtain a text recognition result of the image to be recognized.

[0005] To address the aforementioned technical problems, a second aspect of this application provides an electronic device comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor invokes the program data to execute the method described in the first aspect.

[0006] To address the aforementioned technical problems, a third aspect of this application provides a computer-readable storage medium storing program data thereon, wherein the program data, when executed by a processor, implements the method described in the first aspect.

[0007] The beneficial effects of this application are as follows: Unlike existing technologies, this application obtains multiple visual tokens corresponding to an image to be recognized, inputs all visual tokens corresponding to the image to be recognized into a visual encoder, and extracts features from all visual tokens using the visual encoder to obtain the foreground features of the image to be recognized. Specifically, the visual encoder includes at least one set of visual feature networks and a hybrid expert network. The hybrid expert network includes foreground experts and background experts. The visual feature network extracts features from all visual tokens. The foreground experts select some tokens according to their fit with the visual tokens and extract the features of the corresponding visual tokens. The background experts obtain visual tokens not selected by the foreground experts and extract the features of the corresponding visual tokens. The foreground features extracted by the visual encoder are determined based on the features extracted by the foreground experts, thus avoiding text cropping in the visual encoder. Overall feature extraction is performed through the visual feature network, and the foreground experts in the hybrid expert network select visual tokens with high fit for feature extraction, ensuring the accuracy of the final foreground features. Feature extraction of unselected visual tokens by the background experts reduces the probability of omissions. Furthermore, each visual token only participates in the computation of some experts in the hybrid expert network, saving computational power. The text token corresponding to the prompt text matching the image to be recognized is obtained. The text token is concatenated with the foreground features to obtain the image-text fusion features to be decoded. The features to be decoded are input into the large language model to obtain the text recognition result corresponding to the image to be recognized output by the large language model. Thus, only the foreground features and the text token are concatenated and input into the large language model, which effectively reduces the amount of computation required by the large language model and saves computing power. At the same time, the large language model's capabilities are utilized to output high-precision text recognition results. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating one embodiment of the text recognition method of this application; Figure 2 This is a flowchart illustrating another embodiment of the text recognition method of this application; Figure 3 This is a schematic diagram illustrating an application scenario of one embodiment of the visual feature network and hybrid expert network of this application; Figure 4This is a schematic diagram illustrating an application scenario of one implementation method of hybrid expert network processing of visual tokens in this application; Figure 5 This is a schematic diagram illustrating an application scenario of one embodiment of the visual encoder and large language model used in this application for image processing. Figure 6 This is a schematic diagram of the structure of one embodiment of the electronic device of this application; Figure 7 This is a schematic diagram of one embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0009] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments, and different implementation methods can be adaptively combined. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper means two or more.

[0011] The text recognition method provided in this application is used to process images to recognize text in the images, and its corresponding execution subject is a processing unit capable of data processing.

[0012] Please see Figure 1 , Figure 1 This is a flowchart illustrating one embodiment of the text recognition method of this application. The method includes: S101: Obtain multiple visual tokens corresponding to the image to be recognized, and use a visual encoder to extract features from all visual tokens to obtain the foreground features of the image to be recognized; wherein, the visual encoder includes at least one set of visual feature networks and a hybrid expert network, the hybrid experts include foreground experts and background experts, the visual feature network extracts features from all visual tokens, the foreground experts select some visual tokens according to their fit with the visual tokens and extract features, the background experts extract features from visual tokens not selected by the foreground experts, and the foreground features are determined based on the features extracted by the foreground experts.

[0013] Specifically, multiple visual tokens corresponding to the image to be recognized are obtained. All visual tokens corresponding to the image to be recognized are input into the visual encoder. The visual encoder extracts features from all visual tokens to obtain the foreground features of the image to be recognized. Here, visual tokens are also known as visual tokens. The visual encoder inputs visual features in the form of tokens. After processing, the image to be recognized yields multiple visual tokens. This process is common knowledge in the field of visual Transformers.

[0014] It should be noted that the visual encoder includes at least one set of visual feature networks and a hybrid expert network. The hybrid expert network includes foreground experts and background experts. The visual feature network extracts features from all visual tokens. The foreground experts select some tokens according to their fit with the visual tokens and extract the corresponding visual token features. The background experts acquire visual tokens not selected by the foreground experts and extract the corresponding visual token features. The foreground features extracted by the visual encoder are determined based on the features extracted by the foreground experts. The visual encoder and the large language model together form the text recognition architecture, and both the visual encoder and the large language model have been pre-trained, thus enabling end-to-end output of the final text recognition result after inputting the image to be recognized.

[0015] Understandably, the image to be recognized is matched with the target scene. For different target scenes, the format of the image to be recognized can be matched with the specific scene. Before inputting the image into the visual encoder, the format of the image to be recognized can be unified, for example, unified into any format such as jpg or png.

[0016] In one embodiment, the visual encoder includes multiple interconnected visual feature networks and hybrid expert networks. After obtaining multiple visual tokens corresponding to the image to be recognized, all visual tokens corresponding to the image to be recognized are input into the visual encoder. The visual feature networks extract features from all visual tokens. The extracted features of the visual tokens are input into the hybrid expert network. The foreground expert in the hybrid expert network selects some visual tokens and extracts features according to the fit between the features and the input visual tokens. The features corresponding to the visual tokens not selected by the foreground expert are input to the background expert for feature extraction. The features extracted by the foreground expert and the features extracted by the background expert are fused and input into the next set of visual feature networks and hybrid expert networks, until the foreground features are determined based on the features extracted by the foreground expert in the final hybrid expert network.

[0017] In one embodiment, the visual encoder includes a set of visual feature networks and a hybrid expert network. After obtaining multiple visual tokens corresponding to the image to be recognized, all visual tokens corresponding to the image to be recognized are input into the visual encoder. The visual feature networks are used to extract features from all visual tokens. The extracted features of the visual tokens are input into the hybrid expert network. The foreground expert in the hybrid expert network selects some visual tokens and extracts features according to the fit between the features and the input visual tokens. The features corresponding to the visual tokens not selected by the foreground expert are input to the background expert for feature extraction. Based on the feature difference between the features extracted by the foreground expert and the background expert, the confidence level that the visual tokens processed by the background expert include foreground information is determined. The features of the visual tokens extracted by the foreground expert and the features of the visual tokens extracted by the background expert with a confidence level greater than the confidence level threshold are fused to obtain the foreground features.

[0018] Optionally, in different implementations, the visual encoder may also include network layers for feature alignment or dimension transformation, thereby adjusting the features extracted by different networks. This application does not impose specific limitations on this.

[0019] In some implementation scenarios, the visual feature network and the hybrid expert network together form a coding network combination. The visual encoder includes a combination of multiple sequentially connected coding networks. The visual feature network includes a visual transformer (ViT), and the hybrid expert network includes a hybrid expert with visual preferences (Vision Preference Mixture of Experts (VPMoE)).

[0020] In some implementation scenarios, a visual encoder includes a set of visual feature networks and a hybrid expert network, with a pre-network preceding the visual feature networks for feature dimension transformation and a post-network following the hybrid expert network for feature alignment.

[0021] It should be noted that the hybrid expert network includes multiple foreground experts and multiple background experts. The number of foreground experts and background experts is determined during the training phase. Each foreground expert selects a portion of the visual tokens and extracts features according to their fit with the visual tokens. Visual tokens not selected by any foreground expert are evenly distributed to at least some background experts.

[0022] Understandably, text cropping is avoided in the visual encoder. Overall feature extraction is performed through a visual feature network, and foreground experts in the hybrid expert network select visual tokens with high fit for feature extraction, ensuring the accuracy of the final foreground features. Background experts extract features from unselected visual tokens, reducing the probability of omissions. Furthermore, each visual token only participates in the computation of some experts in the hybrid expert network, saving computing power.

[0023] S102: Obtain the text token corresponding to the prompt text matched by the image to be recognized, concatenate the text token with the foreground features to obtain the features to be decoded, input the features to be decoded into the large language model, and obtain the text recognition result of the image to be recognized.

[0024] Specifically, the text token corresponding to the prompt text matching the image to be recognized is obtained. The text token is then concatenated with the foreground features to obtain the image-text fused features to be decoded. These features are then input into a large language model to obtain the text recognition result corresponding to the image to be recognized, output by the large language model. Here, the text token is also known as the text Token, and the prompt text is also processed into Token form. The conventional text processing procedures will not be elaborated upon in this application.

[0025] Understandably, simply concatenating the foreground features with the text token before inputting them into the large language model effectively reduces the computational load required for processing by the large language model, saving computing power, and utilizes the capabilities of the large language model to output highly accurate text recognition results.

[0026] In one embodiment, a text token corresponding to the prompt text matching the image to be recognized is obtained, the foreground features are aligned with the text token, and the text token is concatenated with the foreground features to obtain the features to be decoded. The features to be decoded are input into a large language model to obtain the text recognition result of the image to be recognized output by the large language model.

[0027] In one embodiment, a text token corresponding to the prompt text matched by the image to be recognized is obtained, and the text token is concatenated with each visual token matched by the foreground features to obtain the features to be decoded. The features to be decoded are input into a large language model to obtain the text recognition result of the image to be recognized output by the large language model.

[0028] It is understood that the features to be decoded include the features obtained after removing background information from the image to be recognized, and the text token corresponds to an instruction for text recognition of the input features to be decoded. The large language model can perform text recognition on the features to be decoded according to the instruction to obtain the text recognition result. The text recognition result is the text content recognized from the image to be recognized. The text recognition result can be used for direct output display, or it can be input into a subsequent network for parsing. The processing of the subsequent network includes, but is not limited to, text correction, keyword extraction, semantic recognition, and summary overview. This application does not impose specific limitations on these aspects.

[0029] It should be noted that the visual encoder and the large language model are trained together. The visual encoder and the large language model are trained in a supervised manner using sample images and their corresponding sample labels from the sample image set, thereby ensuring that the visual encoder and the large language model are adapted to the text recognition scenario required for the application.

[0030] In some implementation scenarios, sample labels include text content labels. During the training process, the large language model ultimately outputs the text prediction result. Based on the text prediction result and the text content label, a loss function is constructed, and then the visual encoder and the large language model are adjusted until the loss converges and the training is completed.

[0031] In some implementation scenarios, sample labels include text location labels and text content labels. During training, the visual encoder outputs the text prediction location when processing sample images, and the large language model finally outputs the text prediction result. Based on the text prediction location and text location label, as well as the text prediction result and text content label, a loss function is constructed, and then the visual encoder and the large language model are adjusted until the loss converges and training is completed.

[0032] In a specific implementation scenario, the target scene corresponding to the image to be recognized is an urban street, the foreground area corresponds to a billboard, and the background area is the area outside the foreground area, including but not limited to areas excluding text such as roads, vehicles, and buildings. The image to be recognized is input into a visual encoder and a large language model to obtain text recognition results, thereby determining the text content corresponding to the billboard in the foreground area, so as to determine the content of the advertisement and the amount of advertisements placed in the urban street. The application of the visual encoder and the large language model in the actual scenario, as well as the use of the text recognition results, can be customized, and this application does not impose specific restrictions on them.

[0033] In a specific implementation scenario, the target scene corresponding to the image to be recognized is an online book, and the online book is in image format. The foreground region corresponds to the text paragraphs in the image, and the background region corresponds to the blank areas. The image to be recognized is input into a visual encoder and a large language model to obtain text recognition results. The text content corresponding to the text paragraphs in the foreground region is determined. After accumulating at least some paragraph text content, the corresponding text content is input into a subsequent network to correct the text content. Based on the keywords and semantics corresponding to the corrected text, a summary text of the corresponding paragraph is generated. The application of the visual encoder and the large language model in the network scenario, as well as the use of the text recognition results, can be customized, and this application does not impose specific restrictions on them.

[0034] The above scheme obtains multiple visual tokens corresponding to the image to be recognized, inputs all visual tokens corresponding to the image to be recognized into a visual encoder, and extracts features from all visual tokens to obtain the foreground features of the image to be recognized. The visual encoder includes at least one set of visual feature networks and a hybrid expert network. The hybrid expert network includes foreground experts and background experts. The visual feature network extracts features from all visual tokens. The foreground experts select some tokens according to their fit with the visual tokens and extract the features of the corresponding visual tokens. The background experts obtain the visual tokens not selected by the foreground experts and extract the features of the corresponding visual tokens. The foreground features extracted by the visual encoder are determined based on the features extracted by the foreground experts, thus avoiding text clipping in the visual encoder. Overall feature extraction is performed through the visual feature network, and the foreground experts in the hybrid expert network select visual tokens with high fit for feature extraction, ensuring the accuracy of the final foreground features. Feature extraction of unselected visual tokens by the background experts reduces the probability of omissions. Furthermore, each visual token only participates in the computation of some experts in the hybrid expert network, saving computational power. The text token corresponding to the prompt text matching the image to be recognized is obtained. The text token is concatenated with the foreground features to obtain the image-text fusion features to be decoded. The features to be decoded are input into the large language model to obtain the text recognition result corresponding to the image to be recognized output by the large language model. Thus, only the foreground features and the text token are concatenated and input into the large language model, which effectively reduces the amount of computation required by the large language model and saves computing power. At the same time, the large language model's capabilities are utilized to output high-precision text recognition results.

[0035] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the text recognition method of this application, the method comprising: S201: Obtain multiple visual tokens corresponding to the image to be recognized, and use a visual encoder to extract features from all visual tokens to obtain the foreground features of the image to be recognized; wherein, the visual encoder includes at least one set of visual feature networks and a hybrid expert network, the hybrid experts include foreground experts and background experts, the visual feature network extracts features from all visual tokens, the foreground experts select some visual tokens according to their fit with the visual tokens and extract features, the background experts extract features from visual tokens not selected by the foreground experts, and the foreground features are determined based on the features extracted by the foreground experts.

[0036] Specifically, multiple visual tokens corresponding to the image to be recognized are obtained, all visual tokens corresponding to the image to be recognized are input into the visual encoder, and the foreground features of the image to be recognized are obtained by the visual encoder through feature extraction of all visual tokens.

[0037] It is understandable that the image to be recognized is processed to obtain multiple visual tokens, and in specific implementation scenarios, a processing method that is adapted to the scenario can be adopted.

[0038] For ease of understanding, this application provides a specific processing method, but it does not limit the specific processing method. First, the image's width and height are adjusted to an integer multiple of the slice's width and height, approximately equal to the original image's aspect ratio. The resized image is then input into the visual embedding layer of a CNN structure, where features from every 2×2 slices are fused into a single visual token. All visual tokens output by the visual embedding layer are denoted as... Where B represents the batch size, N represents the number of tokens per image, and C represents the number of channels for the tokens.

[0039] It should be noted that the visual encoder includes multiple sets of visual feature networks and hybrid expert networks connected in sequence. The features extracted by the visual feature networks are input to the hybrid expert networks connected to them. The features extracted by the foreground experts and background experts in the hybrid expert networks are fused and then input into the visual feature networks until the foreground features are determined based on the features extracted by the foreground experts in the final hybrid expert network.

[0040] Specifically, when all visual tokens are input into the visual feature network, that is, after the features of all previously processed visual tokens are input into the visual feature network, the visual feature network further extracts the overall features of all visual tokens, so that the global visual features of each image learn from each other, and then inputs the extracted features into the hybrid expert network connected to it.

[0041] Furthermore, foreground experts select a subset of visual tokens for feature extraction to uncover foreground-related features, including characters. Background experts process visual tokens not selected by the foreground experts to reduce the probability of missing small texts and misidentifying background elements with similar character shapes as text. The features extracted by the foreground and background experts are fused and input into the next set of visual feature networks and hybrid expert networks connected to them. This process, through multiple sets of networks, improves the accuracy and precision of distinguishing foreground and background during multiple processing steps. Finally, after processing by the last set of visual feature networks and hybrid expert networks, foreground features are determined based on the features extracted by the foreground experts in the final hybrid expert network, thus improving the reliability of foreground features.

[0042] Please see Figure 3 , Figure 3This is a schematic diagram illustrating an application scenario of one embodiment of the visual feature network and hybrid expert network of this application. The visual encoder VPMoE-ViT, with visual preference hybrid expert layers, consists of L encoding blocks. Ordinary visual feature networks (ViT Blocks) and visual preference hybrid expert networks (VPMoE Blocks) are arranged alternately, with the last block being VPMoEBlock. The visual encoder includes multiple sets of visual feature networks and hybrid expert networks. The visual feature networks include a normalization layer (Norm), a multi-head attention layer, and a feed-forward neural network (FFN). The features of all visual tokens in the image are first passed through the ordinary visual feature network (ViT Block) to enable mutual learning of the global visual features of each image. The learned features are then input into the visual preference hybrid expert network (VPMoE Block), which replaces the FFN layer in the ordinary ViT Block with the visual preference hybrid expert layer (VPMoE).

[0043] It should be noted that the hybrid experts include multiple foreground experts and multiple background experts. The foreground features of the image to be recognized are obtained by extracting features from all visual tokens using a visual encoder. This includes performing the following steps for each set of visual feature networks and hybrid expert networks: inputting all visual tokens into the visual feature network to obtain the features of each visual token; determining a subset of visual tokens selected by each foreground expert and extracting their corresponding features based on the fit between the features of each foreground expert and each visual token; and randomly assigning visual tokens not selected by any foreground expert to at least a subset of background experts and extracting their corresponding features.

[0044] Specifically, all visual tokens are input into the visual feature network, and features are extracted from all visual tokens to make the features input into the hybrid expert network global. Based on the fit between the features of each foreground expert and each visual token, the fit ranking between each foreground expert and all visual tokens is determined. From the corresponding ranking, a subset of visual tokens with high fit are selected for the foreground experts and their corresponding features are extracted. Visual tokens not selected by any foreground expert are randomly assigned to at least some background experts.

[0045] Understandably, the visual tokens selected by each foreground expert are usually similar and can be grouped into a single category. This ensures the accuracy of feature extraction for similar visual tokens. Furthermore, the same visual token may be selected by different foreground experts, leading to more precise feature extraction. Each foreground expert selecting only a subset of visual tokens saves computational resources. Additionally, tokens not selected by any foreground expert are assigned to at least a portion of the background features, allowing the same visual token to be processed by different background experts to extract corresponding features, thus ensuring the accuracy of feature extraction.

[0046] In one embodiment, based on the fit between the features of each foreground expert and each visual token, a subset of visual tokens selected by each foreground expert are determined and their corresponding features are extracted. Visual tokens not selected by any foreground expert are randomly assigned to at least a subset of background experts and their corresponding features are extracted. This includes: determining an affinity score between each foreground expert and each visual token based on the fit between the features of each foreground expert and each visual token; selecting a first number of visual tokens with the highest affinity scores for each foreground expert and extracting their corresponding features; and randomly assigning visual tokens not selected by any foreground expert to at least a second number of background experts and extracting their corresponding features. The second number is determined by the product of the total number of background experts and a preset ratio.

[0047] Specifically, an adaptation matrix is ​​generated based on the fit between the features of each foreground expert and each visual token. This matrix is ​​then used to calculate the affinity score between each foreground expert and each visual token. The visual tokens corresponding to each foreground expert are sorted according to their affinity scores, and the top-ranked visual tokens are assigned to the corresponding foreground experts. Features of the corresponding visual tokens are then extracted using these foreground experts. The first number is determined during the training phase and is related to the proportion of the foreground region in the image acquired in the text recognition scenario; a higher proportion results in a larger first number.

[0048] Furthermore, visual tokens not selected by any foreground expert are obtained, and these visual tokens are randomly and evenly distributed among at least a second number of background experts. The features of the visual tokens are then extracted using the respective background experts. The second number is determined by the product of the total number of background experts and a preset ratio, where the preset ratio is greater than 0 and less than or equal to 1. This second number controls the computational power consumption of the background experts, enabling the rational allocation of computational power.

[0049] It should be noted that the features extracted by foreground and background experts in the hybrid expert network are fused based on the following steps: based on the fit between each visual token selected by the foreground expert and the corresponding foreground expert, the fusion weight corresponding to the visual token is determined. Using the fusion weight corresponding to each visual token, the features extracted by the foreground experts are fused to obtain foreground features; the features extracted by the background experts are fused according to the same weight to obtain background features; the foreground features and background features in the hybrid expert network are fused to obtain the fused features of the input visual feature network.

[0050] Specifically, based on the fit between each visual token selected by the foreground expert and the corresponding foreground expert, a fusion weight matching the fit is set for the corresponding visual token. The features of the visual tokens extracted from the foreground features are fused using the fusion weight to ensure the accuracy of the fusion. The foreground features are matched with the features corresponding to all visual tokens selected by the foreground experts.

[0051] Furthermore, the features extracted by different background experts for the same visual token are assigned the same weights, and the features of the visual token extracted by the background experts are fused by averaging to obtain background features. Then, the foreground features and background features in the hybrid expert network are fused to obtain fused features, so that the fused features can be completely input into the subsequent visual feature network and hybrid expert network.

[0052] It should be noted that for the features extracted by the foreground experts in the final hybrid expert network, the same method is used to obtain the foreground features, and the features extracted by the background experts are discarded so that only the final foreground features are retained.

[0053] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an application scenario of one implementation method of hybrid expert network processing of visual tokens in this application. VPMoE contains a total of There are several expert networks, each consisting of an FFN network with the same structure but different parameters, where the foreground expert FE is used to process foreground features. One, background expert BE who processes background features has There are one, that is, there are The hybrid expert network consists of a routing network (Router) and a random assignment network (Random). In VPMoE, the Router first calculates the fit between each visual token and each foreground expert, resulting in a fit matrix. First, The final dimension undergoes Softmax exponential normalization to obtain the affinity score of each visual token for the foreground expert. This involves calculating the affinity score for each foreground expert, and each expert selects the top k tokens by affinity score for learning, where k is the expert capacity, i.e., the initial number of experts. B represents the batch size, N represents the number of tokens per image, and a smaller k indicates fewer visual tokens selected by foreground experts, resulting in higher sparsity of visual features input into the large language model. The final value of k determines the foreground region in a specific scene. Visual tokens not selected by any expert are distributed to at least [number missing] people through a random allocation network following a uniform distribution. Among the background experts (such as) Figure 4 (As shown by the dashed arrow in the middle), to ensure that each background expert receives sufficient visual tokens and that the visual tokens can be processed by multiple background experts.

[0054] Furthermore, regarding the fitness matrix The penultimate dimension is subjected to Softmax exponential normalization to obtain the affinity score of each visual token to all foreground experts. That is, a summation score is calculated for each corresponding visual token. Each visual token input to a foreground expert uses its affinity score with that expert as the aggregation weight. The outputs of the visual tokens after passing through different foreground experts are weighted and averaged to obtain the final foreground feature after VPMoE processing. For visual tokens input to background experts, the average of the corresponding background expert's output is used as the background feature after VPMoE processing.

[0055] Understandably, the visual tokens output by the final hybrid expert network in the visual encoder are recorded and divided into two categories. The visual tokens obtained after foreground expert processing are arranged in index order from front to back, denoted as... The visual tokens obtained through background expert processing are arranged in index order from front to back, and denoted as... .in This indicates the number of visual tokens associated with the foreground region in each image. This indicates the number of visual tokens associated with the background area in each image.

[0056] S202: Obtain the prompt text matching the image to be recognized, input the prompt text into the text processing network provided by the large language model, and obtain the text token corresponding to the prompt text.

[0057] Specifically, the prompt text matching the image to be identified is obtained, the prompt text is processed using the text processing network provided by the large language model to obtain the text token corresponding to the prompt text, and the features corresponding to the prompt text are determined.

[0058] Optionally, the text token is obtained by processing the prompt text through a text segmenter and a text embedding layer, which are set in the large language model.

[0059] S203: Align and concatenate the foreground features with the text token to obtain the features to be decoded. Input the features to be decoded into the large language model to obtain the text recognition result of the image to be recognized.

[0060] Specifically, the foreground features are aligned with the text token, and the aligned foreground features are concatenated with the text token to obtain the features to be decoded. The scale of the features to be decoded is kept consistent. The features to be decoded are then input into the large language model to obtain the text recognition results output by the large language model.

[0061] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an application scenario of one embodiment of the visual encoder and large language model used in this application for image processing. A linear layer is also included between the visual encoder and the large language model. This linear layer aligns foreground features with text tokens. For example... Figure 5 As shown, the application inference stage requires concatenating the aligned foreground features with the text token to obtain the features to be decoded. The process illustrated by the dashed line is only included in the training stage; that is, the text token is concatenated with the background features only during the training stage.

[0062] It should be noted that the visual encoder and the large language model are trained based on the following steps: Multiple visual tokens are obtained for each sample image in the sample image set; all visual tokens corresponding to the sample images are input into the visual encoder to obtain the foreground prediction features and background prediction features processed by the visual encoder, as well as the text prediction positions matched by the foreground prediction features; wherein, the sample images are labeled with text position labels and text content labels; using the training prompt text and text content labels matched by each sample image, the foreground features to be decoded corresponding to the foreground prediction features and the background features to be decoded corresponding to the background prediction features are constructed; the foreground features to be decoded and the background features to be decoded are input into the large language model to obtain the text prediction results of the sample images; based on the text prediction results and text content labels corresponding to all sample images, as well as the text prediction positions and text position labels, the visual encoder and the large language model are adjusted.

[0063] Specifically, a sample image set is acquired, comprising multiple sample images. Each sample image is labeled with a text location and a text content. Multiple visual tokens corresponding to each sample image are obtained. All visual tokens corresponding to the sample image are input into a visual encoder, which then performs image processing according to the method described in the above embodiment to obtain foreground prediction features and background prediction features. Furthermore, during the training phase, the visual encoder also outputs the text prediction location matched by the foreground prediction features. For clarity, the features obtained during the training phase are referred to as prediction features.

[0064] Furthermore, training prompt text matching the sample images is obtained. Using the training prompt text and text content labels matching each sample image, the foreground features to be decoded corresponding to the foreground prediction features and the background features to be decoded corresponding to the background prediction features are constructed. Thus, the training prompt text and text content labels are fused into the foreground prediction features to obtain the foreground features to be decoded, so that the large language model can comprehensively learn text recognition when processing the foreground features to be decoded. Similarly, the training prompt text and text content labels are fused into the background prediction features to obtain the background features to be decoded, so that the large language model can learn how to avoid recognizing the content of the background region when processing the background features to be decoded.

[0065] Understandably, the foreground features and background features to be decoded are input into a large language model to obtain text prediction results for sample images. A loss function is constructed based on the deviation between the text prediction results and text content labels for all sample images, as well as the deviation between the text prediction position and the text position label, to obtain the training loss. The visual encoder and the large language model are then adjusted based on the training loss. The training loss can be calculated based on the cross-entropy function, and this application does not impose specific restrictions on it.

[0066] It should be noted that, using the training prompt text and text content label matched for each sample image, the foreground features to be decoded corresponding to the foreground prediction features and the background features to be decoded corresponding to the background prediction features are constructed. This includes: obtaining the predicted text token corresponding to the training prompt text matched for each sample image and the target text token corresponding to the text content label; concatenating the foreground prediction features with the predicted text token and the target text token to obtain the foreground features to be decoded; and concatenating the background prediction features with the predicted text token and the text token corresponding to the preset symbol to obtain the background features to be decoded.

[0067] Specifically, the training prompt text and text content label of each sample image are processed separately to obtain the predicted text tokens (Prompt Tokens) of the training prompt text and the target text tokens (Target Tokens) of the text content label, where the text content label is the actual text content on the sample image.

[0068] Furthermore, the foreground prediction features are concatenated with the prediction text tokens (Prompt Tokens) and the target text tokens (Target Tokens) to obtain the foreground features to be decoded. To enable large language models to learn features within the foreground region more efficiently, the predicted background features are concatenated with the predicted text tokens (Prompt Tokens) and the text tokens corresponding to preset symbols to obtain the background features to be decoded. In this context, the preset symbol is usually a stop symbol, so that large language models can learn features in the background region more efficiently.

[0069] Optionally, to facilitate batch training, the number of tokens in the foreground prediction features and background prediction features can be uniformly filled to the same amount.

[0070] It is understandable that the trained visual encoder and large language model can implement the text recognition method in any of the above embodiments when applied, and the trained visual encoder and large language model can be applied to perform text recognition on images collected in real-world or network scenarios.

[0071] It should be noted that, in terms of the number of parameters and computational efficiency of the visual encoder, the addition of the multi-expert network has led to an increase in the overall number of parameters, and the ability of the visual encoder to extract rich features has been expanded. Since each visual token only participates in the calculation of some experts when passing through the hybrid expert network, the increase in the total number of model parameters avoids a significant increase in computational load. As a result, it saves a lot of computing power compared with the conventional one-stage method, while ensuring the accuracy of text recognition.

[0072] Please see Figure 6 , Figure 6 This is a schematic diagram of an embodiment of the electronic device of this application. The electronic device 30 includes a memory 301 and a processor 302 coupled to each other. The memory 301 stores program data (not shown in the figure). The processor 302 calls the program data to implement the method in any of the above embodiments. For the description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0073] Please see Figure 7 , Figure 7 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 40 stores program data 400. When the program data 400 is executed by a processor, it implements the method in any of the above embodiments. For a detailed description of the relevant content, please refer to the detailed description of the above method embodiments, which will not be repeated here.

[0074] It should be noted that the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0075] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0076] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] The above description is merely an embodiment of this application and does not limit the scope of protection of this application. Any equivalent structural or procedural transformations made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of protection of this application.

Claims

1. A text recognition method, characterized in that, include: Multiple visual tokens corresponding to an image to be identified are obtained, and a visual encoder is used to extract features from all the visual tokens to obtain the foreground features of the image to be identified. The visual encoder includes at least one set of visual feature networks and a hybrid expert network. The hybrid experts include foreground experts and background experts. The visual feature network extracts features from all the visual tokens. The foreground experts select some visual tokens according to their fit with the visual tokens and extract features. The background experts extract features from the visual tokens not selected by the foreground experts. The foreground features are determined based on the features extracted by the foreground experts. Obtain the text token corresponding to the prompt text matched by the image to be recognized, concatenate the text token with the foreground features to obtain the features to be decoded, input the features to be decoded into a large language model, and obtain the text recognition result of the image to be recognized.

2. The text recognition method according to claim 1, characterized in that, The visual encoder includes multiple sets of visual feature networks and a hybrid expert network connected in sequence. The features extracted by the visual feature networks are input to the hybrid expert network connected to them. The features extracted by the foreground expert and the background expert in the hybrid expert network are fused and then input into the visual feature network until the foreground feature is determined based on the features extracted by the foreground expert in the last hybrid expert network.

3. The text recognition method according to claim 2, characterized in that, The hybrid expert includes multiple foreground experts and multiple background experts; The step of extracting foreground features of the image to be identified by using a visual encoder to extract features from all the visual tokens includes performing the following steps for each set of visual feature networks and the hybrid expert network: Input all the visual tokens into the visual feature network to obtain the features of each visual token; Based on the fit between the features of each foreground expert and each visual token, a subset of visual tokens selected by each foreground expert are determined and their corresponding features are extracted. Visual tokens not selected by any foreground expert are randomly assigned to at least a subset of background experts and their corresponding features are extracted.

4. The text recognition method according to claim 3, characterized in that, The process of determining a subset of visual tokens selected by each foreground expert and extracting their corresponding features based on the fit between the features of each foreground expert and each visual token, and randomly assigning visual tokens not selected by any foreground expert to at least a subset of background experts and extracting their corresponding features, includes: Based on the fit between the features of each foreground expert and each visual token, an affinity score is determined between each foreground expert and each visual token. For each foreground expert, the first number of visual tokens ranked first by the affinity score are selected and their corresponding features are extracted. Visual tokens not selected by any of the foreground experts are randomly assigned to at least a second number of background experts and their corresponding features are extracted; wherein the second number is determined by the product of the number of all background experts and a preset ratio.

5. The text recognition method according to claim 2, characterized in that, The features extracted by the foreground expert and the background expert in the hybrid expert network are fused based on the following steps: Based on the fit between each visual token selected by the foreground expert and the corresponding foreground expert, the fusion weight corresponding to the visual token is determined. Using the fusion weight corresponding to each visual token, the features extracted by the foreground expert are fused to obtain foreground features. The features extracted by the background experts are fused together with the same weight to obtain the background features; The foreground and background features within the hybrid expert network are fused to obtain the fused features input to the visual feature network.

6. The text recognition method according to claim 1, characterized in that, The step of obtaining the text token corresponding to the prompt text matched by the image to be identified, and concatenating the text token with the foreground features to obtain the features to be decoded includes: Obtain the prompt text matched by the image to be identified, input the prompt text into the text processing network provided by the large language model, and obtain the text token corresponding to the prompt text; The foreground features are aligned with the text token and then concatenated to obtain the features to be decoded.

7. The text recognition method according to claim 1, characterized in that, The visual encoder and the large language model are trained based on the following steps: Multiple visual tokens corresponding to each sample image in the sample image set are obtained. All visual tokens corresponding to the sample image are input into the visual encoder to obtain the foreground prediction features and background prediction features processed by the visual encoder, as well as the text prediction position matched by the foreground prediction features; wherein, the sample image is provided with text position label and text content label; Using the training prompt text matched with each sample image and the text content label, construct the foreground feature to be decoded corresponding to the foreground prediction feature and the background feature to be decoded corresponding to the background prediction feature; The foreground features and background features to be decoded are input into the large language model to obtain the text prediction results of the sample images. Based on the text prediction results and text content labels corresponding to all the sample images, as well as the text prediction positions and text position labels, the visual encoder and the large language model are adjusted.

8. The text recognition method according to claim 7, characterized in that, The step of constructing the foreground features to be decoded corresponding to the foreground prediction features and the background features to be decoded corresponding to the background prediction features by using the training prompt text matched by each of the sample images and the text content tags includes: Obtain the predicted text token corresponding to the training prompt text matched by each sample image and the target text token corresponding to the text content label; The foreground prediction feature is concatenated with the predicted text token and the target text token to obtain the foreground feature to be decoded. The background prediction feature is concatenated with the predicted text token and the text token corresponding to the preset symbol to obtain the background feature to be decoded.

9. An electronic device, characterized in that, include: A memory and a processor are coupled to each other, wherein the memory stores program data, and the processor invokes the program data to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing program data thereon, characterized in that, When the program data is executed by the processor, it implements the method as described in any one of claims 1-8.

Images