Scene text recognition method and device based on multi-modal large language model

Abstract

The application relates to a scene text recognition method and device based on a multimodal large language model, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining a first token set formed by tokens whose correlation with an input image meets a set requirement in a text generation process, and obtaining a second token set formed by tokens whose semantic stability meets a set requirement in the text generation process; forming a semantic anchor point paragraph according to tokens continuously appearing in an intersection of the first token set and the second token set, and recording sampling temperatures of positions of all tokens in the text generation process; and adjusting the sampling temperatures of the positions of all tokens in the text generation process based on the semantic anchor point paragraph, and re-performing text generation according to the adjusted sampling temperatures, so that accurate recognition of visual text and inhibition of semantic illusion are realized in image text recognition, and the accuracy and robustness of scene text recognition are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a scene text recognition method and device based on a multi-modal large language model. BACKGROUND

[0002] Scene Text Recognition (STR) refers to a technology of detecting and recognizing text content from images or videos in natural scenes. Unlike traditional document text recognition (such as scanned or printed materials), scene text often faces challenges such as complex background, diverse fonts, light changes, distortion, and the like, and is a core research direction in the fields of computer vision and OCR (Optical Character Recognition).

[0003] In the prior art, traditional scene text processing methods usually adopt a multi-stage processing flow, including text detection, recognition, and language modeling, and the like. Although such methods perform well in specific environments, due to decoupling and error accumulation between processing modules, their generalization ability in complex and open environments is limited.

[0004] With the development of large multi-modal models, such as LLaVA, Qwen2-VL, Mini-Monkey, and the like, researchers have attempted to introduce them into optical character recognition (OCR) related tasks, and have made significant progress in document question answering, graphical interface analysis, and general OCR frameworks. However, the current large multi-modal models still have deficiencies in the ability of "scene text recognition and understanding", especially in the aspect of accurately recognizing image text.

[0005] Research shows that existing large multi-modal models tend to rely on semantic priors for reasoning, that is, when recognizing image text, they do not strictly base the recognition judgment on visual content, but output results based on "guessing" common semantic knowledge (i.e. the semantic illusion phenomenon). In addition, the accuracy of the model significantly decreases when processing text samples that are semantically irrelevant but visually real, indicating that its actual ability to process visual features is limited.

[0006] In view of the above problems, although existing methods have improved MLLM (Multimodal Large Language Model) in visual-linguistic tasks through resolution enhancement, Token encoding optimization, and new task design (such as UReader, Ocean-OCR, GOT-OCR), most models still perform poorly when processing semantically disturbed text (i.e. text with grammatical structure but no actual semantics), with a significant decrease in recognition accuracy, exposing the problem of excessive dependence on semantics and insufficient adaptation to pure visual features.

[0007] And, the existing improvement method depends on training and structural changes: for example, additional modules, specific pre-training methods, resulting in complex actual deployment and high migration cost. For example, the current method for improving text recognition performance usually needs to add branch networks, resolution enhancement modules, visual cropping mechanisms, etc. on the original model, or rely on additional training processes and specific data set support, resulting in a cumbersome model deployment and migration process and poor universality.

[0008] Therefore, how to achieve accurate recognition of visual text and semantic illusion suppression in image text recognition without changing the original MLLM architecture is a key challenge in current technology. SUMMARY

[0009] In view of the above analysis, the embodiments of the present application aim to provide a scene text recognition method and device based on a multi-modal large language model, to solve the problem of improving the recognition accuracy of a multi-modal large language model in scene text recognition without changing the MLLM architecture.

[0010] On the one hand, the embodiments of the present application provide a scene text recognition method based on a multi-modal large language model, comprising obtaining a first token set constituted by tokens whose relevance to input images in a text generation process meets a set requirement, and obtaining a second token set constituted by tokens whose semantic stability in the text generation process meets a set requirement, forming a semantic anchor point paragraph according to tokens that continuously appear in the intersection of the first token set and the second token set, and recording the sampling temperature of each token position in the text generation process; adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph, and re-performing text generation according to the adjusted sampling temperature.

[0011] The beneficial effects of the above technical solution are as follows: without changing the original MLLM architecture, accurate recognition of visual text and semantic illusion suppression in image text recognition are achieved, and the accuracy and robustness of scene text recognition are improved.

[0012] Based on the further improvement of the above method, the first token set composed of tokens meeting the set requirements in the text generation process in relation to the input image comprises: for the token in the text generation process, extracting the cross-attention vector of the token in the cross-attention mechanism; wherein the cross-attention vector reflects the attention degree of the token to each region of the input image; calculating the cross-attention autocorrelation matrix between the cross-attention vectors of the tokens in the set time window, obtaining the related tokens whose cosine similarity of the cross-attention autocorrelation matrix is greater than the preset similarity threshold, and marking the remaining tokens after the related tokens are screened out from the token sequence output by the multi-modal large language model as the first token set.

[0013] The beneficial effect of the above further improvement scheme is that the removal of noise tokens with no high correlation with the image and attention drift in the construction process of the semantic anchor point paragraph is realized, and the accuracy of the semantic anchor point paragraph acquisition is improved.

[0014] Based on the further improvement of the above method, the second token set composed of tokens meeting the set requirements in the text generation process in relation to the input image comprises: for the token in the text generation process, extracting the cross-attention vector of the token in the cross-attention mechanism; wherein the cross-attention vector reflects the attention degree of the token to each region of the input image; calculating the cross-attention autocorrelation matrix between the cross-attention vectors of the tokens in the set time window, obtaining the related tokens whose cosine similarity of the cross-attention autocorrelation matrix is greater than the preset similarity threshold, and marking the remaining tokens after the related tokens are screened out from the token sequence output by the multi-modal large language model as the first token set.

[0015] The beneficial effect of the above further improvement scheme is that the removal of noise tokens with no high correlation with the image and attention drift in the construction process of the semantic anchor point paragraph is realized, and the accuracy of the semantic anchor point paragraph acquisition is improved.

[0016] Based on the further improvement of the above method, the second token set composed of tokens meeting the set requirements in the text generation process in relation to the input image comprises: for the token in the text generation process, extracting the cross-attention vector of the token in the cross-attention mechanism; wherein the cross-attention vector reflects the attention degree of the token to each region of the input image; calculating the cross-attention autocorrelation matrix between the cross-attention vectors of the tokens in the set time window, obtaining the related tokens whose cosine similarity of the cross-attention autocorrelation matrix is greater than the preset similarity threshold, and marking the remaining tokens after the related tokens are screened out from the token sequence output by the multi-modal large language model as the first token set.

[0017] The beneficial effect of the above further improvement scheme is that the accuracy of the second token set result is improved.

[0018] Based on the further improvement of the above method, the adjustment of the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph comprises: maintaining the sampling temperature of the token position or slightly adjusting the sampling temperature by a preset smaller amplitude for the token position at the semantic anchor point paragraph; and reducing the sampling temperature by a preset larger amplitude for the token position at the non-semantic anchor point paragraph.

[0019] The beneficial effect of the above further improvement scheme is to ensure natural and smooth text and rich semantics and hallucination suppression in text generation.

[0020] Based on the further improvement of the above method, before the text generation is performed again according to the adjusted sampling temperature, the method further comprises: obtaining a low-confidence region of the input image whose cross-modal matching confidence is lower than a preset confidence threshold; and reducing the sampling temperature of the token position corresponding to the low-confidence region and / or edge region of the input image.

[0021] The beneficial effect of the above further improvement scheme is to reduce the probability of generating a risky token, making the generated result more reliable.

[0022] Based on the further improvement of the above method, after the text generation is performed again according to the adjusted sampling temperature, the method further comprises: performing a text-image semantic matching evaluation on the generated text to obtain a matching score; in response to the matching score being lower than a preset score threshold, determining a token region not covered by the semantic anchor point paragraph as a low-confidence region and determining a token region covered by the semantic anchor point paragraph as a high-confidence region; further reducing the sampling temperature of the token position of the low-confidence region, rolling back the decoding state of the low-confidence region, and performing text generation on the low-confidence region again; merging the text generated by the low-confidence region again and the text of the high-confidence region generated in the last text generation process as an integrated generated text, and going to the processing step of obtaining the first token set whose relevance to the input image meets the set requirement; and in response to the matching score being greater than the preset threshold, taking the text generated according to the adjusted sampling temperature as the final output text.

[0023] The beneficial effect of the above further improvement scheme is to further improve the accuracy of text generation.

[0024] Based on the further improvement of the above method, the generated text is subjected to image-text semantic matching evaluation to obtain a matching score, including: vector encoding the generated text to obtain a first vector; vector encoding the input image to obtain a second vector; and obtaining the matching score by calculating the correlation between the first vector and the second vector.

[0025] The beneficial effect of the above further improvement is that the accuracy of the matching score is improved.

[0026] In another aspect, the embodiments of the present application provide a scene text recognition device based on a multi-modal large language model, which comprises: a semantic anchor point recognition module, configured to: obtain a first token set constituted by tokens whose relevance to an input image meets a set requirement in a text generation process, and obtain a second token set constituted by tokens whose semantic stability meets a set requirement in the text generation process, form a semantic anchor point paragraph according to tokens that continuously appear in the intersection of the first token set and the second token set, and record the sampling temperature of each token position in the text generation process; a temperature control generation module, configured to: adjust the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph, and re-generate text according to the adjusted sampling temperature.

[0027] The beneficial effect of the above technical solution is as follows: without changing the original MLLM architecture, accurate identification of visual text and semantic hallucination suppression are realized in image text recognition, and the accuracy and robustness of scene text recognition are improved.

[0028] Based on the further improvement of the above device, it further comprises a local rollback generation module, configured to: perform image-text semantic matching evaluation on the generated text to obtain a matching score; in response to the matching score being lower than a preset score threshold, determine a token region that is not covered by the semantic anchor point paragraph as a low-confidence region, and determine a token region covered by the semantic anchor point paragraph as a high-confidence region; further reduce the sampling temperature of the token position in the low-confidence region, rollback the decoding state of the low-confidence region, and re-generate text for the low-confidence region; merge the text re-generated by the low-confidence region and the text of the high-confidence region generated in the last text generation process as integrated generated text, and go to the processing step of obtaining the first token set constituted by tokens whose relevance to an input image meets a set requirement in a text generation process by the semantic anchor point recognition module; in response to the matching score being greater than a preset threshold, the text re-generated according to the adjusted sampling temperature is taken as the final output text.

[0029] The beneficial effect of adopting the further improved scheme is that the accuracy of text generation is further improved.

[0030] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the scene text recognition method based on the multi-modal large language model according to any one of the above when executing the computer program.

[0031] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the scene text recognition method based on the multi-modal large language model according to any one of the above.

[0032] The application further provides a computer program product, which includes a computer program, and the computer program is executable on a processor to implement the scene text recognition method based on the multi-modal large language model according to any one of the above.

[0033] In the application, the above technical solutions can be combined with each other to achieve more preferred combination solutions. Other features and advantages of the application will be described in the subsequent description, and some advantages will become apparent from the description, or will be understood by implementing the application. The purposes and other advantages of the application can be achieved and obtained from the contents specifically indicated in the description and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings are included to provide a further understanding of the application and are incorporated herein and constitute a part of the application. The drawings illustrate embodiments of the application and, together with the description, serve to explain the principles of the application. In the drawings:

[0035] Figure 1 is a flowchart of the scene text recognition method based on the multi-modal large language model provided by the application;

[0036] Figure 2 is one of structural schematic diagrams of the scene text recognition device based on the multi-modal large language model provided by the application;

[0037] Figure 3 is the other of structural schematic diagrams of the scene text recognition device based on the multi-modal large language model provided by the application;

[0038] Figure 4 is a flowchart of the scene text recognition of the scene text recognition device based on the multi-modal large language model provided by the application;

[0039] Figure 5 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0040] Preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this application, and are used to explain the principles of the present application, and are not intended to limit the scope of the present application.

[0041] The present application relates to the field of computer vision and natural language processing, and is particularly applied to scene text recognition and understanding, belongs to the multi-modal information processing technology in the field of artificial intelligence, and is widely applied to automatic driving, product intelligent analysis, auxiliary reading equipment and other practical scenes.

[0042] Traditional or current mainstream multi-modal large language models (MLLMs) tend to speculate results based on semantic common sense rather than accurately identify text regions actually existing in images when processing scene text. Especially when facing non-semantic disturbance text (such as spelling errors, word creation, and random codes), the model is prone to output incorrect results that "look reasonable" but do not match the visual content.

[0043] It is found through research that the current multi-modal model internal attention mechanism does not pay sufficient attention to the text region in the image, and different Transformer layers respond differently to the text region or even experience attention drift, resulting in the loss of text information during transmission and further causing recognition errors. Different Transformer layers exhibit different attention distribution and recognition tendencies when processing visual text, and some intermediate layers are closer to the true label than the final output layer. This indicates that attention drift may be one of the key reasons for semantic illusion. Although the attention mechanism is an important part of the multi-modal model, most current methods treat it as an implicit behavior, lack systematic regulation and guidance methods, and some visual features cannot be effectively reflected in the upper language output.

[0044] To solve this problem, the present application provides a scene text recognition method based on a multi-modal large language model, which can realize attention-guided semantic illusion suppression, has the advantages of training freedom, high compatibility, etc., and can significantly improve the performance and robustness of the MLLM in complex scene text recognition tasks.

[0045] Figure 1 is a flowchart of the scene text recognition method based on the multi-modal large language model provided by the present application. As shown in Figure 1 , the method comprises:

[0046] In step S1, a first token set formed by tokens meeting a set requirement in relevance to the input image in a text generation process is obtained, and a second token set formed by tokens meeting a set requirement in semantic stability in the text generation process is obtained, a semantic anchor point paragraph is formed according to tokens appearing continuously in the second token set, and a sampling temperature of each token position in the text generation process is recorded.

[0047] The multi-modal large language model supports multi-modal input when performing scene text recognition, such as supporting joint input of multiple data types such as text, image, audio, and video (for example, inputting a picture + question, and outputting a text answer). The scene text recognition method provided by the application is mainly applied to image text recognition, and the image can be a single picture or a frame of image in a video stream. When performing scene text recognition, the image or the image + prompt text can be used as input. The multi-modal large language model converts the encoding information of the multi-modal input (such as image, text, audio, etc.) into human understandable output such as text answer through the decoder.

[0048] In this step, the token paragraph that is semantically stable and highly relevant to the input image is detected and marked to form a semantic anchor point paragraph. The first token set formed by tokens meeting a set requirement in relevance to the input image in a text generation process is obtained, and the second token set formed by tokens meeting a set requirement in semantic stability in the text generation process is obtained, and the tokens in the intersection of the first token set and the second token set meet the set requirements of semantic stability and relevance to the input image at the same time. The first token set formed by tokens meeting a set requirement in relevance to the input image in a text generation process and the second token set formed by tokens meeting a set requirement in semantic stability in the text generation process can be dynamically obtained with the text generation process.

[0049] The semantic anchor point paragraph is formed according to the tokens appearing continuously in the intersection of the first token set and the second token set. The semantic anchor point paragraph can be one or more, and the tokens in the semantic anchor point paragraph are the tokens appearing continuously in the intersection of the first token set and the second token set. The sampling temperature of each token position in the text generation process is recorded. The tokens in the semantic anchor point paragraph are highly relevant to the input image, which reduces the dependence of the model on language semantics from the mechanism. In addition, attention drift is an important reason for semantic instability. By setting the tokens in the semantic anchor point paragraph to meet the semantic stability, the attention drift phenomenon can be effectively overcome.

[0050] Step S2, adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor paragraph, and re-performing text generation according to the adjusted sampling temperature.

[0051] In large language models such as GPT or multimodal large models (MLLM), temperature is an important parameter for controlling the "randomness" or "diversity" of output content. It mainly plays a role in the process of sampling generation, that is, when the model predicts the next word or token, it does not always choose the one with the highest probability, but "draws" one from multiple candidate words according to a set of probability distributions. At this time, the temperature determines whether the model is more "cautious" or more "free" to play.

[0052] In this step, the sampling temperature of each token position in the text generation process is finely adjusted based on the semantic anchor paragraph, realizing the differential control of the diversity and certainty of different regions, and re-performing text generation according to the adjusted sampling temperature.

[0053] The scene text recognition method based on the multimodal large language model provided by the application, by acquiring a first token set constituted by tokens whose relevance to the input image in the text generation process meets the set requirement, and acquiring a second token set constituted by tokens whose semantic stability meets the set requirement, forming a semantic anchor paragraph according to the tokens that continuously appear in the intersection of the first token set and the second token set, and recording the sampling temperature of each token position in the text generation process, adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor paragraph, and re-performing text generation according to the adjusted sampling temperature, realizing accurate recognition of visual text and inhibition of semantic hallucination in image text recognition without changing the original MLLM architecture, and improving the accuracy and robustness of scene text recognition.

[0054] According to the present invention, a scene text recognition method based on a multimodal large language model is provided. The step of obtaining a first token set consisting of tokens whose relevance to the input image during the text generation process meets the set requirements includes: for tokens in the text generation process, extracting the cross-attention vector of the token in the image-text cross-attention mechanism; wherein, the cross-attention vector reflects the degree of attention of the token to each region of the input image; calculating the cross-attention autocorrelation matrix between the cross-attention vectors of the tokens within a set time window, obtaining the relevant tokens whose cosine similarity of the cross-attention autocorrelation matrix is ​​greater than a preset similarity threshold, and marking the remaining tokens after filtering out the relevant tokens in the token sequence output by the multimodal large language model as the first token set.

[0055] When obtaining the first set of tokens that meet the set requirements for relevance between the tokens and the input image during the text generation process, firstly, for the tokens in the text generation process, extract the cross-attention vector of the token in the image-text cross-attention mechanism. The cross-attention vector reflects the degree of attention that the token pays to each region of the input image.

[0056] The cross-attention vector is obtained from the Cross-Attention layer in the Transformer, and is represented as:

[0057] ;

[0058] in, Indicates the t-th i The cross-attention vector of each token. This indicates that the t-th word generated by the language-side decoder... i A query vector for each token. This represents the set of key vectors for the input image. This indicates the dimension of the key / query. , where n represents the number of input image tokens.

[0059] The vector reflects the degree of attention paid to each region of the image by the current generated token; the higher the value, the greater the influence of that region on token generation.

[0060] Within a defined time window (e.g., the most recent k tokens, where k can be between 32 and 64), calculate the autocorrelation matrix C∈R between the cross-attention vectors of these tokens. k×k To assess whether attention is focused and stable over time:

[0061] ;

[0062] wherein, denotes the cross-attention self-correlation matrix between the t i th token and the t j th token.

[0063] If the attention vectors between multiple tokens have a high cosine similarity (such as greater than 0.85), it indicates that these tokens pay long-term attention to the same area in the image, such as a corner of the image, and cannot pay attention to the entire image, and the image does not have a high correlation, and attention drift occurs. Therefore, according to the calculation result of the cross-attention self-correlation matrix, noise tokens with attention drift can be screened out, that is, tokens with a high cosine similarity between the attention vectors of multiple tokens.

[0064] Therefore, according to the calculation result of the cross-attention vectors of the tokens, the cross-attention self-correlation matrix between the cross-attention vectors of the tokens within a set time window is calculated, the relevant tokens with a cosine similarity greater than a preset similarity threshold in the cross-attention self-correlation matrix are obtained, and the remaining tokens after the relevant tokens are screened out in the token sequence output by the multi-modal large language model are marked as a first token set as semantic anchor point candidates.

[0065] The method for recognizing scene text based on a multi-modal large language model provided by the present application extracts the cross-attention vectors of tokens in a cross-attention mechanism between text and images for tokens in a text generation process, the cross-attention vectors reflect the attention degree of the tokens to each region of the input image, the cross-attention self-correlation matrix between the cross-attention vectors of the tokens within a set time window is calculated, the relevant tokens with a cosine similarity greater than a preset similarity threshold in the cross-attention self-correlation matrix are obtained, and the remaining tokens after the relevant tokens are screened out in the token sequence output by the multi-modal large language model are marked as a first token set. In the construction process of the semantic anchor point paragraph, the removal of noise tokens with a low correlation with the image and attention drift is realized, and the accuracy of obtaining the semantic anchor point paragraph is improved.

[0066] According to the scene text recognition method based on the multi-modal large language model provided by the application, the second token set composed of tokens with semantic stability meeting the set requirement in the text generation process is obtained, comprising: for the token in the text generation process, extracting the prediction distribution of the token in each layer of the Transformer, calculating the consistency index of the prediction distribution according to the prediction distribution of the token in each layer of the Transformer, and marking the token with the consistency index of the prediction distribution meeting the requirement as the second token set.

[0067] The consistency of the prediction distribution of the token in each layer of the Transformer is high, which indicates that the semantic output is stable and is not disturbed by the context. The token with low consistency of the prediction distribution in each layer of the Transformer indicates that the token is greatly affected by the semantic illusion in the high layer, and is a token with unstable semantics.

[0068] The embodiment obtains the second token set as a semantic anchor candidate. When the second token set composed of tokens with semantic stability meeting the set requirement in the text generation process is obtained, for the token in the text generation process, the prediction distribution of the token in each layer of the Transformer is extracted, the consistency index of the prediction distribution is calculated according to the prediction distribution of the token in each layer of the Transformer, and the token with the consistency index of the prediction distribution meeting the requirement is marked as the second token set.

[0069] The scene text recognition method based on the multi-modal large language model provided by the application realizes the removal of tokens with unstable semantics in the semantic anchor paragraph construction process by extracting the prediction distribution of the token in each layer of the Transformer for the token in the text generation process, calculating the consistency index of the prediction distribution according to the prediction distribution of the token in each layer of the Transformer, and marking the token with the consistency index of the prediction distribution meeting the requirement as the second token set, and further improves the accuracy of the semantic anchor paragraph acquisition.

[0070] According to the scene text recognition method based on the multi-modal large language model provided by the application, the consistency index of the prediction distribution is calculated according to the prediction distribution of the token in each layer of the Transformer, comprising: calculating the KL divergence of the prediction distribution of the token in each adjacent layer of the Transformer, and determining the token with the average value of the KL divergence less than the preset threshold as the token with the consistency index of the prediction distribution meeting the requirement.

[0071] The embodiment evaluates the consistency of the prediction distribution of the token at each layer of the Transformer by calculating the KL divergence. Specifically, by calculating the KL divergence of the prediction distribution of each two adjacent layers, the token whose average value of the KL divergence is less than a preset threshold (such as 0.1) is determined as the token whose consistency index of the prediction distribution meets the requirement. ;

[0072] wherein, denotes the prediction distribution of the token at the layer and the layer of the Transformer, denotes the prediction distribution of the token at the layer of the Transformer, denotes the prediction distribution of the token at the layer of the Transformer, denotes the vector of the jth dimension of the prediction distribution of the token at the layer of the Transformer, denotes the vector of the jth dimension of the prediction distribution of the token at the layer of the Transformer.

[0073] The scene text recognition method based on the multi-modal large language model provided by the application determines the token whose consistency index of the prediction distribution meets the requirement by calculating the KL divergence of the prediction distribution of each two adjacent layers of the Transformer, and the average value of the KL divergence is less than a preset threshold, thereby improving the accuracy of the second token set result.

[0074] According to the scene text recognition method based on the multi-modal large language model provided by the application, the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph is adjusted, which comprises: for the token position at the semantic anchor point paragraph, maintaining the sampling temperature of the token position or slightly adjusting the sampling temperature by a preset smaller amplitude; for the token position at the non-semantic anchor point paragraph, reducing the sampling temperature by a preset larger amplitude.

[0075] Wherein, slightly adjusting the sampling temperature by a preset smaller amplitude can be adjusting the sampling temperature to 85%-95% of the original sampling temperature, and reducing the sampling temperature by a preset larger amplitude can be adjusting the sampling temperature to 40%-60% of the original sampling temperature.

[0076] When text generation is first performed, a temperature mapping table TokenTempMap needs to be initialized in advance, which is an array [T1, T2,...,Tn] of a structure body, used to record the sampling temperature of each token position. The initialization value is Tinit by default, for example, 0.5.

[0077] When adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph, for the token position at the semantic anchor point paragraph, the sampling temperature of the token position is maintained or slightly adjusted by a preset smaller amplitude to ensure natural and fluent text and rich semantics, such as setting the sampling temperature of the token position at the semantic anchor point paragraph to T anchor ≈0.5; for the token position at the non-semantic anchor point paragraph, the sampling temperature is reduced by a preset larger amplitude, such as reducing the temperature to T non <0.5, for example, 0.3 or 0.2, to reduce sampling diversity, enhance generation certainty, and suppress hallucination generation.

[0078] The scene text recognition method based on a multi-modal large language model provided by the application maintains the sampling temperature of the token position at the semantic anchor point paragraph or slightly adjusts the sampling temperature by a preset smaller amplitude, reduces the sampling temperature of the token position at the non-semantic anchor point paragraph by a preset larger amplitude, and realizes the guarantee of natural and fluent text and rich semantics and hallucination suppression in text generation.

[0079] According to the scene text recognition method based on a multi-modal large language model provided by the application, before the text generation is performed again according to the adjusted sampling temperature, the method further includes: obtaining a low-confidence region of the input image whose cross-modal matching confidence is lower than a preset confidence threshold; and reducing the sampling temperature of the token position corresponding to the low-confidence region and / or the edge region of the input image.

[0080] For the edge region and / or the region with lower cross-modal matching confidence (such as CLIPScore lower than the threshold) of the image, a temperature disturbance factor δT is superimposed:

[0081] ;

[0082] Wherein, represents the sampling temperature before adjustment.

[0083] This additional cooling strategy can reduce the probability of risk tokens being sampled and generated, making the generation more credible.

[0084] The low-confidence region can be determined according to the relationship between the cross-modal matching confidence of the image and a preset confidence threshold.

[0085] TokenTempMap is dynamically used during token-by-token sampling. In each token generation stage, the sampling uses the temperature corresponding to the current token:

[0086] ;

[0087] wherein, the generation probability of token , is the current logits vector, from TokenTempMap; the lower the temperature, the sharper the distribution, and the more certain the result.

[0088] During the generation process, if a semantic anchor point is newly identified or the attention region drifts, TokenTempMap will be dynamically updated, so that the sampling strategy evolves with the generation process, ensuring that local accuracy and global fluency are considered.

[0089] The scene text recognition method based on the multi-modal large language model provided by the application reduces the sampling temperature of the token position corresponding to the low-confidence region and / or the edge region of the input image by obtaining the low-confidence region of the input image whose cross-modal matching confidence is lower than a preset confidence threshold, so as to reduce the probability of risk tokens being sampled and generated, and make the generated result more reliable.

[0090] According to the scene text recognition method based on the multi-modal large language model provided by the application, after the text generation is performed again according to the adjusted sampling temperature, the method further comprises: performing a picture-text semantic matching evaluation on the generated text to obtain a matching score; in response to the matching score being lower than a preset score threshold, determining a token region that is not covered by the semantic anchor point paragraph as a low-confidence region and determining a token region that is covered by the semantic anchor point paragraph as a high-confidence region; further reducing the sampling temperature of the token position of the low-confidence region, rolling back the decoding state of the low-confidence region, and performing text generation on the low-confidence region again; merging the text generated by the low-confidence region again and the text of the high-confidence region generated in the last text generation process as integrated generated text, and going to the processing step of obtaining the first token set constituted by the tokens whose relevance to the input image meets the set requirement.

[0091] The image semantic reverse matching and anchor point coverage verification are performed on the generated content to determine the semantic reliability of the generated text, and local rollback and regeneration are triggered if necessary to ensure the semantic fidelity of the generated content. After the text generation is completed, the text is subjected to image-text semantic matching evaluation, low-confidence areas are identified and local rollback rewriting is performed, forming a closed-loop enhancement mechanism.

[0092] The generated text is subjected to image-text semantic matching evaluation to obtain a matching score. If the matching score is lower than a preset score threshold, the token region not covered by the semantic anchor point paragraph is determined as a low-confidence region, and the token region covered by the semantic anchor point paragraph is determined as a high-confidence region. The sampling temperature of the token position in the low-confidence region is further reduced (for example, the sampling temperature is reduced to 0.3 times the previous sampling temperature), the decoding state of the low-confidence region is rolled back to the starting point (or offset point) of the region, and the decoder is used again to generate text for the low-confidence region based on the reduced sampling temperature, ensuring that the content in this segment is more closely related to the image semantic content on the basis of language reasonableness.

[0093] The text generated by the low-confidence region is merged with the text generated by the high-confidence region in the previous text generation process to form the integrated generated text, and the processing step of obtaining the first token set constituted by the tokens that meet the set requirements in terms of relevance to the input image in the text generation process is performed. That is, based on the full-region text obtained through local rollback, further execution is performed:

[0094] The first token set constituted by the tokens that meet the set requirements in terms of relevance to the input image in the text generation process, and the second token set constituted by the tokens that meet the set requirements in terms of semantic stability in the text generation process are obtained. The tokens that continuously appear in the intersection of the first token set and the second token set form a semantic anchor point paragraph, and the sampling temperature of each token position in the text generation process is recorded. Based on the semantic anchor point paragraph, the sampling temperature of each token position in the text generation process is adjusted, and text generation is performed again based on the adjusted sampling temperature.

[0095] Further, the generated text is subjected to image-text semantic matching evaluation to obtain a matching score; in response to the matching score being lower than a preset score threshold, a token region not covered by the semantic anchor point paragraph is determined as a low-confidence region, and a token region covered by the semantic anchor point paragraph is determined as a high-confidence region; the sampling temperature of the token position in the low-confidence region is further reduced, the decoding state of the low-confidence region is rolled back, and the low-confidence region is subjected to text generation again; wherein a local token rollback mechanism is started for the low-confidence region, and a more stringent temperature and sampling strategy is used for re-generation, so as to ensure that the text is faithful to the semantic reduction of the image; the text re-generated by the low-confidence region is merged with the text of the high-confidence region generated in the last text generation process, as the integrated generated text, and is transferred to the processing step of obtaining the first token set constituted by the tokens whose relevance to the input image meets the set requirement, that is, as long as the image-text semantic matching evaluation of the generated text does not meet the requirement, the above process is repeatedly executed to form a complete quality control closed loop. By dynamically adjusting the temperature, the local details and global coherence of the overall generation process are ensured, and the accuracy and stability of the generated text are continuously improved.

[0096] If the matching score obtained by the image-text semantic matching evaluation of the generated text is greater than the preset threshold, the generated text is taken as the final output text.

[0097] The scene text recognition method based on the multi-modal large language model provided by the application, by performing image-text semantic matching evaluation on the generated text, obtaining a matching score, in response to the matching score being lower than a preset score threshold, determining a token region not covered by the semantic anchor point paragraph as a low-confidence region, determining a token region covered by the semantic anchor point paragraph as a high-confidence region, further reducing the sampling temperature of the token position in the low-confidence region, rolling back the decoding state of the low-confidence region, and re-generating text for the low-confidence region, merging the text re-generated by the low-confidence region with the text of the high-confidence region generated in the last text generation process, as the integrated generated text, and transferring to the processing step of obtaining the first token set constituted by the tokens whose relevance to the input image meets the set requirement, in response to the matching score being greater than the preset threshold, the text re-generated according to the adjusted sampling temperature is taken as the final output text, which further improves the accuracy of text generation.

[0098] According to the scene text recognition method based on the multi-modal large language model provided by the application, the image-text semantic matching evaluation of the generated text is performed to obtain a matching score, which includes: the generated text is subjected to vector encoding to obtain a first vector; the input image is subjected to vector encoding to obtain a second vector; and the matching score is obtained by calculating the correlation between the first vector and the second vector.

[0099] In the text generated by the semantic matching evaluation of the picture, the matching score is obtained, the generated text is vector encoded to obtain the first vector, the input image is vector encoded to obtain the second vector, and the correlation between the first vector and the second vector is calculated to obtain the matching score. The matching score is used to evaluate the semantic coupling degree of the generated text and the input image.

[0100] The matching score is represented as:

[0101] ;

[0102] Wherein, The matching score is represented as: The first vector is represented as: The second vector is represented as: If the matching score is lower than the threshold (such as 0.25), it is considered as semantic disengagement. CLIP or BLIP type image-text encoder can be used to obtain the matching score.

[0103] The scene text recognition method based on the multi-modal large language model provided by the application improves the accuracy of the matching score by vector encoding the generated text to obtain the first vector, vector encoding the input image to obtain the second vector, and calculating the correlation between the first vector and the second vector.

[0104] The application realizes explicit modeling, sampling control and posterior correction of visual regions in multi-modal generation tasks through the synergistic effect of semantic anchor point paragraph recognition, temperature regulation generation and local rollback generation. This technical solution effectively solves the key problems of semantic bias dependence, visual attention drift and complex deployment in existing methods in image-text recognition and understanding without changing the original model structure, and brings the following beneficial effects:

[0105] I. Explicitly model the image visual region to effectively suppress semantic prior bias

[0106] The application dynamically detects the semantic anchor points in the generation process, i.e. those token fragments with stable high response on cross-modal attention and consistent prediction distribution in multi-layer Transformer. These anchors reduce the model's dependence on language semantics from a mechanism.

[0107] II. Relieve attention drift and enhance visual region response consistency

[0108] By calculating the KL divergence across multiple layers, token regions with consistent multi-layer responses can be selected to effectively filter attention drift or unstable attention. In combination with the local rollback generation mechanism, bidirectional constraints from forward generation to posterior matching are realized.

[0109] III. Provide fine-grained sampling regulation mechanism to improve stability and precision of generation

[0110] Construct TokenTempMap to realize token-level dynamic temperature regulation mechanism. The sampling temperature of non-anchor point area is reduced to effectively suppress the generation of "hallucination"; the temperature of anchor point area is fine-tuned to ensure semantic coherence and language fluency.

[0111] IV. Construct dynamic feedback loop to significantly improve visual fidelity

[0112] Introduce reverse semantic matching and local token rollback mechanism to correct "low matching area" after text generation, ensure that the final text truly represents the information contained in the image, and realize the whole-link optimization of "forward anchoring + process regulation + post-correction".

[0113] In summary, the present application effectively addresses the problems of heavy semantic dependence, weak visual modeling, and high structural complexity in current multi-modal image-text understanding tasks through semantic anchor point identification, dynamic temperature control sampling, and post-checking mechanism, with the advantages of high precision, strong robustness, and friendly deployment, and is suitable for various application scenarios such as image-text recognition, image-text question answering, and OCR assisted understanding.

[0114] The multi-modal large language model-based scene text recognition device provided by the present application is described below, and the multi-modal large language model-based scene text recognition device described below can be correspondingly referred to the multi-modal large language model-based scene text recognition method described above.

[0115] Figure 2 is one of the structural diagrams of the multi-modal large language model-based scene text recognition device provided by the present application. As shown in Figure 2 The device includes a semantic anchor point identification module 10 and a temperature regulation generation module 20, wherein: the semantic anchor point identification module 10 is used to: obtain a first token set constituted by tokens whose relevance to an input image meets a set requirement in a text generation process, and obtain a second token set constituted by tokens whose semantic stability meets a set requirement in the text generation process, form a semantic anchor point paragraph according to tokens that continuously appear in the intersection of the first token set and the second token set, and record the sampling temperature of each token position in the text generation process; the temperature regulation generation module 20 is used to: adjust the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph, and regenerate the text according to the adjusted sampling temperature.

[0116] The application provides a scene text recognition device based on a multi-modal large language model, which comprises a first token set obtaining module, a second token set obtaining module, a semantic anchor point paragraph forming module, a sampling temperature adjusting module and a text generation module.

[0117] Figure 3 Fig. 2 is a structural schematic diagram of the scene text recognition device based on the multi-modal large language model according to the application. Figure 3 As shown in Fig. 2, the device further comprises a local rollback generation module 30, which is configured to perform graphic-text semantic matching evaluation on the generated text to obtain a matching score, determine a token region not covered by the semantic anchor point paragraph as a low-confidence region and a token region covered by the semantic anchor point paragraph as a high-confidence region in response to the matching score being lower than a preset score threshold, further reduce the sampling temperature of the token position in the low-confidence region, rollback the decoding state of the low-confidence region, and re-perform text generation on the low-confidence region, merge the text generated by the low-confidence region and the text generated by the high-confidence region in the last text generation process as integrated generated text, and transfer to the processing step of the first token set obtaining module, and output the text generated by re-performing text generation according to the adjusted sampling temperature as the final output text in response to the matching score being greater than the preset threshold.

[0118] The application provides a scene text recognition device based on a multi-modal large language model, which evaluates the generated text through image-text semantic matching to obtain a matching score, determines a token region not covered by a semantic anchor point paragraph as a low-confidence region and a token region covered by the semantic anchor point paragraph as a high-confidence region in response to the matching score being lower than a preset score threshold, further reduces the sampling temperature of the token position in the low-confidence region, reverts the decoding state of the low-confidence region, re-generates text for the low-confidence region, merges the re-generated text for the low-confidence region with the text of the high-confidence region generated in the last text generation process, takes the merged text as the integrated generated text, and goes to the processing step of obtaining the first token set constituted by the tokens whose relevance to the input image meets the set requirement in the text generation process, and in response to the matching score being greater than the preset threshold, takes the text re-generated according to the adjusted sampling temperature as the final output text, thereby further improving the accuracy of text generation.

[0119] Figure 4 is a process schematic diagram of the scene text recognition device based on the multi-modal large language model provided by the application for scene text recognition. Figure 4 As shown in the figure, the process includes:

[0120] inputting an image + initial text prompt into the multi-modal large language model;

[0121] using a semantic anchor point recognition module to obtain a first token set constituted by tokens whose relevance to the input image meets a set requirement in the text generation process and a second token set constituted by tokens whose semantic stability meets a set requirement in the text generation process, forming a semantic anchor point paragraph according to the tokens continuously appearing in the intersection of the first token set and the second token set, and recording the sampling temperature of each token position in the text generation process;

[0122] using a temperature regulation generation module to adjust the sampling temperature of each token position in the text generation process based on the semantic anchor point paragraph, and re-generating text according to the adjusted sampling temperature;

[0123] The local rollback generation module is used to perform picture-text semantic matching evaluation on the generated text to obtain a matching score; in response to the matching score being lower than a preset score threshold, determining a token region not covered by the semantic anchor paragraph as a low-trust region and determining a token region covered by the semantic anchor paragraph as a high-trust region; further reducing the sampling temperature of the token position in the low-trust region, rolling back the decoding state of the low-trust region, and re-performing text generation on the low-trust region; merging the text generated by the low-trust region and the text generated by the high-trust region in the last text generation process as integrated generated text, and proceeding to the processing step of the first token set constituted by the tokens whose relevance to the input image meets the set requirement in the text generation process of the semantic anchor recognition module; in response to the matching score being greater than the preset threshold, the text generated by re-performing text generation according to the adjusted sampling temperature is taken as the final output text.

[0124] Figure 5 An example of an entity structure diagram of an electronic device is shown in Figure 5 The electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can invoke logical instructions in the memory 530 to execute a scene text recognition method based on a multi-modal large language model, which includes: obtaining a first token set constituted by tokens whose relevance to an input image meets a set requirement in a text generation process, and obtaining a second token set constituted by tokens whose semantic stability meets a set requirement in the text generation process, forming a semantic anchor paragraph according to tokens that continuously appear in the intersection of the first token set and the second token set, and recording the sampling temperature of each token position in the text generation process; adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor paragraph, and re-performing text generation according to the adjusted sampling temperature.

[0125] Moreover, the logic instructions in the memory 530 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0126] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the scene text recognition method based on a multi-modal large language model provided by the above-mentioned methods. The method comprises: obtaining a first token set constituted by tokens whose relevance to an input image meets a set requirement in a text generation process, and obtaining a second token set constituted by tokens whose semantic stability meets a set requirement in the text generation process, forming a semantic anchor paragraph according to tokens that continuously appear in the intersection of the first token set and the second token set, and recording the sampling temperature of each token position in the text generation process; adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor paragraph, and re-performing text generation according to the adjusted sampling temperature.

[0127] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the scene text recognition method based on a multi-modal large language model provided by the above-mentioned methods. The method comprises: obtaining a first token set constituted by tokens whose relevance to an input image meets a set requirement in a text generation process, and obtaining a second token set constituted by tokens whose semantic stability meets a set requirement in the text generation process, forming a semantic anchor paragraph according to tokens that continuously appear in the intersection of the first token set and the second token set, and recording the sampling temperature of each token position in the text generation process; adjusting the sampling temperature of each token position in the text generation process based on the semantic anchor paragraph, and re-performing text generation according to the adjusted sampling temperature.

[0128] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiment methods can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer readable storage medium. The computer readable storage medium is a disk, an optical disk, a read-only memory or a random access memory, etc.

[0129] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0130] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and a necessary general hardware platform, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in each embodiment or some parts of the embodiment.

[0131] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

[0132] The above is only a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements easily thought of by those skilled in the art within the technical range disclosed by the present application should be covered within the protection scope of the present application.

Claims

1. A scene text recognition method based on a multimodal large language model, characterized in that, include: A first set of tokens that meet the set requirements for relevance to the input image during text generation is obtained, and a second set of tokens that meet the set requirements for semantic stability during text generation is obtained. Semantic anchor paragraphs are formed based on the tokens that appear consecutively in the intersection of the first set of tokens and the second set of tokens, and the sampling temperature of each token position during text generation is recorded. Based on the semantic anchor paragraph, the sampling temperature at each token position during the text generation process is adjusted, and the text is regenerated according to the adjusted sampling temperature. The first token set, which consists of tokens whose relevance to the input image meets the set requirements during the text generation process, includes: For the token in the text generation process, extract the cross-attention vector of the token in the image-text cross-attention mechanism; wherein, the cross-attention vector reflects the degree of attention the token pays to each region of the input image; Calculate the cross-attention autocorrelation matrix between the cross-attention vectors of tokens within a set time window, obtain the relevant tokens whose cosine similarity to the cross-attention autocorrelation matrix is ​​greater than a preset similarity threshold, and mark the remaining tokens in the token sequence output by the multimodal large language model after filtering out the relevant tokens as the first token set. The second token set, which consists of tokens whose semantic stability meets the set requirements during the text generation process, includes: For the token in the text generation process, extract the predicted distribution of the token in each layer of the Transformer, calculate the consistency index of the predicted distribution based on the predicted distribution of the token in each layer of the Transformer, and mark the tokens whose predictive distribution consistency index meets the requirements as the second token set.

2. The scene text recognition method based on a multimodal large language model according to claim 1, characterized in that, The step of calculating the consistency index of the prediction distribution based on the prediction distribution of the token in each layer of the Transformer includes: Calculate the KL divergence of the predicted distribution of the token in each pair of adjacent layers of the Transformer, and determine the tokens whose average KL divergence is less than a preset threshold as tokens whose consistency index of the predicted distribution meets the requirements.

3. The scene text recognition method based on a multimodal large language model according to claim 1, characterized in that, The adjustment of the sampling temperature at each token position during text generation based on the semantic anchor paragraph includes: For the token position at the semantic anchor paragraph, maintain the sampling temperature at the token position or fine-tune the sampling temperature by a preset small amount; For token positions that are not located at the semantic anchor paragraphs, the sampling temperature is reduced by a relatively large preset amount.

4. The scene text recognition method based on a multimodal large language model according to claim 1, characterized in that, Before regenerating the text based on the adjusted sampling temperature, the method further includes: The low-confidence regions of the input image whose cross-modal matching confidence is lower than a preset confidence threshold are identified. Reduce the sampling temperature of the token positions corresponding to the low-confidence regions and / or edge regions of the input image.

5. The scene text recognition method based on a multimodal large language model according to claim 1, characterized in that, After regenerating the text based on the adjusted sampling temperature, the method further includes: The generated text is evaluated using image-text semantic matching to obtain a matching score; In response to the matching score being lower than a preset score threshold, the token region not covered by the semantic anchor paragraph is determined to be a low-confidence region, and the token region covered by the semantic anchor paragraph is determined to be a high-confidence region. Further reduce the sampling temperature of the token position in the low-confidence region, roll back the decoding state of the low-confidence region, and regenerate the text in the low-confidence region; The text regenerated in the low-confidence region is merged with the text in the high-confidence region generated in the previous text generation process to form the integrated generated text, and then the process is moved to the step of obtaining the first token set composed of tokens whose correlation with the input image meets the set requirements during the text generation process. If the matching score is greater than a preset threshold, the text obtained by regenerating the text based on the adjusted sampling temperature will be used as the final output text.

6. The scene text recognition method based on a multimodal large language model according to claim 5, characterized in that, The step of performing image-text semantic matching evaluation on the generated text to obtain a matching score includes: The generated text is vector-encoded to obtain a first vector; The input image is vector-encoded to obtain a second vector; The matching score is obtained by calculating the correlation between the first vector and the second vector.

7. A scene text recognition device based on a multimodal large language model, characterized in that, include: The semantic anchor recognition module is used to: obtain a first set of tokens that meet the set requirements for relevance with the input image during the text generation process, and obtain a second set of tokens that meet the set requirements for semantic stability during the text generation process; form semantic anchor paragraphs based on the tokens that appear consecutively in the intersection of the first set of tokens and the second set of tokens; and record the sampling temperature of each token position during the text generation process. The temperature control generation module is used to: adjust the sampling temperature at each token position during the text generation process based on the semantic anchor paragraph, and regenerate the text according to the adjusted sampling temperature; When the semantic anchor recognition module is used to obtain a first token set consisting of tokens whose relevance to the input image meets the set requirements during the text generation process, it is specifically used to: extract the cross-attention vector of the token in the image-text cross-attention mechanism for the token in the text generation process; wherein, the cross-attention vector reflects the degree of attention of the token to each region of the input image; calculate the cross-attention autocorrelation matrix between the cross-attention vectors of the tokens within a set time window, obtain the relevant tokens whose cosine similarity of the cross-attention autocorrelation matrix is ​​greater than a preset similarity threshold, and mark the remaining tokens after filtering out the relevant tokens in the token sequence output by the multimodal large language model as the first token set; When the semantic anchor recognition module is used to obtain the second token set consisting of tokens whose semantic stability meets the set requirements during the text generation process, it is specifically used to: extract the prediction distribution of the token in each layer of the Transformer for the token in the text generation process, calculate the consistency index of the prediction distribution based on the prediction distribution of the token in each layer of the Transformer, and mark the tokens whose consistency index of the prediction distribution meets the requirements as the second token set.

8. The scene text recognition device based on a multimodal large language model according to claim 7, characterized in that, It also includes a local rollback generation module; The local rollback generation module is used for: The generated text is evaluated using image-text semantic matching to obtain a matching score; In response to the matching score being lower than a preset score threshold, the token region not covered by the semantic anchor paragraph is determined to be a low-confidence region, and the token region covered by the semantic anchor paragraph is determined to be a high-confidence region. Further reduce the sampling temperature of the token position in the low-confidence region, roll back the decoding state of the low-confidence region, and regenerate the text in the low-confidence region; The text regenerated in the low-confidence region is merged with the text in the high-confidence region generated in the previous text generation process to form the integrated generated text. Then, the process is transferred to the semantic anchor recognition module to obtain the first token set composed of tokens whose correlation with the input image meets the set requirements during the text generation process. If the matching score is greater than a preset threshold, the text obtained by regenerating the text based on the adjusted sampling temperature will be used as the final output text.

Images