Control methods and electronic devices for outputting lexical units of pre-trained language models

Abstract

This application discloses a method and electronic device for controlling the output lexical units of a pre-trained language model, relating to the fields of computer vision and natural language processing. The method includes introducing visual features of text regions in the input image as cross-modal constraints. By locating the text regions of the input image and extracting visual features, the generation process is strongly bound to the visual and textual semantics of the input, providing a semantic anchor for subsequent temperature adjustment and avoiding semantic imbalance caused by indiscriminate temperature control. By calculating the cross-modal consistency score between the output lexical units and the text regions, a dynamic quantitative evaluation of the semantic adaptability of each lexical unit during the generation process is achieved. Based on the cross-modal consistency score, the sampling temperature is adaptively adjusted, enabling the model to optimize the temperature setting in real time according to semantic alignment during the generation process. Ultimately, a dynamic and refined balance between determinism and diversity / flexibility in the generation results is achieved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and natural language processing, and in particular to a method and electronic device for controlling the output lexical units of a pre-trained language model. Background Technology

[0002] In the interdisciplinary field of computer vision and natural language processing, scene text recognition is widely used in scenarios such as intelligent transportation. The current mainstream approach employs an "encode-decode" multimodal large-scale model framework: first, a visual encoder extracts image feature vectors, which are then embedded into a pre-defined text prompt template to construct a multimodal input sequence. After being input into the model, an autoregressive method predicts the output sequence containing text content and location information. The final recognition result is then obtained through linear mapping, decoding, and post-processing.

[0003] In language model decoding, temperature is a key hyperparameter controlling output diversity. Lower temperatures (close to 0) lead to more deterministic outputs, which is beneficial for factual and accuracy tasks; while higher temperatures (close to 1 or higher) increase the randomness and creativity of the output. This scheme uses a globally fixed temperature parameter for output prediction. A globally fixed temperature parameter is a "one-size-fits-all" strategy. When flexibility or diverse descriptions of reasoning are required, a fixed temperature limits the model's generation space, resulting in rigid text. When determinism of the result is required, a fixed temperature is difficult to meet high accuracy requirements, ultimately resulting in generated results that are either too flexible or inaccurate. Summary of the Invention

[0004] This application provides a method and electronic device for controlling the output lexical units of a pre-trained language model, which at least solves the technical problem that related technologies use globally fixed temperature parameters, which cannot adapt to the dynamic needs of different stages in the generation process, and thus cannot achieve a dynamic and refined balance between the determinism and diversity / flexibility of the generation results.

[0005] This application provides a method for controlling the output lexical units of a pre-trained language model, comprising: acquiring an input image of the pre-trained language model; locating text regions in the input image and extracting visual features of the text regions; calculating a cross-modal consistency score between the output lexical units and the text regions based on the visual features and the hidden state vector of the pre-trained language model decoder; wherein the hidden state vector is used to characterize the semantic state when the pre-trained language model generates output lexical units corresponding to the input image; the cross-modal consistency score is used to characterize the degree of semantic alignment between the output lexical units and the text regions; and adjusting the sampling temperature of the pre-trained language model based on the cross-modal consistency score, wherein the sampling temperature is used to control the randomness and diversity of the output lexical units of the language model.

[0006] This application also provides a control device for the output lexical units of a pre-trained language model, including:

[0007] The acquisition module is used to acquire the input image of the pre-trained language model;

[0008] The visual feature extraction module is used to locate text regions in the input image and extract the visual features of the text regions.

[0009] The computation module is used to calculate the cross-modal consistency score between the output word and the text region based on visual features and the hidden state vector of the pre-trained language model decoder. The hidden state vector is used to represent the semantic state when the pre-trained language model generates the output word corresponding to the input image; the cross-modal consistency score is used to represent the degree of semantic alignment between the output word and the text region.

[0010] The adjustment module is used to adjust the sampling temperature of the pre-trained language model based on the cross-modal consistency score. The sampling temperature is used to control the randomness and diversity of the language model's output lexical units.

[0011] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the control method for outputting lexical units of the pre-trained language model when executing the computer program.

[0012] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the control method for outputting lexical units of the pre-trained language model described above.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the control method for outputting lexical units of the pre-trained language model described above.

[0014] This application introduces visual features of text regions in the input image as a basis for cross-modal constraints. By locating the text regions in the input image and extracting visual features, the generation process is strongly bound to the visual and textual semantics of the input, providing a semantic anchor for subsequent temperature adjustment and avoiding the semantic imbalance problem caused by indiscriminate temperature control. By calculating the cross-modal consistency score between the output tokens and the text regions, a dynamic quantitative evaluation of the semantic adaptability of each token in the generation process is achieved. Based on the cross-modal consistency score, the sampling temperature is adaptively adjusted, enabling the model to optimize the temperature setting in real time according to the semantic alignment during the generation process. Ultimately, a dynamic and refined balance between the determinism and diversity / flexibility of the generation results is achieved. Attached Figure Description

[0015] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating the process of handling scene text recognition tasks using relevant technologies;

[0017] Figure 2 A schematic diagram of the specific hardware architecture on which the execution of a method for controlling the output lexical units of a pre-trained language model provided in this application embodiment depends;

[0018] Figure 3 A flowchart illustrating a method for controlling the output lexical units of a pre-trained language model, as provided in an embodiment of this application;

[0019] Figure 4 A schematic diagram of the structure of a control device for outputting lexical units of a pre-trained language model provided in an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To more clearly illustrate the embodiments of this application, the technical terms used in the embodiments will be briefly introduced below:

[0024] Large Language Models (LLMs) are a class of deep learning models based on the Transformer architecture. They have billions to trillions of parameters, are self-supervised pre-trained on massive amounts of unlabeled text data (such as predicting the next word in a text), and possess deep understanding of human natural language, high-quality generation of text that conforms to semantic logic, and also have the ability to reason, store and transfer knowledge.

[0025] Transformer is a neural network architecture built entirely on a multi-head self-attention mechanism. It is designed specifically for processing sequence data. Through an encoder-decoder structure, combined with modules such as word embedding, position encoding, and feedforward networks, it achieves efficient capture of global dependencies in sequences and supports a deep learning infrastructure for parallel computing.

[0026] In the interdisciplinary field of computer vision and natural language processing, scene text recognition tasks have significant application value. They aim to accurately extract text content from complex scene images and determine their spatial location, and are widely used in scenarios such as intelligent transportation, new retail, and image retrieval.

[0027] Currently, the industry typically employs a multimodal large language model technology framework based on "encoding-decoding" when handling this task. The specific implementation process of this technical solution is as follows: Figure 1As shown: First, the input scene image is fed into a visual encoder, which extracts features from the image to obtain a sequence of image feature vectors that represent the global information of the image. Then, based on the task requirements of scene text recognition, a text prompt template is pre-set (e.g., "Please identify the text in the image and its location:" or "The texts in the image...").<image_features> are:” and so on), and the previously extracted image feature sequence is used as a special visual token, which is embedded into a specific preset position in the text prompt template (such as in the template above).<image_features> The image feature vector sequence is concatenated with the text prompt template to construct a multimodal input sequence. This multimodal input sequence is then fed into a multimodal large language model, which uses an autoregressive approach to progressively predict the output sequence. This output sequence must simultaneously contain text content information and the text's spatial location information within the image (spatial location information is typically represented by the coordinates of the bounding box center point, bounding box width and height parameters, or bounding box corner coordinates). In the output stage, the token sequence generated by the multimodal large language model first passes through a linear mapping layer to transform it into specific character categories or spatial location coordinate parameters. It is then processed by a decoding module and a post-processing module, which typically includes dictionary lookup, inverse coordinate normalization transformation, and non-maximum suppression. Finally, the recognition result of each text instance and its spatial location in the image are obtained.

[0028] Although the aforementioned multimodal large language model technology based on the "encode-decode" framework has been applied to scene text recognition tasks, practical verification has shown that this technology still has significant limitations and cannot meet the requirements of high-precision and high-reliability scene text recognition. The specific problems are as follows:

[0029] Firstly, the core of the solution relies on the statistical correlation formed by the multimodal large language model during the pre-training process of large-scale image and text data. That is, the model indirectly obtains the correspondence between visual features and language features by learning the correlation between the overall image and the text description. However, this correspondence is only based on the correlation between the overall image and the overall text, and does not establish a clear mapping relationship between the local area of ​​the image and the text token. As a result, the model cannot evaluate and dynamically correct the matching degree between each generated text token and the local area of ​​the image in real time during the inference stage. Consequently, when the model generates specific text content, it is easy to have "illusion" phenomenon or semantic deviation problem, that is, generate text descriptions that do not match the actual text content contained in the image, which seriously reduces the accuracy of scene text recognition results.

[0030] Secondly, this scheme uses a globally fixed temperature parameter in the output prediction process of a multimodal large language model. This parameter setting method cannot adapt to the dynamic needs of different stages in the model generation process: when the model needs to make reasonable inferences or generate diverse text descriptions, the fixed temperature parameter will limit the model's generation flexibility, resulting in rigid and unnatural generated text content; while when the model needs to ensure the determinism of the generation results, the fixed temperature parameter is difficult to meet the high-precision generation requirements. This one-size-fits-all temperature parameter setting strategy cannot achieve a dynamic and refined balance between the determinism of the generation results and the diversity / flexibility.

[0031] Thirdly, considering the two issues mentioned above, multimodal large language models, in scene text recognition tasks, lack a dynamic perception and control mechanism for the alignment relationship between local image semantics and generated tokens. This ultimately leads to problems such as insufficient determinism, semantic deviation, or poor flexibility in the generated results. For example, when performing text recognition on a street scene image containing a shop sign, if the sign actually contains the text "COFFEEHOUSE", when the model generates the token "C", if its visual attention can be focused on the local image area where the letter "C" is located in the upper left corner of the sign, it means that the generated token is aligned with the local image semantics, and the recognition result is accurate. However, when the model generates the token "E", if its visual attention shifts to the background area below the sign or the light spot reflection in the image, rather than the local image area where the letter "E" is located, it indicates that the generated token deviates from the local image semantics, ultimately leading to an incorrect recognition result, fully demonstrating the limitations of the relevant technical solutions.

[0032] To address all or some of the aforementioned technical problems, this application introduces visual features of text regions in the input image as a cross-modal constraint basis. By locating the text regions in the input image and extracting their visual features, the generation process is strongly bound to the visual and textual semantics of the input, providing a semantic anchor for subsequent temperature adjustment and avoiding the semantic imbalance problem caused by indiscriminate temperature control. By calculating the cross-modal consistency score between the output word and the text region, a dynamic quantitative evaluation of the semantic adaptability of each word in the generation process is achieved. Specifically, the hidden state vector of the pre-trained language model decoder accurately represents the semantic state of the currently generated word, while the cross-modal consistency score quantifies the semantic alignment between the output word and the core input textual information by fusing this semantic state with the visual features of the text region. A higher score indicates that the generation result of the current word better matches the input semantic expectation, and the deterministic requirement has been met; a lower score indicates that the current word may have a semantic deviation risk, requiring adjustment of temperature parameters to balance determinism and diversity. This dynamic feedback mechanism based on real-time semantic evaluation replaces the static global temperature setting in related technologies, enabling temperature adjustments to precisely match the semantic requirements of each generation stage and achieving fine-grained control with on-demand adjustments.

[0033] Finally, this method adaptively adjusts the sampling temperature based on cross-modal consistency scores, constructing a dynamic balance mechanism between determinism and diversity. When the cross-modal consistency score is high, it indicates that the current lexical unit has accurately aligned with the semantics of the input text. In this case, the sampling temperature can be appropriately increased to enhance the randomness of lexical unit selection without compromising the determinism of the core semantics, thereby improving the diversity and expressive flexibility of the generated results. When the cross-modal consistency score is low, it indicates that the current lexical unit has a risk of semantic deviation. In this case, the sampling temperature needs to be lowered to reduce semantic fluctuations caused by random sampling, forcing the model to generate lexical units that better fit the input semantics, thus ensuring the determinism and accuracy of the generated results. This dynamic adjustment strategy of increasing temperature for high alignment and increasing diversity, and decreasing temperature for low alignment to maintain determinism completely solves the pain point of fixed temperature parameters in related technologies, which cannot meet the needs of different generation stages. It enables the model to optimize the temperature settings in real time according to the semantic alignment during the generation process, ultimately achieving a dynamic and refined balance between the determinism and diversity / flexibility of the generated results.

[0034] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 2 The diagram shows a specific hardware architecture on which the control method for the output lexical units of the pre-trained language model depends, including: a visual feature acceleration unit, a text lexical processing unit, a consistency calculation unit, a global statistics module, a temperature calculation unit, and a sampling execution unit.

[0036] The visual feature acceleration unit uses a graphics processing unit (GPU) as its core hardware carrier and relies on tensor cores to achieve parallel visual computing, and is responsible for performing tasks such as text region detection and whole-image feature extraction.

[0037] The text lexical processing unit mainly consists of a central processing unit (CPU) and an AI acceleration chip, responsible for processing the lexical sequences of text prompts and pre-trained language output. The CPU performs text segmentation through multi-core parallel processing and completes the concatenation of text prompt templates and visual features. The AI ​​acceleration chip converts text lexical units into semantic vectors of the same dimension as the visual features, while also supporting matrix operations in the linear projection layer. The feature interaction bus handles data transmission between the visual acceleration unit and the text lexical processing unit.

[0038] The consistency calculation unit is responsible for normalizing the visual feature set, calculating the magnitude of each visual vector and performing division operations, as well as calculating the cross-modal consistency score.

[0039] The global statistics module, with the CPU as its core, is responsible for generating the global consistency mean and standard deviation of the sequence. It completes the statistical calculation of the mean and standard deviation by traversing the cross-modal consistency scores.

[0040] The temperature calculation unit is used to determine whether the temperature needs to be adjusted. If adjustment is needed, the exponential calculator is called and the calculated initial temperature is limited to a preset temperature range to ensure that the sampled temperature does not have extreme values.

[0041] The sampling execution unit adjusts the probability distribution of the logarithmic odds vector and generates word sampling based on the adjusted sampling temperature.

[0042] After the visual feature acceleration unit completes image feature extraction, it transmits the feature data to the consistency calculation unit and triggers the global statistics module to update the global consistency mean and standard deviation. After the cross-modal consistency score is output, the temperature calculation unit performs temperature adjustment and then transmits the adjusted sampling temperature to the sampling execution unit.

[0043] Embodiments of this application provide a method for controlling the output lexical units of a pre-trained language model, such as... Figure 3 As shown, the method includes the following steps:

[0044] S301. Obtain the input image for the pre-trained language model.

[0045] Among them, the input image Input images can include, but are not limited to, single still images and video stream frames. The input image may carry a text prompt, which can be an image-based question-and-answer input, such as "Please describe the text appearing in the image." A pre-trained language model can infer the output words based on the input image.

[0046] S302. Locate the text region in the input image and extract the visual features of the text region.

[0047] In some embodiments, when locating text regions in an input image, a pre-trained text detection model is used to perform region detection on the input image to obtain text regions. The set of text regions can be represented as follows: , Used to represent the first in the input image The spatial location of the detected text region ( ) and width ,high This ensures that the boundary division and spatial location representation of text areas are unique and accurate.

[0048] The pre-trained text detection model can be a lightweight scene-specific text detection model, such as a Differentiable Binarization Network (DBNet), to reduce computational resource consumption and inference latency while ensuring detection performance. By introducing a pre-trained text detection model and adopting a standardized region parameter representation method, accurate quantification and efficient output of idle information in text regions can be achieved; the choice of a lightweight model further optimizes resource consumption after deployment.

[0049] When extracting visual features of text regions, a pre-trained visual model is first used to extract features from the input image to obtain a global feature map. Then, the region of interest alignment (RoIAlign) algorithm is used to extract the visual features of each text region from the global feature map.

[0050] The pre-trained visual model can be a visual Transformer model such as the Swin Transformer. The 768-dimensional high-dimensional output of Swin-T is suitable for high-precision scenarios, while the lightweight configuration of 256 or 512 dimensions can reduce computational overhead while ensuring feature effectiveness, making it suitable for embedded devices or real-time processing scenarios.

[0051] The global feature map is a two-dimensional feature map of the intermediate layer. , Indicates the first The feature extraction operation of the layer ultimately yields a two-dimensional feature map. ,in , Overall downsampling factor for the backbone network; channel dimension The dimension is determined by the output dimension of the last layer of the network. For example, the output dimension of Swin-T is 768, while the lightweight model can have 256 or 512 dimensions.

[0052] Using the region of interest alignment algorithm in the two-dimensional feature map Extract a specific text region visual features ,in, This refers to the fixed space size after alignment.

[0053] The above-mentioned feature extraction advantage of the pre-trained visual model based on the window attention mechanism can capture the long-distance dependency relationship between pixels and global semantic information in the input image. The generated global feature map contains the overall visual context of the image. Furthermore, the structured and high-density representation of feature information is achieved through standardized dimensional design. On this basis, the RoIAlign algorithm is introduced to extract the visual features of a single text region from the global feature map, achieving accurate alignment between the region of interest and the feature map, so that the visual features retain the key visual information of the text region.

[0054] After extracting the visual features of each text region, average pooling can be performed on the visual features of the text regions to obtain the visual feature representation of the text regions. Then, the visual feature representation is mapped into a visual semantic vector through a linear projection layer.

[0055] Visual semantic vectors are high-level visual semantic representations of text regions. These representations are mapped to the same semantic embedding space as word units, thus maintaining comparability and consistency with semantic embedding vectors within the semantic embedding space. The set of visual semantic vectors can be represented as... Among them, visual semantic vectors . It is a linear projection matrix used to represent visual features from the visual feature dimension. (e.g., 512 or 1024) mapped to the embedding dimension of the pre-trained language model (Usually 512, 768, or 1024); This is the bias term of the projection, used to perform translation correction on the mapping result. Each visual semantic vector v j A semantic embedding vector corresponding to a text region can be matched and aligned with lexical units in cross-modal models.

[0056] The 3D visual features output by RoIAlign are compressed using average pooling, aggregating high-dimensional information into a one-dimensional visual feature representation. This preserves the core information of the visual features of the text region while eliminating redundant details in the spatial dimension, allowing the feature representation to focus on the essential visual attributes of the text region. The introduction of a linear projection layer enables the precise transformation of visual features into a semantic space. Through parameter learning, the visual feature representation is semantically encoded and mapped into a visual semantic vector to uncover the potential correlation between visual features and text semantics. This transforms the underlying visual information into a semantically discriminative vector representation, ensuring that the differences in features between different text regions accurately correspond to their semantic differences.

[0057] For example, the pseudocode is as follows:

[0058] import numpy as np

[0059] class InputModule:

[0060] def __init__(self,backbone,text_detector,roi_align,pooling,token_dim):

[0061] """

[0062] backbone: Visual feature extraction backbone network

[0063] text_detector: Text detection models (DBNet, etc.)

[0064] roi_align:RoIAlign operation function

[0065] pooling: Region feature pooling function

[0066] token_dim: Language token embedding dimension d

[0067] """

[0068] self.backbone = backbone

[0069] self.text_detector=text_detector

[0070] self.roi_align=roi_align

[0071] self.pooling = pooling

[0072] self.token_dim=token_dim

[0073] #Linear projection matrix W_v and bias b_v

[0074] self.W_v=np.random.randn(token_dim,backbone.output_dim)#dxC

[0075] self.b_v=np.random.randn(token_dim)#d

[0076] def detect_text_regions(self,image):

[0077] """

[0078] Text detection returns candidate text boxes R={r_1,r_2,...,r_n}

[0079] Each r_j = (x, y, w, h)

[0080] """

[0081] regions = self.text_detector(image) # Pseudo-function, returns a list

[0082] return regions

[0083] def extract_feature_map(self,image):

[0084] """

[0085] Full image feature extraction F=Backbone(I)

[0086] Output F∈R^(H`xW`xC)

[0087] """

[0088] F = self.backbone(image)

[0089] return F

[0090] def extract_region_feature(self,feature_map,region):

[0091] """

[0092] Perform RoIAlign+ pooling on a single text region

[0093] f_j=RoIAlign(F,r_j)->CxPxP

[0094] f_tilde_j = Pooling(f_j)->C

[0095] """

[0096] f_j=self.roi_align(feature_map,region)#CxPxP

[0097] f_tilde_j=self.pooling(f_j)#C

[0098] return f_tilde_j

[0099] def project_to_token_space(self,f_tilde_j):

[0100] """

[0101] Map visual features to language token embedding space

[0102] v_j=W_vf_tilde_j+b_v∈R^d

[0103] """

[0104] v_j = self.W_v@f_tilde_j + self.b_v

[0105] return v_j

[0106] def forward(self,image,prompt=None):

[0107] """

[0108] Complete process of input module

[0109] Input: image∈R^(HxWx3), with optional prompt

[0110] Output: Visual feature set V={v_1,v_2,...,v_n},v_j∈R^d

[0111] """

[0112] # 1. Text Region Detection

[0113] R=self.detect_text_regions(image)#R={r_1,r_2,...,r_n}

[0114] #2. Feature Extraction of the Whole Image

[0115] F=self.extract_feature_map(image)#F∈R^(H`xW`xC)

[0116] #3. Extract features from each text region and map them to the token space.

[0117] V=[]

[0118] for r_j in R:

[0119] f_tilde_j=self.extract_region_feature(F,r_j)#f_tilde_j∈R^C

[0120] v_j=self.project_to_token_space(f_tilde_j)#v_j∈R^d

[0121] V.append(v_j)

[0122] V=np.stack(V,axis=0)#nxd

[0123] return V”

[0124] This pseudocode transforms text regions in the input image into a set of visual features that can be aligned with the output lexical units of the pre-trained language model, providing a foundational input for subsequent cross-modal consistency assessment and sampling temperature control.

[0125] First, from the perspective of class initialization and dependency injection, the constructor "init" of "InputModule" receives five core components through parameters, each corresponding to a key step in the processing flow: "backbone" is a pre-trained visual extraction model (such as Swin Transformer), responsible for extracting high-level visual features from the entire image; "text_detector" is a pre-trained text detection model (such as DBNet), used to locate text regions in the input image; "roi_align" is the region of interest alignment operation function, which accurately extracts local features of text regions from the global feature map; "pooling" is the region feature pooling function, which transforms two-dimensional local features into compact one-dimensional vectors; and "token_dim" is the embedding dimension of the language token (such as 512 or 768), used to determine the target dimension of the final mapping of visual features. Meanwhile, the constructor initializes the linear projection matrix "self.W_v" (with dimensions "token_dim×backbone.output_dim") and the bias term "self.b_v" (with dimensions "token_dim"). These two parameters are the core carriers for mapping visual features to the language token space. Their initial values ​​are randomly generated by "np.random.randn" and can be optimized through model training to ensure the alignment accuracy of cross-modal features.

[0126] Secondly, the five member functions defined within the class correspond to the detailed steps of input processing, forming a logical closed loop. The "detect_text_regions" function is the first step in the process. After receiving the input image, it calls the injected "text_detector" model to detect text regions in the input image and returns a list of "regions" containing multiple text boxes. Each text box records the top-left corner coordinates (x, y), width (w), and height (h) of the text region in the form of "(x, y, w, h)". This step provides spatial localization for subsequent feature extraction that focuses on text regions and ignores irrelevant backgrounds. The "extract_feature_map" function is responsible for the overall image feature extraction. It calls the "backbone" network to process the input image and outputs a two-dimensional feature map F with dimensions "H`×W`×C", where H` and W` are the dimensions of the image after downsampling by the backbone network, and C is the number of feature channels. This feature map contains global high-level visual information of the image and is the basis for subsequent extraction of local text features.

[0127] The "extract_region_feature" function is a crucial step in connecting the overall image features with local text features. For each text box region, the "roi_align" operation is used to extract the corresponding local feature block f_j (with dimensions C×P×P, where P×P is the fixed spatial size after alignment) from the overall image feature map. The advantage of this step is that it can accurately capture the detailed features of the text region without destroying the correspondence in the feature space. Then, the "pooling" function is called to perform average pooling on f_j, compressing the two-dimensional local feature block into a one-dimensional region feature vector f_tilde_j (with dimensions C), which meets the processing requirements of language models for one-dimensional semantic vectors. At the same time, the average pooling operation can also enhance the robustness of the features and reduce local noise interference. The "project_to_token_space" function implements the spatial mapping from visual features to linguistic features. It uses matrix multiplication "self.W_v@f_tilde_j" to convert the dimension of the one-dimensional visual feature f_tilde_j from C to token_dim. Then, it adds a bias term self.b_v for translation correction, finally obtaining a visual semantic vector v_j (with the dimension token_dim) of the same dimension as the linguistic token. This step is the core of achieving cross-modal feature alignment, ensuring that visual features can be matched and evaluated for consistency with the linguistic token in the same semantic space.

[0128] Finally, the "forward" function, acting as the "main entry point" of the input module, connects all the sub-steps to form a complete processing flow: first, it calls "detect_text_regions" to obtain the set of text boxes R; then, it uses "extract_feature_map" to obtain the features of the entire image; subsequently, it iterates through each text box, sequentially executing "extract_region_feature" and "project_to_token_space" to obtain the corresponding visual semantic vector v_j; and collects all v_j into a list V. Finally, it uses "np.stack(V,axis=0)" to convert the list into a matrix of dimension n×token_dim (where n is the number of text boxes) and returns it. It's worth noting that the "forward" function also supports receiving optional text prompts. Although these are not directly involved in feature processing in the current code, they provide an extension interface for subsequent multimodal inputs, satisfying the design requirements of multimodal input pairs.

[0129] The above embodiments achieve accurate local text region localization through text detection and RoIAlign, avoiding interference from background information in global features; visual features are mapped to the language token space through linear projection, providing a prerequisite for subsequent cross-modal consistency evaluation; at the same time, the modular design gives the modules good flexibility and scalability, and different backbone networks, text detection models or pooling methods can be replaced according to task requirements to adapt to various multimodal tasks such as scene text recognition and image-text question answering.

[0130] S303. Calculate the cross-modal consistency score between the output lexical unit and the text region based on the visual features and the hidden state vector of the pre-trained language model decoder.

[0131] Among them, the hidden state vector is used to represent the probabilistic state when the language model generates output words; the cross-modal consistency score is used to represent the semantic alignment of the output words with the text region.

[0132] In some embodiments, when performing step S303, the visual semantic vector and the latent state vector are first normalized to obtain the standard visual semantic vector and the standard latent state vector; then the cosine similarity between the standard visual semantic vector and the standard latent state vector is calculated, and the cross-modal consistency score is determined based on the cosine similarity.

[0133] Among them, the visual semantic vector originates from the semantic mapping of the visual features of the image, while the hidden state vector comes from the text semantic encoding of the pre-trained language model.

[0134] Specifically, each visual semantic vector in the visual semantic vector set is normalized to obtain a standardized visual feature set. When the pre-trained language model generates the i-th word, it extracts the hidden state vector of the model's decoder. After normalization, the standardized hidden state vector is obtained. Calculate the cosine similarity between the standardized latent state vector and the standardized visual semantic vector. .

[0135] The above embodiments eliminate the interference of differences in dimensional scale between visual semantic vectors and latent state vectors through normalization processing. Normalization processing maps these two types of vectors to a unit hypersphere space. The resulting standard visual semantic vectors and standard latent state vectors have the same modulus constraint, making the angle between the vectors the core indicator of semantic relevance. Then, the cosine similarity is used to accurately quantify this angle relationship. The higher the similarity, the stronger the consistency between visual semantics and text semantics, and vice versa. This quantification method can objectively reflect the matching degree of cross-modal information, effectively avoid misjudgment caused by scale differences, and improve the accuracy of cross-modal consistency assessment.

[0136] Optionally, in determining the cross-modal consistency score based on cosine similarity, the region index of the text region corresponding to the maximum cosine similarity is first determined. This region index is used to visualize the association between the output tokens and the text region. The maximum cosine similarity is then used as the cross-modal consistency score corresponding to the region index.

[0137] Specifically, select the largest cosine similarity. , ,Record It can be used for visualization or subsequent dynamic adjustment of sampling temperature. The maximum cosine similarity is used as the cross-modal consistency score of the i-th word, which reflects the semantic alignment between the word and the text region. This ensures that each word can be associated with the text region with the highest semantic fit, effectively avoiding problems such as fuzzy multi-region matching and semantic alignment deviation.

[0138] The aforementioned optional implementation achieves a precise one-to-one association between words and text regions by filtering for the maximum cosine similarity and locking the corresponding region index, thus improving the accuracy of cross-modal semantic alignment. The maximum cosine similarity is directly used as the cross-modal consistency score for that word, quantifying the semantic alignment strength between the word and the corresponding text region. By recording the region index corresponding to the maximum similarity, each output word can be bound to a specific text region in the input image, and the correspondence between words and text regions can be intuitively presented through visualization. This visualization capability not only facilitates technical personnel in verifying the rationality of the alignment results but also allows for quick location of alignment errors, providing an intuitive reference for model optimization and parameter adjustment.

[0139] For example, the pseudocode is as follows:

[0140] import numpy as np

[0141] class CrossModalConsistency:

[0142] def__init__(self):

[0143] pass

[0144] @staticmethod

[0145] def l2_normalize(x,axis=-1,eps=1e-8):

[0146] """

[0147] L2 normalization

[0148] x: np.array, any shape

[0149] axis: Along which dimension is normalized?

[0150] """

[0151] norm=np.linalg.norm(x,ord=2,axis=axis,keepdims=True)

[0152] return x / (norm+eps)

[0153] def compute_consistency(self,visual_features,token_hidden):

[0154] """

[0155] visual_features: V∈R^(nxd), the set of image visual features

[0156] token_hidden: t_i∈R^d, the current hidden state of the tokendecoder.

[0157] return:

[0158] S_i: Cross-modal consistency score

[0159] j_star: Index of the text region corresponding to the maximum similarity

[0160] """

[0161] #1. Perform L2 normalization on visual features

[0162] V_tilde=self.l2_normalize(visual_features,axis=1)#nxd

[0163] #2. Perform L2 normalization on the current hidden state of the token.

[0164] `t_tilde=self.l2_normalize(token_hidden,axis=0)#d` normalizes a one-dimensional vector.

[0165] #3. Calculate the cosine similarity s_ij = t_i^T * v_j

[0166] # Equivalent to matrix multiplication: 1xd@dxn=1xn

[0167] s_ij = np.dot(V_tilde, t_tilde) # n, where s_ij is the similarity score for each visual region.

[0168] #4. Maximum similarity selection

[0169] j_star=np.argmax(s_ij)

[0170] S_i=s_ij[j_star]

[0171] returnS_i,j_star”

[0172] This pseudocode quantifies the semantic alignment between the generated language token and the local visual features of the image. Through feature normalization, similarity calculation, and the selection of the maximum associated region, it outputs the cross-modal consistency score and the corresponding visual region index, which are key indicators for subsequent sampling temperature control.

[0173] From the overall design of the class, "CrossModalConsistency" adopts a lightweight architecture. The constructor "init" is only implemented by empty "pass" and no additional member variables are initialized. This is because the core logic of the module (normalization and similarity calculation) does not depend on predefined parameters. It only needs to receive input data to complete the calculation, which is in line with the design concept of stateless utility classes and is convenient to be flexibly called in the multimodal model inference process.

[0174] The core functionality of the class is handled by two functions. The static method `l2_normalize`, annotated with `@staticmethod`, performs L2 normalization on the input features, a crucial preprocessing step ensuring cross-modal feature comparability. Its principle is to first calculate the L2 norm (i.e., the magnitude of the vector) of the input array `x` along the specified dimension `axis` using `np.linalg.norm(x,ord=2,axis=axis,keepdims=True)`, then divide the original array by this norm to make the normalized feature vector have a magnitude of 1. Simultaneously, a minimum value `eps=1e-8` is added to the denominator to avoid division errors caused by the original feature vector having a magnitude of 0. This method supports input arrays of arbitrary shapes, adapting to both subsequent two-dimensional visual feature sets (n×d dimensions) and one-dimensional token latent state vectors (d dimensions), providing a universal tool for unified processing of cross-modal features.

[0175] The `compute_consistency` function is the core execution entry point of the module, receiving two key inputs: "visual_features" (a set of image visual features of dimension n×d, where n is the number of text regions and d is the feature dimension, i.e., the set of visual semantic vectors V output by the module) and "token_hidden" (the hidden state t_i of the current token decoder in dimension d, representing the semantic state when the language model generated the token), ultimately outputting two core results: S_i (cross-modal consistency score) and j_star (the index of the text region most relevant to the current token). Its computation process can be divided into four steps:

[0176] The first step is visual feature normalization. The "l2_normalize" method is called to process "visual_features" along dimension 1 (i.e., feature dimension d), resulting in a standardized visual feature set V_tilde (still n×d dimension). The purpose of this step is to eliminate differences in modulus between features from different visual regions. For example, if a text region's features have a larger modulus due to high brightness, without normalization, subsequent similarity calculations will be biased towards that region. After normalization, all visual semantic vectors are on the same modulus scale, ensuring that similarity reflects only the consistency of semantic direction, rather than differences in numerical magnitude.

[0177] The second step is token hidden state normalization. The "l2_normalize" method is called to normalize the one-dimensional vector "token_hidden" along dimension 0 (i.e., the entire vector dimension d), resulting in "t_tilde" (still dimension d). This step is to maintain consistency with the normalization standard of visual features, ensuring that cross-modal features are comparable in the same semantic space coordinate system. When the modulus of both the language token feature and the visual region feature is 1, their dot product is directly equivalent to cosine similarity, accurately measuring the degree of semantic alignment.

[0178] The third step is cosine similarity calculation, implemented using "np.dot(V_tilde,t_tilde)": "V_tilde" is an n×d two-dimensional array, and "t_tilde" is a d-dimensional one-dimensional array. The dot product operation between them essentially involves taking the normalized feature vector of each visual region (each row of "V_tilde") and the normalized token latent state vector (t_tilde) as the dot product, ultimately resulting in a one-dimensional array s_ij of length n. Each element in the array corresponds to the cosine similarity between a visual region and the current token. The cosine similarity ranges from [-1, 1]. The closer the value is to 1, the closer the two are in the semantic space, meaning the higher the semantic alignment between the token and the visual region.

[0179] The fourth step is maximum similarity selection. The index j_star corresponding to the maximum value in s_ij is found using "np.argmax(s_ij)". This index represents the text region that best matches the semantics of the current token. This maximum value is then used as the cross-modal consistency score S_i for the current token. The logic of this step is to focus on the most relevant visual region. In scene text recognition tasks, each generated token (such as letters or characters) theoretically corresponds to only one text region in the image. By selecting the region with the highest similarity, the visual basis of the token can be accurately located, avoiding alignment evaluation bias caused by interference from multiple regions.

[0180] Traditional multimodal models rely solely on global feature associations and cannot determine the correspondence between generated tokens and local image regions. In contrast, "CrossModalConsistency" eliminates feature scale interference through L2 normalization and quantifies semantic direction consistency through cosine similarity. The final outputs S_i and j_star not only provide a quantitative basis for sampling temperature control (e.g., a high S_i reduces the temperature to enhance certainty), but also enable the tracing of "token-visual region" through j_star, supporting the interpretability of the model's generated results (e.g., visualizing the image region corresponding to the token).

[0181] S304. Adjust the sampling temperature of the pre-trained language model based on the cross-modal consistency score.

[0182] Sampling temperature controls the randomness and diversity of the language model's output. It determines whether the pre-trained language model selects output words more cautiously or more freely. Cross-modal consistency scores directly reflect the semantic matching quality between words and text regions. Adjusting the sampling temperature based on this score essentially integrates visual semantic alignment feedback into the text generation process. This allows for dynamic correction of subsequent generation logic based on the quality of preceding generation results, ensuring that the pre-trained language model's output possesses both accuracy in high-confidence scenarios and error correction capabilities in low-confidence scenarios, thus improving the overall reliability of the output.

[0183] In some embodiments, if the cross-modal consistency score is greater than a preset score, the sampling temperature is reduced; if the cross-modal consistency score is less than a preset score, the sampling temperature is increased.

[0184] When the cross-modal consistency score is higher than the preset score, it indicates that the output words of the pre-trained language model are highly aligned with the text regions in the input image, resulting in high reliability of the generated results. In this case, the sampling temperature can be reduced to compress the probability distribution range generated by the pre-trained language model, making the output words more accurate and avoiding semantic deviation caused by overexploration. When the cross-modal consistency score is lower than the preset score, it indicates that the alignment between the output words of the pre-trained language model and the text regions in the input image is insufficient, suggesting that the image regions are blurry or the attention of the pre-trained language model is scattered, resulting in high uncertainty in the generated results. In this case, the sampling temperature can be increased to allow the pre-trained language model to explore more possibilities, increase the flexibility of the output words, effectively avoid the accumulation of misjudgments caused by a single probability selection, and improve the model's error correction ability and adaptability to ambiguous semantics and complex scenes.

[0185] In some embodiments, adjusting the sampling temperature of the pre-trained language model based on the cross-modal consistency score includes: subtracting the cross-modal consistency score from the global mean and taking the absolute value; if the absolute value is greater than or equal to the consistency threshold, calculating the initial temperature; and determining the sampling temperature based on the initial temperature, wherein the sampling temperature is within a preset temperature range.

[0186] By calculating the absolute value of the difference between the cross-modal consistency score and the global mean, the deviation of the alignment quality of a single lexical unit from the overall average level can be accurately quantified. When this absolute value is greater than or equal to the consistency threshold, it indicates that the cross-modal alignment quality of the current lexical unit is abnormal. At this time, the initial temperature calculation is triggered and the sampling temperature is adjusted, making the temperature adjustment more targeted. For lexical units with deviations less than the threshold, their alignment quality is within the normal fluctuation range, and no additional temperature adjustment is required, avoiding the generation logic disorder caused by excessive intervention. This deviation judgment mechanism based on global statistical features breaks through the limitation of ignoring individual differences by single threshold judgment. It enables temperature adjustment to accurately respond to abnormal alignment scenarios while respecting the generation rules under normal scenarios, improving the rationality and accuracy of the adjustment strategy.

[0187] If the absolute value is greater than or equal to the consistency threshold, the initial temperature is calculated, including: if the cross-modal consistency score is greater than the global mean, the initial temperature is determined to be less than the baseline temperature; if the cross-modal consistency score is less than the global mean, the initial temperature is determined to be greater than the baseline temperature. The baseline temperature, as the model's default generation parameter, represents the balancing strategy under normal scenarios.

[0188] Introducing the global mean as a dynamic reference benchmark reflects the overall alignment level of the current batch, making the judgment of deviation direction more relevant to specific scenarios. When the cross-modal consistency score is greater than the global mean, it indicates that the cross-modal alignment quality of the current word is better than the overall average level, and the confidence of semantic matching is higher. In this case, setting the initial temperature to be lower than the benchmark temperature can compress the generation probability distribution, making the model more inclined to select candidate words with the best probability, thus consolidating the accurate generation results brought by high-confidence alignment. When the cross-modal consistency score is less than the global mean, it indicates that the alignment quality of the current word is lower than the overall level, and there is uncertainty in semantic matching. In this case, setting the initial temperature to be higher than the benchmark temperature can broaden the model's probability selection space, encourage the model to explore more potentially reasonable words, and effectively compensate for the generation bias caused by insufficient alignment quality. This adjustment strategy makes the initial temperature highly consistent with the semantic connotation of alignment quality, improving the targeting and rationality of temperature adjustment.

[0189] The sampling temperature is determined based on the initial temperature, including: if the initial temperature is lower than the lower limit of the preset temperature range, the sampling temperature is determined to be equal to the lower limit; if the initial temperature is higher than the lower limit but lower than the upper limit of the preset temperature range, the sampling temperature is determined to be equal to the initial temperature; if the initial temperature is higher than the upper limit, the sampling temperature is determined to be equal to the upper limit. This rigid constraint design of the upper and lower temperature limits establishes a safe operating range for the sampling temperature, fundamentally avoiding the damage to the generated results caused by extreme temperatures.

[0190] When the initial temperature is below the lower limit, the lower limit is used as the final sampling temperature to ensure that the model retains its basic exploratory nature and avoids overly rigid generated results. When the initial temperature is above the upper limit, the upper limit is used to constrain the sampling temperature to prevent the generation logic from getting out of control and to ensure the coherence of semantic expression. This boundary constraint mechanism sets a safe threshold for the generation process, keeping the sampling temperature within a reasonable range and improving the stability and reliability of cross-modal text generation results.

[0191] Optionally, the sampling temperature can be calculated based on the cross-modal consistency score according to the following formula (1).

[0192] (1)

[0193] In formula (1), The reference temperature is usually set to 1.0, but can be adjusted according to the conservatism of the task. For the first Cross-modal consistency score of each token This is the global consistency mean of the generated sequence.

[0194] As a consistency threshold, only when Adjust the temperature as needed;

[0195]

[0196] in It is an adjustable coefficient. The standard deviation of the cross-modal consistency score of the generated sequence.

[0197] To adjust the coefficient and control the sensitivity of temperature to uniformity changes, an experimental range of 0.5 to 5.0 can be selected. These are the upper and lower limits of temperature. The value is set to 0.5 to 0.7 to avoid overly extreme sampling behavior.

[0198] Specifically, the steps for calculating the sampling temperature are as follows:

[0199] The first step is to define the basic parameters and global statistical characteristics required for the calculation; this is a prerequisite for temperature calculation. First, the values ​​of three core fixed parameters need to be determined: one is the reference temperature T. base The default value is 1.0, which can be adjusted according to the conservative deterministic requirements of the task. For example, the baseline value can be appropriately lowered for industrial quality inspection scenarios with extremely high accuracy requirements. Secondly, there is an adjustment coefficient, which needs to be selected experimentally from the range of (0.5~5.0) to control the sensitivity of temperature to changes in consistency scores. The larger α is, the higher the consistency score. i The greater the difference, the more significant the temperature fluctuation; thirdly, the upper and lower limits of temperature T min With T max T min The temperature is fixed between 0.5 and 0.7 to avoid excessive hardening of the product due to excessively low temperatures. max This is set according to task requirements to prevent semantic confusion caused by excessive temperature. Simultaneously, two global statistical features of the generated sequence need to be calculated: one is the global consistency mean. That is, the cross-modal consistency score S of all currently generated tokens. i The average value reflects the overall visual-linguistic alignment level of the sequence; the second is the standard deviation of the consistency score σ. S tokenS has been generated. i The degree of dispersion is used to measure the range of fluctuations in the alignment within a sequence.

[0200] The second step is to calculate the consistency threshold τ, which delineates the boundary between "no temperature adjustment required" and "temperature adjustment required". The calculation logic for the threshold τ is dynamically set based on the degree of sequence fluctuation, and the formula is as follows: Where k is an adjustable coefficient selected from (0.3~1.0), the smaller the value of k, the smaller τ, which means a slight S i Deviation requires temperature adjustment, and the adjustment frequency is higher; the larger the k value, the larger the τ, only when S iTemperature adjustment is only performed when the temperature deviates significantly from the global mean to avoid generation instability caused by frequent fine-tuning. The introduction of τ filters out S... i The temperature is adjusted only for tokens that significantly deviate from the overall alignment level of the sequence, thus balancing the accuracy and stability of temperature control.

[0201] The third step is to determine whether the temperature needs to be adjusted, based on S. i and Branch selection is performed based on the relationship between the difference and τ. The cross-modal consistency score S of the current token is then used for selection. i Compared with the global mean Taking the difference and the absolute value, we get |Si- |: If the absolute value is less than τ, it means that the alignment of the current token is not significantly different from the overall sequence level, and there is no need to adjust the temperature; simply set the sampling temperature T directly. i Set as reference temperature T base To maintain the stability of the generation process; if the absolute value is greater than or equal to τ, it indicates that the alignment of the current token deviates significantly from the overall level of the sequence, and dynamic temperature adjustment logic needs to be entered to calculate the appropriate sampling temperature through the formula.

[0202] The fourth step is to dynamically calculate the sampling temperature and limit extreme values ​​to ensure that temperature adjustments conform to the core logic of alignment determining randomness. When dynamic temperature adjustment is required, it is first performed using the exponential formula (Tbasecdotexp[-alpha(Si- )]) Calculate the initial temperature: If Si> (This indicates that the current token alignment is higher than the sequence mean), then Si- If the value is positive, the initial temperature will be less than T. base To achieve high alignment and lower temperature, reduce generation randomness and enhance determinism; if Si < This indicates that the current token alignment is lower than the sequence mean, then Si- If the value is negative, the initial temperature will be greater than T. base This allows for lower alignment rise temperatures and increases generation flexibility to explore more possibilities.

[0203] Subsequently, the initial temperature is limited to T using the clip function. min With T max Between: If the initial temperature is below T min Then force it to be set to T min To avoid over-determinism leading to rigid generation; if the initial temperature is higher than T max Then force it to be set to T max To avoid excessive randomness leading to semantic confusion, the final temperature T is obtained. iThis refers to the sampling temperature that adapts to the current token alignment state. It can be used to adjust the softmax probability distribution of decoderlogits, thereby achieving token-level generation quality control.

[0204] In some embodiments, the method further includes: adjusting the log-odds vector of the pre-trained language model based on the sampled temperature, wherein the log-odds vector corresponds to the output word, and each element in the log-odds vector is the original preference score of the pre-trained language model for the candidate word; and normalizing the adjusted log-odds vector to obtain the probability distribution of the output word.

[0205] In this context, the elements in the log-odds vector represent the model's original preference scores for candidate nouns, and their numerical differences directly reflect the priority ranking of nouns.

[0206] Adjusting the log-odds vector of the pre-trained language model based on the sampled temperature includes: dividing each element in the log-odds vector by the sampled temperature to obtain the adjusted log-odds vector.

[0207] The adjusted log odds vector is normalized to obtain the probability distribution of the pre-trained language model output, including: calculating the exponential score of each element in the adjusted log odds vector; and calculating the probability distribution of the language model output lexical based on the exponential score of each element.

[0208] Decoder log odds vector Divide by To adjust the softmax probability distribution ,in, Let w represent the probability distribution of the i-th token selecting candidate word w.

[0209] Subsequently, random sampling is performed from this distribution according to probability to obtain the final generated token.

[0210] The above method, by scaling the log-odds vector by division according to the sampling temperature, establishes a direct correlation between temperature and the concentration of the probability distribution, enabling fine-grained control over word selection preferences. The softmax function transforms the scaled log-odds vector into a distribution that conforms to probability axioms, making the probability of word selection quantifiable and traceable.

[0211] calculate The specific steps are as follows: First, obtain the decoder log-odds vector z of the decoder corresponding to the i-th token. i In tasks such as text generation, the model's decoder outputs a log-odds vector z with dimensions identical to the vocabulary size for the generation of the i-th token. i Each element z in the vector w(w represents a candidate word in the vocabulary), which is the model's original prediction score for selecting candidate word w for the i-th token. This score is not normalized, and its value only reflects the model's preference for candidate word w, and cannot be directly used as a probability.

[0212] Next, using the sampling temperature T i For the log-probability vector z i Adjust each element in the vector zi. Restore the original score z corresponding to each candidate word w in the vector zi. w Divide by the current token's unique sampling temperature T i The adjusted score (z) is obtained. w / T i Sampling temperature T i Its function is to control the sharpness of the probability distribution: when T i When T < 1, the adjusted score difference will be amplified, and the final probability distribution will be more concentrated (the model tends to select candidate words with higher original scores); when T i When T > 1, the score difference will be reduced and the probability distribution will be smoother (the randomness of the model selecting different candidate words increases); when T i When =1, the score is not adjusted, and the probability distribution perfectly matches the trend of the original logits.

[0213] Finally, the adjusted score is normalized using the softmax function to obtain the probability P of selecting candidate word w for the i-th token. i (w). In the specific calculation, first calculate the adjusted score (z) for each candidate word w. w / T i Calculate the exponential function exp(z) w / T i (Convert the scores to non-negative values, while amplifying the advantage of larger scores); then calculate the sum of the exponential scores of all candidate words k, using this as the normalized denominator; finally, divide the exponential score of candidate word w by this sum to obtain P. i (w). This process ensures that the sum of the probabilities of all candidate words is 1, and P i The value of (w) directly reflects the probability that the i-th token will select candidate word w. i The larger (w) is, the more the model tends to identify the i-th token as candidate word w under the current sampling temperature Ti adjustment.

[0214] In summary, the method for controlling the output tokens of a pre-trained language model provided in this application utilizes a pre-trained text detection model to locate text regions in an image, and simultaneously uses a pre-trained visual model to extract the global feature map of the image (including visual features of text regions), thereby obtaining the visual semantic vectors of each text region. This provides a visual semantic basis for the local text regions of the image for subsequent evaluation. Next, the method combines the language model decoder to generate a hidden state vector representing the semantic state of the current output token, and calculates the cross-modal consistency score between the output token and the text box region. This score accurately reflects the semantic alignment degree between the output token and the text box region. By quantifying the semantic alignment quality (i.e., the cross-modal consistency score) between the output token and the text region of the input image into a temperature adjustment signal, and combining global mean comparison, deviation direction judgment, and temperature boundary constraints, the sampling temperature of each token can accurately match the actual needs of its generation stage. This dynamic adjustment logic per token eliminates the one-size-fits-all limitation of fixed temperature on tokens of different generation stages and semantic qualities, allowing the temperature parameter to respond in real time to dynamic changes during the generation process, achieving a refined balance between determinism and diversity.

[0215] When the cross-modal consistency score is high, low sampling temperature amplifies the differences in word preference in the log-odds vector, causing the softmax probability distribution to focus on the optimal candidate word, effectively avoiding redundant exploration or semantic deviation that may occur under fixed temperature, and ensuring the accuracy and stability of the generated results. When the consistency score is low, high sampling temperature narrows the differences in word preference, broadens the range of probability distribution, encourages the model to explore more semantically related words, makes up for the information gap caused by visual feature extraction bias or poor text region quality, and reduces the risk of erroneous generation that may accumulate under fixed temperature.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0217] like Figure 4 As shown, embodiments of this application also provide a control device for the output lexical units of a pre-trained language model, the device comprising:

[0218] The acquisition module 401 is used to acquire the input image of the pre-trained language model and the corresponding output words of the input image;

[0219] The visual feature extraction module 402 is used to locate the text region in the input image and extract the visual features of the text region;

[0220] The calculation module 403 is used to calculate the cross-modal consistency score between the output word and the text region based on visual features and the hidden state vector of the pre-trained language model decoder; wherein, the hidden state vector is used to represent the semantic state when the pre-trained language model generates the output word; the cross-modal consistency score is used to represent the degree of semantic alignment between the output word and the text region.

[0221] The adjustment module 404 is used to adjust the sampling temperature of the pre-trained language model based on the cross-modal consistency score. The sampling temperature is used to control the randomness and diversity of the language model's output lexical units.

[0222] As an optional implementation provided in this application, the visual feature extraction module 402 is used to: perform region detection on the input image using a pre-trained text detection model to obtain the text region of the input image; perform feature extraction on the input image using a pre-trained visual model to obtain the global feature map of the input image; and extract the visual features of the text region from the global feature map using a region of interest alignment algorithm.

[0223] As an optional implementation provided in this application, the calculation module 403 is used to: perform average pooling on the visual features to obtain the visual feature representation of the text region; map the visual feature representation to the visual semantic vector of the text region through a linear projection layer; normalize the visual semantic vector and the latent state vector to obtain the standard visual semantic vector and the standard latent state vector; calculate the cosine similarity between the standard visual semantic vector and the standard latent state vector; and determine the cross-modal consistency score based on the cosine similarity.

[0224] As an optional implementation provided in this application, the calculation module 403 is used to: determine the region index of the text region corresponding to the maximum cosine similarity; the region index is used to visualize the association between the output word and the text region corresponding to the maximum cosine similarity; and the maximum cosine similarity is used as the cross-modal consistency score corresponding to the region index.

[0225] As an optional implementation provided in this application embodiment, the adjustment module 404 is used to: reduce the sampling temperature if the cross-modal consistency score is greater than the preset score; and increase the sampling temperature if the cross-modal consistency score is less than the preset score.

[0226] As an optional implementation provided in this application, the adjustment module 404 is used to: subtract the cross-modal consistency score from the global mean and take the absolute value; if the absolute value is greater than or equal to the consistency threshold, calculate the initial temperature; determine the sampling temperature based on the initial temperature, and the sampling temperature is within a preset temperature range.

[0227] As an optional implementation provided in this application, the adjustment module 404 is used to: determine that the initial temperature is less than the reference temperature when the cross-modal consistency score is greater than the global mean; and determine that the initial temperature is greater than the reference temperature when the cross-modal consistency score is less than the global mean.

[0228] As an optional implementation provided in this application, the adjustment module 404 is used to: determine that the sampling temperature is equal to the lower limit if the initial temperature is lower than the lower limit of the preset temperature range; determine that the sampling temperature is equal to the initial temperature if the initial temperature is higher than the lower limit and lower than the upper limit of the preset temperature range; and determine that the sampling temperature is equal to the upper limit if the initial temperature is higher than the upper limit of the temperature range.

[0229] As an optional implementation provided in this application, the device further includes a probability distribution calculation module, used for: adjusting the log-odds vector of the pre-trained language model based on the sampling temperature, wherein the log-odds vector corresponds to the output word, and each element in the log-odds vector is the original preference score of the pre-trained language model for the candidate word; and normalizing the adjusted log-odds vector to obtain the probability distribution output by the pre-trained language model.

[0230] As an optional implementation provided in this application, the probability distribution calculation module, when adjusting the log odds vector of the pre-trained language model based on the sampling temperature, is used to: divide each element in the log odds vector by the sampling temperature to obtain the adjusted log odds vector.

[0231] As an optional implementation provided in this application, the probability distribution calculation module, in the process of normalizing the adjusted log odds vector to obtain the probability distribution output by the pre-trained language model, is used to: calculate the exponential score of each element of the adjusted log odds vector; and calculate the probability distribution of the language model output lexical based on the exponential score of each element.

[0232] For a description of the features in the embodiment corresponding to the control device for the output lexical units of the pre-trained language model, please refer to the relevant description of the embodiment corresponding to the control method for the output lexical units of the pre-trained language model, which will not be repeated here.

[0233] like Figure 5 As shown, embodiments of this application also provide an electronic device, including a memory 501 and a processor 502. The memory 501 stores a computer program, and the processor 502 is configured to run the computer program to execute the steps in any of the above embodiments of the control method for outputting lexical units of a pre-trained language model.

[0234] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the control method for outputting lexical units of a pre-trained language model at runtime.

[0235] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0236] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the control method for outputting lexical units of a pre-trained language model.

[0237] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the control method for outputting lexical units of a pre-trained language model.

[0238] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0239] The foregoing has provided a detailed description of the method and electronic device for controlling the output lexical units of a pre-trained language model provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for controlling the output lexical units of a pre-trained language model, characterized in that, include: Obtain the input image for the pre-trained language model; Locate the text region in the input image and extract the visual features of the text region; Based on visual features and the hidden state vector of the pre-trained language model decoder, a cross-modal consistency score is calculated between the output word and the text region; wherein, the hidden state vector is used to characterize the semantic state when the pre-trained language model generates the output word corresponding to the input image; and the cross-modal consistency score is used to characterize the degree of semantic alignment between the output word and the text region. The cross-modal consistency score is subtracted from the global mean, and the absolute value is taken. If the cross-modal consistency score is greater than the global mean, the initial temperature is determined to be less than the reference temperature; if the cross-modal consistency score is less than the global mean, the initial temperature is determined to be greater than the reference temperature. If the initial temperature is lower than the lower limit of the preset temperature range, then the sampling temperature is determined to be equal to the lower limit of the preset temperature range; If the initial temperature is higher than the lower limit of the temperature range but lower than the upper limit of the preset temperature range, then the sampling temperature is determined to be equal to the initial temperature. If the initial temperature is higher than the upper temperature limit, then the sampling temperature is determined to be equal to the upper temperature limit; the sampling temperature is used to control the randomness and diversity of the language model output lexical units.

2. The method according to claim 1, characterized in that, The process of locating the text region in the input image and extracting the visual features of the text region includes: The input image is used to perform region detection using a pre-trained text detection model to obtain the text regions of the input image; The input image is used to extract features using a pre-trained visual model to obtain a global feature map of the input image; Visual features of the text region are extracted from the global feature map using a region of interest alignment algorithm.

3. The method according to claim 1, characterized in that, The calculation of the cross-modal consistency score between the output lexical unit and the text region based on visual features and the hidden state vector of the pre-trained language model decoder includes: Average pooling is performed on the visual features to obtain the visual feature representation of the text region; The visual feature representation is mapped to the visual semantic vector of the text region through a linear projection layer; The visual semantic vector and the hidden state vector are normalized to obtain standard visual semantic vector and standard hidden state vector. Calculate the cosine similarity between the standard visual semantic vector and the standard hidden state vector; The cross-modal consistency score is determined based on the cosine similarity.

4. The method according to claim 3, characterized in that, Determining the cross-modal consistency score based on the cosine similarity includes: Determine the region index of the text region corresponding to the maximum cosine similarity; the region index is used to visualize the association between the output word and the text region corresponding to the maximum cosine similarity. The maximum cosine similarity is used as the cross-modal consistency score corresponding to the region index.

5. The method according to claim 1, characterized in that, The method further includes: The log-odds vector of the pre-trained language model is adjusted based on the sampling temperature. The log-odds vector corresponds to the output lexical. Each element in the log-odds vector is the original preference score of the pre-trained language model for the candidate lexical. The adjusted log-odds vector is normalized to obtain the probability distribution output by the pre-trained language model.

6. The method according to claim 5, characterized in that, The normalization process of the adjusted logarithmic probability vector to obtain the probability distribution output by the pre-trained language model includes: For the adjusted log-odds vector, calculate the exponential score for each element; The probability distribution of the language model output lexical units is calculated based on the exponential score of each element.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the control method for outputting lexical units of a pre-trained language model as described in any one of claims 1 to 6 when executing the computer program.

Images