A confidence-weighted-based visual token robustness training method

Abstract

This invention, titled "A Robust Training Method for Visual Tokens Based on Confidence Weighting," belongs to the field of computer vision and multimodal large model technology. The technical problem it addresses is that existing technologies use OCR recognition results as hard labels for visual token training, leading to OCR errors that easily contaminate the visual codebook. Furthermore, OCR confidence is only used for filtering low-quality images, without embedding a loss function to dynamically adjust the supervision strength, resulting in poor robustness in complex industrial scenarios such as hydropower engineering drawings and insufficient accuracy in downstream multimodal task processing. The key technical points are: obtaining the OCR-recognized text and character-level confidence of the input image; calculating the supervision weights through a confidence weighting function; constructing a confidence-weighted KL divergence knowledge distillation loss, which is weighted and jointly combined with the reconstruction loss to obtain the total loss, and then training the visual discrete encoder end-to-end; and combining it with a semantic graph of hydropower engineering terminology and a triplet contrastive loss to enhance the industry-specific semantic expression.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal large model technology, specifically involving a robust training method for visual tokens based on confidence weighting. Background Technology

[0002] With the rapid development of multimodal large model technology, visual tokenization is a core foundational step in achieving cross-modal semantic alignment between images and text. This step uses visual encoders such as discrete variational autoencoders (dVAE) to convert continuous image visual features into discrete visual token sequences, enabling image information to be processed by large models in a text-like manner. This provides fundamental support for subsequent downstream tasks such as intelligent drawing recognition, technical terminology retrieval, and intelligent equipment status verification.

[0003] Currently, for visual tokenization training of images containing text regions, mainstream multimodal modeling solutions (such as BLIP and Flava) all employ OCR-assisted supervised training: an Optical Character Recognition (OCR) engine extracts the recognition results of text regions in the image, and the recognized technical terms are directly used as hard labels for the dVAE visual encoder, forcing the visual features of the corresponding image block to align with the textual semantics of the term, thereby achieving the binding of visual tokens to textual semantics. However, in practical industrial applications, especially in the processing of hydropower engineering drawings, the above-mentioned existing technologies have serious technical shortcomings: (1) Existing solutions directly use OCR recognition results as hard labels for supervised training without considering the error risks of OCR recognition. Hydropower engineering drawings often have problems such as blurry scanning, dirt and occlusion, low contrast, and uneven lighting, which can easily lead to OCR recognition errors. The incorrect recognition results will serve as incorrect supervision signals, directly polluting the semantic system of the visual codebook, resulting in semantic expression deviation of visual tokens, and ultimately significantly reducing the processing accuracy and reliability of downstream multimodal tasks; (2) Existing technologies make very low use of the confidence information output by the OCR engine. They only use the OCR confidence for filtering low-quality images and do not embed the confidence information into the loss function design of the training process. The character-level confidence output by the OCR engine is the core indicator reflecting the reliability of the recognition result. Existing solutions cannot dynamically adjust the strength of the OCR supervision signal based on this confidence information. They use the same supervision strength for high-reliability recognition results and low-reliability recognition results. They cannot strengthen the alignment effect of correct semantics through high-confidence results, nor can they suppress the semantic pollution of the visual codebook by low-confidence error results. In the end, the visual token model trained has extremely poor robustness in complex industrial scenarios and cannot meet the high-reliability application requirements of industrial scenarios such as hydropower engineering.

[0004] In view of this, the present invention is hereby proposed. Summary of the Invention

[0005] To address the aforementioned technical problems in existing technologies, this invention provides a robust training method for visual tokens based on confidence weighting. This method solves the overall technical problems in existing technologies, such as OCR recognition errors easily causing semantic pollution of the visual codebook, OCR confidence information not being effectively embedded in the training process to achieve dynamic adjustment of supervision intensity, resulting in poor robustness of the visual token model and insufficient accuracy and reliability of downstream multimodal task processing.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A confidence-weighted visual token robustness training method includes: S1. Obtain an input image containing a text region, perform OCR recognition on the text region of the input image to obtain the recognized text result, and the character-level confidence value corresponding to each recognized character in the recognized text result; S2. Based on the character-level confidence value, the supervision weight value corresponding to the recognized text result is calculated using a preset confidence weighting function; S3. Obtain the semantic embedding corresponding to the recognized text result, and the visual tag embedding obtained after visual feature extraction and discrete variational autoencoder encoding of the input image; S4. Based on the supervision weight values, construct a confidence-weighted knowledge distillation loss function; S5. The confidence-weighted knowledge distillation loss function is combined with the reconstruction loss function of the discrete variational autoencoder in a weighted manner to obtain the total loss function; S6. Based on the total loss function, perform end-to-end training on the visual discrete encoder to obtain the trained visual tokenized model.

[0007] Furthermore, in step S1, the text region of the input image is recognized by OCR through the PaddleOCR engine or a commercial OCR service; the character-level confidence value ranges from [0,1].

[0008] Further, in step S2, the expression for the confidence weighting function is:

[0009] in, To identify the confidence value corresponding to the text result, These are preset hyperparameters used to control the confidence decay rate.

[0010] Furthermore, the preset hyperparameter values ​​are preset based on the OCR recognition quality of the training dataset.

[0011] Furthermore, in step S4, the confidence-weighted knowledge distillation loss function uses KL divergence as the basic loss, and its expression is:

[0012] in, To identify the supervision weight values ​​corresponding to the text results, For the Kulbeck-Leibler divergence, and For the preset projection matrix, To identify the semantic embedding corresponding to the text results, The preset temperature coefficient, Embed the visual token corresponding to the input image.

[0013] Further, in step S5, the expression for the reconstruction loss function is:

[0014] in, For visual transformers, For image patches of the input image, The visual token is embedded as a result of encoding the corresponding image block using a discrete variational autoencoder.

[0015] Further, in step S5, the expression for the total loss function is:

[0016] in, and The preset loss weight coefficients, To reconstruct the loss function, The knowledge distillation loss function is weighted by confidence level.

[0017] Furthermore, in step S3, the visual features of the input image are extracted using the visual transformer ViT to obtain the visual features corresponding to each image block of the input image.

[0018] Furthermore, the input image containing the text area includes hydropower engineering drawings and industrial control panel images.

[0019] Furthermore, in step S6, the visual discrete encoder includes a visual feature extraction module and a discrete variational autoencoder module.

[0020] The beneficial effects of this invention are as follows: (1) Suppress semantic pollution: Use OCR confidence as dynamic supervision weight to reduce the semantic pollution of visual codebook by low-quality OCR recognition results from the root, and avoid semantic deviation of visual token caused by OCR error; (2) Adaptive supervised training: to achieve adaptive training of high-confidence OCR with strong supervision and low-confidence OCR with weak supervision, so that visual tokens can take into account both text semantic alignment and visual feature universality; (3) Improve scene robustness: make the model more stable in complex scenes such as image blurring, occlusion, and low contrast, and enhance the visual token's ability to adapt to harsh image conditions. (4) Optimize supervision and implementation: Make full use of OCR confidence information to improve the rationality of supervision signals; the training framework is an end-to-end integrable solution with no additional hardware / data dependencies, making engineering implementation simpler. Attached Figure Description

[0021] Figure 1 A flowchart of a visual token robustness training method provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0023] It should be noted that, unless otherwise specifically stated, the relative arrangement and numerical expressions of the components and steps described in these embodiments should not be construed as limiting the scope of the invention.

[0024] The following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail herein, but where applicable, such techniques, methods, and apparatus should be considered part of this specification.

[0025] Example See Figure 1 , Figure 1 This is a flowchart of a confidence-weighted visual token robustness training method proposed in this invention. Specific steps may include: S1. Data preprocessing and OCR recognition: Obtain an input image containing text regions, perform OCR recognition on the text regions of the input image to obtain the recognized text result, and the character-level confidence value corresponding to each recognized character in the recognized text result; The input image containing text regions is acquired. In this embodiment, the input images are a layout diagram of a pumped storage power station and an image of an industrial control panel. The PaddleOCR engine is used to perform character-by-character recognition on all text regions in the input image, outputting the recognized text result and the character-level confidence value corresponding to each recognized character. ,in .

[0026] For the identified professional terms composed of consecutive characters The average confidence score of all characters in the term is taken as the overall confidence score for that term. .

[0027] The input images containing text areas include hydropower engineering drawings and industrial control panel images.

[0028] S2. Confidence weight calculation: Based on the character-level confidence value, the supervision weight value corresponding to the recognized text result is calculated through a preset confidence weighting function; the confidence value is calculated as the sum of each character in the text string / the average of the number of characters. Based on the overall confidence score corresponding to the term The supervision weight value corresponding to the term is calculated using a preset confidence weighting function. The expression for the confidence weighting function is:

[0029] in, To identify the confidence value corresponding to the text result, These are preset hyperparameters used to control the confidence decay rate. In this embodiment, based on the average quality of OCR recognition from the hydropower engineering drawing dataset, the confidence decay rate is... The default value is 2; when When, corresponding to high-confidence, clear text regions, the supervision weight is close to 1; when At that time, for the low-confidence blurred / occluded text region, the supervision weight rapidly decays to below 0.25.

[0030] S3, Semantic Embedding and Visual Feature Extraction: Obtain the semantic embedding corresponding to the recognized text result, and the visual tag embedding obtained after visual feature extraction and discrete variational autoencoder encoding of the input image; Specifically, a semantic relationship map of terms in the field of hydropower engineering is constructed in advance based on the "Design Code for Pumped Storage Power Stations" NB / T10072. The map defines the inclusion relationship (subclass → parent class) and functional association relationship (functional chain) between terms. Based on this semantic relationship map, semantic embeddings corresponding to each professional term in the recognition text results are generated. .

[0031] Simultaneously, a visual transformer (ViT) is used to segment the input image and extract visual features, obtaining the visual features corresponding to each image block. These visual features are then input into a discrete variational autoencoder (dVAE) for encoding, yielding the visual tag embeddings for the corresponding image blocks. .

[0032] S4. Construct a confidence-weighted knowledge distillation loss function: Based on the supervision weight values, construct a confidence-weighted knowledge distillation loss function; The confidence-weighted knowledge distillation loss function uses KL divergence as the basic loss, and its expression is:

[0033] in, To identify the supervision weight values ​​corresponding to the text results, For the Kulbeck-Leibler divergence, and These are preset projection matrices, used to project semantic embeddings and visual mark embeddings onto the same dimensional space; To identify the semantic embedding corresponding to the text results, This is a preset temperature coefficient used to smooth the probability distribution; Embed the visual token corresponding to the input image.

[0034] Through this loss function, high-confidence OCR results will exert strong supervision on the visual encoder, forcing the visual tag embedding to align with the correct text semantics; low-confidence OCR results will only contribute weak gradients, avoiding the contamination of the visual codebook by erroneous semantics.

[0035] S5. Construct the total loss function: Combine the confidence-weighted knowledge distillation loss function with the reconstruction loss function of the discrete variational autoencoder to obtain the total loss function; The reconstruction loss function is used to constrain the reconstruction accuracy of visual features, and its expression is:

[0036] in, For visual transformers, For image patches of the input image, The visual token is embedded as a result of encoding the corresponding image block using a discrete variational autoencoder.

[0037] The expression for the total loss function is:

[0038] in, and The preset loss weight coefficients, To reconstruct the loss function, The knowledge distillation loss function is weighted by confidence level.

[0039] S6. End-to-end model training: Based on the total loss function, the visual discrete encoder is trained end-to-end to obtain the trained visual tokenized model; the visual discrete encoder includes a visual feature extraction module and a discrete variational autoencoder module.

[0040] After training, the tokens in the visual codebook are closely aligned with the correct industry-specific semantics under high-confidence OCR, and retain general visual features under low-confidence OCR. During inference, the visual token sequence generated by dVAE is highly robust to interference such as OCR errors, image blurring, and occlusion, which can effectively ensure the accuracy of downstream tasks such as intelligent drawing recognition, equipment status verification, and technical terminology retrieval.

[0041] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A confidence-weighted based visual token robustness training method, characterized in that, include: S1. Obtain an input image containing a text region, perform OCR recognition on the text region of the input image to obtain the recognized text result, and the character-level confidence value corresponding to each recognized character in the recognized text result; S2. Based on the character-level confidence value, the supervision weight value corresponding to the recognized text result is calculated using a preset confidence weighting function; S3. Obtain the semantic embedding corresponding to the recognized text result, and the visual tag embedding obtained after visual feature extraction and discrete variational autoencoder encoding of the input image; S4. Based on the supervision weight values, construct a confidence-weighted knowledge distillation loss function; S5. The confidence-weighted knowledge distillation loss function is combined with the reconstruction loss function of the discrete variational autoencoder in a weighted manner to obtain the total loss function; S6. Based on the total loss function, perform end-to-end training on the visual discrete encoder to obtain the trained visual tokenized model.

2. The confidence-weighted based visual token robustness training method according to claim 1, wherein, In step S1, the text region of the input image is recognized by OCR using the PaddleOCR engine or a commercial OCR service; the character-level confidence value ranges from [0,1].

3. The confidence-weighted visual token robustness training method according to claim 1, wherein, In step S2, the expression for the confidence weighting function is: in, To identify the confidence value corresponding to the text result, These are preset hyperparameters used to control the confidence decay rate.

4. The confidence-weighted visual token robustness training method according to claim 3, characterized in that, The preset hyperparameter values ​​are preset based on the OCR recognition quality of the training dataset.

5. The confidence-weighted visual token robustness training method according to claim 1, characterized in that, In step S4, the confidence-weighted knowledge distillation loss function uses KL divergence as the basic loss, and its expression is: in, To identify the supervision weight values ​​corresponding to the text results, For the Kulbeck-Leibler divergence, and For the preset projection matrix, To identify the semantic embedding corresponding to the text results, The preset temperature coefficient, Embed the visual token corresponding to the input image.

6. The confidence-weighted visual token robustness training method according to claim 1, characterized in that, In step S5, the expression for the reconstruction loss function is: in, For visual transformers, For image patches of the input image, The visual token is embedded as a result of encoding the corresponding image block using a discrete variational autoencoder.

7. The confidence-weighted visual token robustness training method according to claim 1, characterized in that, In step S5, the expression for the total loss function is: in, and The preset loss weight coefficients, To reconstruct the loss function, The knowledge distillation loss function is weighted by confidence level.

8. The confidence-weighted visual token robustness training method according to claim 1, characterized in that, In step S3, the visual features of the input image are extracted by the visual transformer ViT to obtain the visual features corresponding to each image block of the input image.

9. The confidence-weighted visual token robustness training method according to claim 1, characterized in that, The input images containing text areas include hydropower engineering drawings and industrial control panel images.

10. The confidence-weighted visual token robustness training method according to claim 1, characterized in that, In step S6, the visual discrete encoder includes a visual feature extraction module and a discrete variational autoencoder module.

Images