Target processing model training method and device, image processing method and device, equipment and medium

Abstract

The invention discloses a target processing model training method and device, an image processing method and device, equipment and a medium. The method comprises the steps of obtaining a target sample set; a first model is trained based on the first image sample set to obtain a second model, the first model comprises a first to-be-trained image encoder, a second target image encoder and a to-be-trained decoder, and the second model comprises a first target image encoder, a second target image encoder and a target decoder; the size of an image input by the first to-be-trained image encoder is larger than that of an image input by the second target image encoder; generating a to-be-trained processing model according to the first target image encoder, the second target image encoder and the to-be-trained natural language model; the to-be-trained processing model is trained according to the prompt word set, the first image sample set and the second image sample set, and the target processing model is obtained.By means of the technical scheme, the training efficiency and precision of the model can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a target processing model training method, image processing method, apparatus, device and medium. Background Technology

[0002] Text images (including but not limited to scanned documents, PPTs, Word documents, PDFs, etc.) are important carriers of information and a crucial input source for AI applications (such as chatbot applications). To convert text images into a form that computers can understand, a combination of multiple models, such as Optical Character Recognition (OCR) and Key Information Extraction (KIE), is typically required. This leads to the following problems:

[0003] OCR, as a pre-processing model, extracts characters embedded in an image into an editable format. It typically includes element detection, region cropping, and character recognition. Each module is prone to getting trapped in local optima, resulting in high maintenance costs.

[0004] The KIE model typically requires pre-defining the scope of key information in order to function effectively, which severely restricts its usability. Summary of the Invention

[0005] This invention provides a target processing model training method, an image processing method, an apparatus, a device, and a medium to solve at least one of the above-mentioned problems.

[0006] According to one aspect of the present invention, a method for training a target processing model is provided, comprising:

[0007] Obtain a target sample set, wherein the target sample set includes a first image sample set and a second image sample set, wherein the image samples in the first image sample set carry labels, and the labels include: text information and image type;

[0008] The first model is trained based on the first image sample set to obtain the second model. The first model includes a first image encoder to be trained, a second target image encoder, and a decoder to be trained. The second model includes a first target image encoder, a second target image encoder, and a target decoder. The image size input to the first image encoder to be trained is larger than the image size input to the second target image encoder.

[0009] Based on the first target image encoder, the second target image encoder, and the natural language model to be trained, a training processing model is generated.

[0010] The target processing model is obtained by training the processing model based on the prompt word set, the first image sample set, and the second image sample set.

[0011] According to another aspect of the present invention, an image processing method is provided, comprising:

[0012] Obtain the image to be processed and the target prompt word, wherein the target prompt word is any prompt word in the prompt word set;

[0013] The image to be processed and the target prompt word are input into the target processing model to obtain the processing result corresponding to the target prompt word. The target processing model is trained based on the target processing model training method described in any embodiment.

[0014] According to another aspect of the present invention, a target processing model training apparatus is provided, the target processing model training apparatus comprising:

[0015] The first acquisition module is used to acquire a target sample set, wherein the target sample set includes a first image sample set and a second image sample set, wherein the image samples in the first image sample set carry labels, and the labels include: text information and image type;

[0016] A first training module is used to train a first model based on the first image sample set to obtain a second model. The first model includes a first image encoder to be trained, a second target image encoder, and a decoder to be trained. The second model includes a first target image encoder, a second target image encoder, and a target decoder. The image size input to the first image encoder to be trained is larger than the image size input to the second target image encoder.

[0017] The training model generation module is used to generate a training model based on the first target image encoder, the second target image encoder, and the natural language model to be trained.

[0018] The second training module is used to train the processing model to be trained based on the prompt word set, the first image sample set, and the second image sample set to obtain the target processing model.

[0019] According to another aspect of the present invention, an image processing apparatus is provided, the image processing apparatus comprising:

[0020] The second acquisition module is used to acquire the image to be processed and the target prompt word, wherein the target prompt word is any prompt word in the prompt word set;

[0021] The image processing module is used to input the image to be processed and the target prompt word into the target processing model to obtain the processing result corresponding to the target prompt word. The target processing model is trained based on the target processing model training method described in the embodiments of the present invention.

[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0023] At least one processor; and

[0024] A memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target processing model training method or the image processing method according to any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the target processing model training method or the image processing method according to any embodiment of the present invention.

[0027] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements a target processing model training method or an image processing method as described in any of the embodiments of the present invention.

[0028] This invention provides an embodiment that obtains a target sample set; trains a first model based on the first image sample set to obtain a second model. The first model includes a first image encoder to be trained, a second target image encoder, and a decoder to be trained. The second model includes a first target image encoder, a second target image encoder, and a target decoder. The image size input to the first image encoder to be trained is larger than the image size input to the second target image encoder. A processing model to be trained is generated based on the first target image encoder, the second target image encoder, and the natural language model to be trained. The processing model to be trained is then trained based on a set of prompt words, the first image sample set, and the second image sample set to obtain a target processing model. This approach improves the training efficiency and accuracy of the model.

[0029] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0030] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart of a target processing model training method according to an embodiment of the present invention;

[0032] Figure 2 This is a flowchart of another target processing model training method in an embodiment of the present invention;

[0033] Figure 3 This is a flowchart of an image processing method according to an embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the structure of a target processing model training device according to an embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the structure of an image processing device according to an embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0040] Example 1

[0041] Figure 1 This is a flowchart illustrating a target processing model training method provided in an embodiment of the present invention. This embodiment is applicable to target processing model training. The method can be executed by the target processing model training device in this embodiment of the present invention, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0042] S110, Obtain the target sample set.

[0043] The target sample set includes a first image sample set and a second image sample set. The image samples in the first image sample set carry labels, which include text information and image type.

[0044] S120, the first model is trained based on the first image sample set to obtain the second model.

[0045] The first model includes a first image encoder to be trained, a second target image encoder, and a decoder to be trained. The second model includes a first target image encoder, a second target image encoder, and a target decoder. The image size input to the first image encoder to be trained is larger than the image size input to the second target image encoder.

[0046] In this embodiment, the first model further includes a fully connected layer to be trained and an input layer. For example, the input terminals of the first image encoder to be trained and the second target image encoder are both connected to the input layer, the output terminals of the first image encoder to be trained and the second target image encoder are both connected to the fully connected layer to be trained, and the fully connected layer to be trained is connected to the decoder to be trained.

[0047] In this embodiment, the method for training the first model based on the first image sample set to obtain the second model can be as follows: input image samples from the first image sample set into the first model to obtain predicted text information and predicted image type; based on the differences between the predicted text information and the text information carried by the image samples, and the differences between the predicted image type and the image type carried by the image samples, train the parameters of the first model to obtain the second model. It should be noted that during the training of the first model, the parameters of the second target image encoder in the first model remain unchanged; only the parameters of the first image encoder to be trained and the decoder to be trained are adjusted.

[0048] In this embodiment, the image samples in the first image sample set are scaled to obtain image samples of a first size and image samples of a second size, where the first size is larger than the second size. The image samples of the first size are input into a first image encoder to be trained, and the image samples of the second size are input into a second target image encoder. The outputs of the first image encoder to be trained and the outputs of the second target image encoder are input into a decoder to be trained to obtain predicted text information and predicted image type. The first size can be 1024×1024, and the second size can be 224×224.

[0049] Because existing solutions cannot perceive low-level features, the conventional approach to ensure that the encoder can perceive low-level features is to increase the input scale of the image. However, this approach does not fundamentally solve the problem because the large-scale model used is primarily for inference tasks and biased towards semantic extraction, resulting in insufficient character perception capabilities. Therefore, this invention, while retaining high-level semantic features, adds low-level features. Regardless of the original scale of the input image, it is uniformly scaled to two scales: 1024×1024 and 224×224. The larger scale is used as the input to image encoder 1, and the smaller scale is used as the input to image encoder 2. Image encoder 1 uses a pre-trained model focused on visual perception (such as the Swin-Transformer series) to capture low-level features. Image encoder 2 uses a CLIP model image encoder, which is frozen throughout the process and does not participate in training, to capture high-level semantic expressions. The outputs of image encoder 1 and image encoder 2 are fused to adapt to the subsequent decoder. Fusion methods include, but are not limited to, concatenation and self-attention. To adapt to the input scale of the subsequent decoder, a dimensionality transformation is required through a fully connected layer. To ensure computational efficiency, concatenation is preferred as the fusion method. S130, generate a training processing model based on the first target image encoder, the second target image encoder, and the natural language model to be trained.

[0050] In this embodiment, since the natural language model to be trained focuses on the understanding of text images, the model size of the natural language model to be trained can range from 2B to 671B. Considering the costs of training, deployment, and inference, it is generally appropriate to choose around 2B. For example, the natural language model to be trained can be a DeepSeek 1.5B model.

[0051] In this embodiment, the processing model to be trained includes, from input to output, an input layer, an encoder, a fully connected layer, and a natural language model to be trained. The encoder includes a first target image encoder and a second target image encoder. The input ends of the first target image encoder and the second target image encoder are both connected to the output end of the input layer. The output ends of the first target image encoder and the second target image encoder are both connected to the input end of the fully connected layer. The output end of the fully connected layer is connected to the input end of the natural language model to be trained.

[0052] S140, the processing model to be trained is trained according to the prompt word set, the first image sample set and the second image sample set to obtain the target processing model.

[0053] Optionally, the set of prompt words includes: text recognition prompt words, layout prompt words, and structured prompt words;

[0054] The target processing model is trained based on the prompt word set, the first image sample set, and the second image sample set, including:

[0055] The image samples in the second image sample set are input into the text recognition model to obtain the text recognition results and confidence scores corresponding to each image sample in the second image sample set.

[0056] The second image sample set is filtered based on the confidence level of each image sample in the second image sample set to obtain the first training sample set.

[0057] Wherein, the confidence level of the image samples in the first training sample set is greater than the first confidence threshold.

[0058] The image samples and text recognition prompts from the first training sample set are input into the model to be trained to obtain the predicted text recognition results.

[0059] A first score is determined based on the predicted text recognition result and the text recognition result corresponding to the image sample.

[0060] In this embodiment, the method for determining the first score based on the predicted text recognition result and the text recognition result corresponding to the image sample can be as follows: obtain the length of the longest common substring of the predicted text recognition result and the text recognition result corresponding to the image sample, and the length of the string of the text recognition result corresponding to the image sample; and use the ratio of the length of the longest common substring to the length of the string of the text recognition result corresponding to the image sample as the first score.

[0061] The image samples from the second image sample set are input into multiple layout analysis modules to obtain the layout analysis results, confidence levels, and crossover ratios corresponding to each layout analysis model.

[0062] Based on the layout analysis results, confidence level, and crossover ratio corresponding to each layout analysis model, the image samples in the second image sample set are filtered to obtain the second training sample set.

[0063] In this case, the confidence level of the image samples in the second training sample set is greater than the second confidence level threshold, and the cross-union ratio is greater than the cross-union ratio threshold.

[0064] In this embodiment, based on the layout analysis results, confidence level, and intersection-union ratio (IOU) corresponding to each layout analysis model, the image samples in the second image sample set are filtered, and only the results with the same category, high confidence level (>= second confidence level threshold), and intersection-union ratio (IOU) >= intersection-union ratio threshold are retained.

[0065] In this embodiment, a set (>=2) of layout analysis models performs layout analysis on the same set of images (these images are obtained by adding noise to the same image), retaining only the results with the same category, high confidence (>=0.7), and IOU>=0.8. (IOU is a method to measure whether two rectangles are in the same position), and it is represented as {cls_t,pos_t} (cls_t represents the category, and pos_t represents the position coordinates).

[0066] The image samples and layout prompts from the second training sample set are input into the model to be trained to obtain the predicted layout analysis results.

[0067] The second score is determined based on the predicted layout analysis results and the layout analysis results corresponding to the image samples.

[0068] In this embodiment, the method for determining the second score based on the predicted layout analysis results and the layout analysis results corresponding to the image samples can be as follows: obtain the sum of the distance intersection-union ratios (DIU) of the predicted position coordinates and the target position coordinates corresponding to the same category in the predicted layout analysis results and the layout analysis results corresponding to the image samples; and use the ratio of the sum of the distance intersection-union ratios to the logarithm of the predicted layout information included in the predicted layout analysis results as the second score.

[0069] The image samples from the second image sample set are input into the structure recognition model to obtain the structure recognition result.

[0070] A relationship tree is generated based on the structure recognition results.

[0071] The relationship tree includes a root node and leaf nodes.

[0072] The image samples and structured prompts from the second image sample set are input into the processing model to be trained to obtain the predicted layout analysis results.

[0073] Based on the relationship tree and the analysis results of the predicted layout, a third score is determined.

[0074] In this embodiment, the method for determining the third score based on the relationship tree and the predicted layout analysis results can be as follows: generate a predicted relationship tree based on the predicted layout analysis results; determine the third score based on the number of nodes in the predicted relationship tree, the number of nodes in the relationship tree, and the number of matching nodes in the predicted relationship tree, wherein the matching node is a node in the predicted relationship tree that matches a node in the relationship tree.

[0075] The parameters of the processing model to be trained are trained based on the first score, the second score, and the third score to obtain the target processing model.

[0076] In this embodiment, the target processing model is obtained by training the parameters of the processing model to be trained based on the first score, the second score, and the third score.

[0077] Optionally, a first score is determined based on the predicted text recognition result and the text recognition result corresponding to the image sample, including:

[0078] Obtain the length of the longest common substring of the predicted text recognition result and the text recognition result corresponding to the image sample, and the length of the string of the text recognition result corresponding to the image sample.

[0079] The ratio of the length of the longest common substring to the length of the string in the text recognition result corresponding to the image sample is used as the first score.

[0080] In this embodiment, the image is first input into various text recognition models (such as OCR models). The output of the text recognition model has a confidence score, indicating the reliability of the current result. Then, low-confidence results are filtered out, and the corresponding image is used as input for the reinforcement learning stage. The input of DeepSeek1.5B is obtained, and the evaluation is performed as follows: str_ocr represents the output of the text recognition model, and str_o represents the output of the model to be trained: calculate all the longest common substrings of str_ocr and str_o, and count the number of characters A; calculate the number of characters B of str_ocr; score1 = A / B, and its value is greater than 0 and less than or equal to 1.

[0081] Optionally, the predicted layout analysis results include: multiple pairs of predicted layout information, each pair of predicted layout information including: category and predicted position coordinates corresponding to the category; the layout analysis results corresponding to the image sample include: multiple pairs of target layout information, each pair of target layout information including: category and target position coordinates corresponding to the category.

[0082] Based on the predicted layout analysis results and the layout analysis results corresponding to the image samples, a second score is determined, including:

[0083] Obtain the sum of the distances between the predicted position coordinates and the target position coordinates corresponding to the same category in the predicted layout analysis results and the layout analysis results corresponding to the image samples.

[0084] The second score is the ratio of the sum of the intersection and union of the distances to the logarithm of the predicted layout information included in the predicted layout analysis results.

[0085] In this embodiment, the corresponding image (before noise is added) is input into the model to be trained, and the output {cls_0,pos_0} is obtained. The calculation logic of the evaluation function is as follows: count the number of outputs of this model, that is, how many {cls_0,pos_0} there are, denoted as A_layout; count the number of cls_o == cls_t, and obtain the corresponding pos_0,pos_t; calculate the DIOU (an algorithm for measuring two rectangular boxes, which comprehensively considers overlap, center position, and aspect ratio) of the corresponding pos_0 and pos_t, and score2 = sum of all DIOU values ​​ / A_layout.

[0086] However, the above operation still has a drawback: because only the portion with high confidence is retained in {cls_t,pos_t}, a portion of the layout analysis model's output results will be filtered out, and in extreme cases, all results will be filtered out. In this situation, even if the correct result is obtained after inputting the image into the model to be trained, the calculated score2=0 is obviously unreasonable.

[0087] To address this issue, all images are first processed through the layout analysis model to obtain all {CLS_T, POS_T} values. Then, a portion of the results is randomly sampled and reassembled into a new image. This results in a set of training data for layout analysis (images + corresponding annotations), denoted as Data_layout. This data is then used as input for reinforcement learning. Because the corresponding {cls_t, pos_t} values ​​for this new data are known, it is unnecessary to process it through the layout analysis model again.

[0088] Optionally, based on the relationship tree and the predicted layout analysis results, a third score is determined, including:

[0089] Based on the analysis results of the predicted layout, a prediction relationship tree is generated.

[0090] The third score is determined based on the number of nodes in the predicted relationship tree, the number of nodes in the relationship tree, the number of matching nodes in the predicted relationship tree, the value of the nodes in the predicted relationship tree, and the value of the nodes in the relationship tree.

[0091] Wherein, the matching node is a node in the predicted relation tree that matches a node in the relation tree.

[0092] In this embodiment, the method for determining the third score based on the number of nodes in the predicted relationship tree, the number of nodes in the relationship tree, the number of matching nodes in the predicted relationship tree, the value of the nodes in the predicted relationship tree, and the value of the nodes in the relationship tree can be as follows: determine the node score based on the number of nodes in the predicted relationship tree, the number of nodes in the relationship tree, and the number of matching nodes in the predicted relationship tree; determine the node value score based on the value of the nodes in the predicted relationship tree, the value of the nodes in the relationship tree, the number of nodes in the predicted relationship tree, and the number of nodes in the relationship tree; and take the average of the node score and the node value score as the third score.

[0093] In this embodiment, the method for determining the node score based on the number of nodes in the predicted relation tree, the number of nodes in the relation tree, and the number of matching nodes in the predicted relation tree can be as follows:

[0094] a1=2×(total_match / total_node_o)×(total_match / total_node_gt);

[0095] a2=(total_match / total_node_o)+(total_match / total_node_gt);

[0096] score3_1 = a1 / a2;

[0097] Where total_node_gt is the number of nodes in the relation tree, total_node_o is the number of nodes in the predicted relation tree, and total_match is the number of matching nodes in the predicted relation tree (it iterates through each node in tree_o, and if the node can find a matching node in tree_gt, it is denoted as node_match).

[0098] In this embodiment, the method for determining the node value score based on the node value in the predicted relation tree, the node value in the relation tree, the number of nodes in the predicted relation tree, and the number of nodes in the relation tree can be as follows:

[0099] ;

[0100] b2=(total_match2 / total_node_o)+(total_match2 / total_node_gt);

[0101] `score3_2 = b1 / b2`. Here, `total_match2` is determined based on the similarity between `str1` and `str3`. `str1` is the text sequence obtained by traversing upwards from the current node to the root node and downwards to the last leaf node. Using the value of the current node as the target, all nodes with the same value are found in `tree_gt`. If no matching node is found, the current iteration ends. If a matching node is found, it is denoted as `node_temp`. Traversing upwards from `node_temp` to the root node and downwards to the last leaf node yields `str2`. If a set of `node_temp` nodes also contains a corresponding `str2` node, the node most similar to `str1` is selected using edit distance. The most similar node is denoted as `str3`.

[0102] In this embodiment, as Figure 2As shown, the model training process is divided into two stages: supervised learning and reinforcement learning. After the supervised learning stage, the model is directly reused in the reinforcement learning stage. Supervised learning stage: Because existing solutions cannot perceive low-level features, the conventional approach to ensure the encoder can perceive low-level features is to increase the input scale of the image. However, this approach does not fundamentally solve the problem because the large-scale model used is primarily for inference tasks and biased towards semantic extraction, resulting in insufficient character perception capabilities. Based on this, this embodiment of the invention adds low-level features while retaining high-level semantic features. Regardless of the original scale of the input image, it is uniformly scaled to two scales: 1024×1024 and 224×224. The larger scale is used as the input of image encoder 1, and the smaller scale is used as the input of image encoder 2. Image encoder 1 uses a pre-trained model focused on visual perception (such as the Swin-Transformer series) to capture low-level features. Image encoder 2 uses the CLIP model image encoder, which is frozen throughout the process and does not participate in training, to capture high-level semantic expressions. The outputs of image encoder 1 and image encoder 2 are fused to adapt to the subsequent decoder. Fusion methods include, but are not limited to, splicing and self-attention. To adapt to the input scale of the subsequent decoder, a dimensionality transformation is required through fully connected layers. To ensure computational efficiency, a concatenation fusion method is preferred. To enhance visual perception capabilities, the decoder does not use the inference model mode commonly used in LLMs, but instead opts for a BERT architecture design with fewer parameters, such as the OPT125M model. Its output has two forms: text information and image type. Text information consists of all the text contained in the image, while image types can include scanned documents, PPTs, natural scenes, non-text images, invoices, contracts, charts, tables, and plain text. The text information output prompts the network to learn low-level features, while the image type prompts the network to learn high-level semantic features. During this training process, open-source and synthetic data can be used for supervised training to ensure the image encoder has the ability to capture low-level features. After this stage, the model gains visual perception capabilities through text recognition tasks, and its high-level semantic extraction capabilities are enhanced through classification tasks. This lays a solid foundation for the next stage of reinforcement learning. The first half of the reinforcement learning phase is the same as the supervised learning phase, except that the decoder is replaced with the natural language model to be trained (e.g., DeepSeek 1.5B), and a natural language description instruction task is added. The purpose of using reinforcement learning to replace supervised learning is to leverage the powerful task understanding capabilities of the DeepSeek series, and to optimize the model using its proposed Group Relative Policy Optimization (GPRO) reinforcement learning algorithm.The key to reinforcement learning lies in the design of the reward function, which is used during the training phase to evaluate the quality of the output and adjust the model accordingly to produce better results. Typically, the reward function is a powerful model; for example, in the natural language processing domain, DeepSeekV3 can be used, where the output of DeepSeek 1.5B is input into DeepSeekV3 to generate an evaluation score, and the GPRO algorithm is used for model optimization. However, the most practical problem is that for text-image understanding tasks, there are no powerful models available. To address this issue, this invention provides the following evaluation function:

[0103] Evaluation Function 1: Text Recognition (Implicit Reading Order):

[0104] Currently, numerous text recognition models exist, but they are typically designed for specific tasks, such as handwriting recognition or special symbol recognition. However, the output of these models can be used to aid in evaluation. Specifically, the following steps are taken: First, input the image into various text recognition models (such as OCR models). The output of each text recognition model will have a confidence score, indicating the reliability of the current result. Then, filter out results with low confidence scores. Use the corresponding image as input for the reinforcement learning stage, obtaining the input from DeepSeek 1.5B. The evaluation is performed as follows: str_ocr represents the output of the text recognition model, and str_o represents the output of the model to be trained: Calculate all the longest common substrings of str_ocr and str_o, and count the number of characters A; calculate the number of characters B in str_ocr; score1 = A / B, whose value is greater than 0 and less than or equal to 1. Note that if the data itself is labeled, it can be used directly.

[0105] Evaluation Function 2: Page Layout Analysis

[0106] Similar to text recognition, existing layout analysis models are used to process the data and obtain results. These results, along with the output of the model to be trained, are then fed into an evaluation function for scoring. The output of the layout analysis model is {cls, pos=x1,y1,x2,y2} (cls represents the category, such as title, plain text, chart, etc., and pos=x1,y1,x2,y2 represent the position coordinates of that category on the original image, i.e., the coordinates of the top left and bottom right corners). To ensure the reliability of the layout analysis model's output, a multi-model prediction approach is used, as follows:

[0107] A set of (>=2) layout analysis models performs layout analysis on the same set of images (these images are obtained by adding noise to the same image). Only the results with the same category, high confidence (>=0.7), and IOU>=0.8 are retained. (IOU is a method to measure whether two bounding boxes are in the same position), and it is represented as {cls_t,pos_t} (cls_t represents the category, and pos_t represents the position coordinates).

[0108] The corresponding image (before noise is added) is input into the model to be trained, and the output {cls_0,pos_0} is obtained. The calculation logic of the evaluation function is as follows: count the number of outputs of this model, that is, how many {cls_0,pos_0} there are, denoted as A_layout; count the number of cls_o == cls_t, and obtain the corresponding pos_0,pos_t; calculate the DIOU (an algorithm for measuring two rectangles, which comprehensively considers overlap, center position, and aspect ratio) of the corresponding pos_0 and pos_t, and score2 = sum of all DIOU values ​​ / A_layout.

[0109] However, the above operation still has a drawback: because only the portion with high confidence is retained in {cls_t,pos_t}, a portion of the layout analysis model's output results will be filtered out, and in extreme cases, all results will be filtered out. In this situation, even if the correct result is obtained after inputting the image into the model to be trained, the calculated score2=0 is obviously unreasonable.

[0110] To address this issue, all images are first processed through the layout analysis model to obtain all {CLS_T, POS_T} values. Then, a portion of the results is randomly sampled and reassembled into a new image. This results in a set of training data for layout analysis (images + corresponding annotations), denoted as Data_layout. This data is then used as input for reinforcement learning. Because the corresponding {cls_t, pos_t} values ​​for this new data are known, it is unnecessary to process it through the layout analysis model again. It's important to note that if the data itself is already labeled, it can be used directly.

[0111] Evaluation Function 3: Structured:

[0112] Structured data is mainly used in two aspects: one is a key-value structure, such as a bill, and the other is an Excel spreadsheet structure.

[0113] In the current solution, the structured output format differs significantly for these two types. For example, for invoices, the structured output format is a JSON string: {"Full Name":"Hengda Landscape Group Co., Ltd.", "Account Number":"82100",...}

[0114] For Excel spreadsheets, the format is generally Markdown, typically represented as:

[0115] |Name|Gender|

[0116] |----|----|

[0117] Zhang San | Male |

[0118] Zhang Si | Female |

[0119] As can be seen, there are obvious differences between the two, making them inconvenient to handle uniformly. To solve the above problems, this embodiment of the invention designs a tree structure for unified representation, with the table title as the root node and all others as leaf nodes, which can also have leaf nodes.

[0120] The logic for calculating the subsequent evaluation function is relatively straightforward. Tree_gt represents the true structure, tree_o represents the output of the model to be trained, and score3 = similarity function (tree_gt, tree_o).

[0121] The similarity calculation method consists of two parts:

[0122] The first part ignores the specific content of each node, assigning node numbers from top to bottom and left to right, focusing only on whether nodes match. That is, it only considers the structure itself. The calculation logic is as follows: Count the total number of nodes in `tree_gt`, denoted as `total_node_gt`; count the total number of nodes in `tree_o`, denoted as `total_node_o`; traverse each node in `tree_o` sequentially. If a matching node can be found in `tree_gt`, it is recorded as `node_match`. The matching criterion is to traverse upwards from the current node to the root node and downwards to the last leaf node. Because nodes are numbered, a sequence of numbers is obtained. Assuming the current node is numbered 3, the tree structure has only 3 levels, its parent node is numbered 0, and the leaf node is numbered 5, then this sequence of numbers is (0-3-5). If the two sequences of numbers are the same, it indicates a match; otherwise, it does not match. Calculate the total number of `node_match`, denoted as `total_match`.

[0123] a1=2×(total_match / total_node_o)×(total_match / total_node_gt);

[0124] a2=(total_match / total_node_o)+(total_match / total_node_gt);

[0125] score3_1 = a1 / a2.

[0126] The second part needs to consider the specific values ​​of the nodes, but does not require numbering the nodes. The calculation logic is as follows: Count the number of all nodes in tree_gt, denoted as total_node_gt2; Count the number of all nodes in tree_o, denoted as total_node_o2; Set total_match2=0.0; Iterate through each node in tree_o, find the most matching node in tree_gt, and calculate the matching value. The calculation logic for the matching value is as follows:

[0127] 1) Traverse upwards from the current node to the root node, and downwards to the last leaf node to obtain a text sequence str1.

[0128] 2) Using the current node's value as the target, find all nodes in tree_gt with the same value. If none are found, end the current loop. If found, mark it as node_temp. Traverse upwards from node_temp until the root node, and downwards until the last leaf node, obtaining a text sequence str2.

[0129] 3) If a set of node_temp is obtained in 2), and the corresponding str2 is also a set, then select the set that is most similar to str1. The similarity algorithm uses edit distance. The most similar one is denoted as str3. In some cases, str3 == str2.

[0130] 4) Calculate the similarity between str1 and str3 using edit distance and normalize it to the range of 0-1, denoted as node_sim;

[0131] total_match2 += node_sim;

[0132] ;

[0133] b2=(total_match2 / total_node_o)+(total_match2 / total_node_gt);

[0134] score3_2 = b1 / b2.

[0135] The final total score3 = (score3_1 + score3_2) / 2. Different weightings can also be set, because the averaging method is equivalent to all weights being 0.5.

[0136] In summary, the final evaluation score for this invention is:

[0137] score = a × score1 + b × score2 + c × score3, where a, b, and c are weighting coefficients, all of which are 1 by default.

[0138] The technical solution of this embodiment obtains a target sample set; trains a first model based on a first image sample set to obtain a second model, wherein the first model includes a first image encoder to be trained, a second target image encoder, and a decoder to be trained, and the second model includes a first target image encoder, a second target image encoder, and a target decoder, wherein the image size input to the first image encoder to be trained is larger than the image size input to the second target image encoder; a processing model to be trained is generated based on the first target image encoder, the second target image encoder, and the natural language model to be trained; and the processing model to be trained is trained based on the prompt word set, the first image sample set, and the second image sample set to obtain a target processing model, which can improve the training efficiency and accuracy of the model.

[0139] Example 2

[0140] Figure 3 This is a flowchart illustrating an image processing method provided in an embodiment of the present invention. This embodiment is applicable to image processing situations. The method can be executed by an image processing device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 3 As shown, the method specifically includes the following steps:

[0141] S310, acquire the image to be processed and the target prompt word.

[0142] The target prompt word is any prompt word in the prompt word set. That is, the target prompt word is any prompt word selected from text recognition prompt words, layout prompt words, and structured prompt words.

[0143] S320, input the image to be processed and the target prompt word into the target processing model to obtain the processing result corresponding to the target prompt word.

[0144] The target processing model is trained based on the target processing model training method described in the above embodiments.

[0145] In this embodiment, if the target prompt is a text recognition prompt, the processing result corresponding to the target prompt is the text recognition result; if the target prompt is a layout prompt, the processing result corresponding to the target prompt is the layout recognition result; if the target prompt is a structured prompt, the processing result corresponding to the target prompt is the structured recognition result.

[0146] The technical solution of this embodiment obtains an image to be processed and a target prompt word, wherein the target prompt word is any prompt word in the prompt word set; inputs the image to be processed and the target prompt word into a target processing model to obtain the processing result corresponding to the target prompt word, which can improve the efficiency and accuracy of the obtained processing result.

[0147] Example 3

[0148] Figure 4 This is a schematic diagram of a target processing model training device provided in an embodiment of the present invention. This embodiment is applicable to target processing model training. The device can be implemented using software and / or hardware, and can be integrated into any device that provides target processing model training functionality, such as… Figure 4 As shown, the target processing model training device specifically includes: a first acquisition module 410, a first training module 420, a processing model generation module 430 to be trained, and a second training module 440.

[0149] The first acquisition module is used to acquire a target sample set, wherein the target sample set includes a first image sample set and a second image sample set, wherein the image samples in the first image sample set carry labels, and the labels include: text information and image type;

[0150] A first training module is used to train a first model based on the first image sample set to obtain a second model. The first model includes a first image encoder to be trained, a second target image encoder, and a decoder to be trained. The second model includes a first target image encoder, a second target image encoder, and a target decoder. The image size input to the first image encoder to be trained is larger than the image size input to the second target image encoder.

[0151] The training model generation module is used to generate a training model based on the first target image encoder, the second target image encoder, and the natural language model to be trained.

[0152] The second training module is used to train the processing model to be trained based on the prompt word set, the first image sample set, and the second image sample set to obtain the target processing model.

[0153] The above-described products can perform the methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects for performing the methods.

[0154] Example 4

[0155] Figure 5This is a schematic diagram of an image processing apparatus provided in an embodiment of the present invention. This embodiment is applicable to image processing applications. The apparatus can be implemented using software and / or hardware, and can be integrated into any device that provides image processing functionality, such as… Figure 5 As shown, the image processing device specifically includes a second acquisition module 510 and an image processing module 520.

[0156] The second acquisition module is used to acquire the image to be processed and the target prompt word, wherein the target prompt word is any prompt word in the prompt word set;

[0157] The image processing module is used to input the image to be processed and the target prompt word into the target processing model to obtain the processing result corresponding to the target prompt word. The target processing model is trained based on the target processing model training method provided in the embodiments of the present invention.

[0158] Example 5

[0159] Figure 6 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0160] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0161] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0162] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as target processing model training methods, or image processing methods.

[0163] In some embodiments, the target processing model training method, or the image processing method, may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the target processing model training method, or the image processing method, described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the target processing model training method, or the image processing method, by any other suitable means (e.g., by means of firmware).

[0164] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0165] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0166] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0167] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0168] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0169] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0170] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0171] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the target processing model training method or the image processing method according to any embodiment of the invention.

[0172] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0173] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training a target processing model, characterized in that, The method comprises the following steps: acquiring a target sample set, wherein the target sample set comprises a first image sample set and a second image sample set, an image sample in the first image sample set carries a label, and the label comprises text information and an image type; training a first model based on the first image sample set to obtain a second model, wherein the first model comprises a first to-be-trained image encoder, a second target image encoder, and a to-be-trained decoder, the second model comprises a first target image encoder, a second target image encoder, and a target decoder, and an image size input to the first to-be-trained image encoder is greater than an image size input to the second target image encoder; generating a to-be-trained processing model according to the first target image encoder, the second target image encoder, and a to-be-trained natural language model; training the to-be-trained processing model according to a prompt word set, the first image sample set, and the second image sample set to obtain a target processing model.

2. The method of claim 1, wherein, The prompt word set comprises a text recognition prompt word, a layout prompt word, and a structured prompt word; training the to-be-trained processing model according to the prompt word set, the first image sample set, and the second image sample set to obtain the target processing model comprises: inputting an image sample in the second image sample set into a text recognition model to obtain a text recognition result and a confidence corresponding to each image sample in the second image sample set; screening the second image sample set according to the confidence corresponding to each image sample in the second image sample set to obtain a first training sample set, wherein the confidence corresponding to an image sample in the first training sample set is greater than a first confidence threshold; inputting the image sample in the first training sample set and a text recognition prompt word into the to-be-trained processing model to obtain a predicted text recognition result; determining a first score according to the predicted text recognition result and the text recognition result corresponding to the image sample; inputting the image sample in the second image sample set into a plurality of layout analysis modules to obtain a layout analysis result, a confidence, and an intersection-over-union corresponding to each layout analysis model; screening the image sample in the second image sample set according to the layout analysis result, the confidence, and the intersection-over-union corresponding to each layout analysis model to obtain a second training sample set, wherein the confidence corresponding to an image sample in the second training sample set is greater than a second confidence threshold, and the intersection-over-union is greater than an intersection-over-union threshold; inputting the image sample in the second training sample set and a layout prompt word into the to-be-trained processing model to obtain a predicted layout analysis result; determining a second score according to the predicted layout analysis result and the layout analysis result corresponding to the image sample; inputting the image sample in the second image sample set into a structure recognition model to obtain a structure recognition result; generating a relationship tree according to the structure recognition result, wherein the relationship tree comprises a root node and a leaf node; inputting the image sample in the second image sample set and a structured prompt word into the to-be-trained processing model to obtain a predicted layout analysis result; determining a third score according to the relationship tree and the predicted layout analysis result; According to the first score, the second score and the third score, parameters of the to-be-trained processing model are trained to obtain a target processing model.

3. The method of claim 2, wherein, According to the predicted text recognition result and the text recognition result corresponding to the image sample, a first score is determined, including: The length of the maximum common substring of the predicted text recognition result and the text recognition result corresponding to the image sample is obtained, and the length of the string of the text recognition result corresponding to the image sample is obtained; The ratio of the length of the maximum common substring to the length of the string of the text recognition result corresponding to the image sample is taken as the first score.

4. The method of claim 2, wherein, The predicted layout analysis result includes a plurality of pairs of predicted layout information, and each pair of predicted layout information includes a category and a predicted position coordinate corresponding to the category. The layout analysis result corresponding to the image sample includes a plurality of pairs of target layout information, and each pair of target layout information includes a category and a target position coordinate corresponding to the category. According to the predicted layout analysis result and the layout analysis result corresponding to the image sample, a second score is determined, including: The sum of the distance intersection ratios of the predicted position coordinates and the target position coordinates corresponding to the same category in the predicted layout analysis result and the layout analysis result corresponding to the image sample is obtained. The ratio of the sum of the distance intersection ratios to the number of pairs of predicted layout information included in the predicted layout analysis result is taken as the second score.

5. The method of claim 2, wherein, According to the relationship tree and the predicted layout analysis result, a third score is determined, including: A predicted relationship tree is generated according to the predicted layout analysis result; The number of nodes in the predicted relationship tree, the number of nodes in the relationship tree, the number of matching nodes in the predicted relationship tree, the values of the nodes in the predicted relationship tree and the values of the nodes in the relationship tree are used to determine the third score, wherein the matching nodes are nodes in the predicted relationship tree that match the nodes in the relationship tree.

6. An image processing method characterized by, It includes: A to-be-processed image and a target prompt word are obtained, wherein the target prompt word is any prompt word in a prompt word set; The to-be-processed image and the target prompt word are input into a target processing model to obtain a processing result corresponding to the target prompt word, wherein the target processing model is trained based on the target processing model training method in any of claims 1-5. 7.A target processing model training apparatus characterized by comprising: It includes: A first obtaining module is configured to obtain a target sample set, wherein the target sample set includes a first image sample set and a second image sample set, wherein an image sample in the first image sample set carries a label, and the label includes text information and an image type. A first training module is configured to train a first model based on the first image sample set to obtain a second model, wherein the first model includes a first to-be-trained image encoder, a second target image encoder and a to-be-trained decoder, and the second model includes a first target image encoder, a second target image encoder and a target decoder, and the image size input by the first to-be-trained image encoder is greater than the image size input by the second target image encoder. The to-be-trained processing model generation module is configured to generate a to-be-trained processing model according to the first target image encoder, the second target image encoder, and the to-be-trained natural language model; The second training module is configured to train the to-be-trained processing model according to the prompt word set, the first image sample set, and the second image sample set to obtain a target processing model.

8. An image processing apparatus characterized by comprising: The method comprises: The second acquisition module is configured to acquire a to-be-processed image and a target prompt word, wherein the target prompt word is any prompt word in the prompt word set; The image processing module is configured to input the to-be-processed image and the target prompt word into the target processing model to obtain a processing result corresponding to the target prompt word, wherein the target processing model is trained based on the target processing model training method in any one of claims 1-5.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the target processing model training method in any one of claims 1-5 or the image processing method in claim 6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute the target processing model training method in any one of claims 1-5 or the image processing method in claim 6 when executed.

Images