Model training method, text recognition method, and related devices

Abstract

This application discloses a model training method, a text recognition method, and related apparatus. The model training method includes: acquiring a pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks; pre-training a text recognition model to be trained based on the pre-training sample set to obtain a pre-trained text recognition model; the text recognition model is used to perform a text information recognition task indicated by the task prompt information on an input image according to task prompt information, obtaining the recognition result of the input image in the text information recognition task; acquiring a fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to paragraph text recognition tasks; and fine-tuning the pre-trained text recognition model based on the fine-tuning sample set to obtain a target text recognition model for performing paragraph text recognition tasks. This method can improve the recognition accuracy of paragraph-level text recognition tasks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model training method, a text recognition method, and related apparatus. Background Technology

[0002] Paragraph-level text recognition is a task that identifies the text contained in each paragraph of an input image. Related technologies typically employ multiple models in conjunction to perform paragraph-level text recognition. Specifically, an Optical Character Recognition (OCR) model can be used to identify each line of text and its position in the input image. A paragraph model can then be used to identify the positions of each paragraph in the input image. Finally, the positions of each line of text are matched with the positions of each paragraph to determine the text contained in each paragraph, thus obtaining the paragraph-level text recognition result.

[0003] However, the above methods require multiple models to work together to complete paragraph-level text recognition tasks. In this process, any error in any step will lead to an incorrect final text recognition result. Therefore, the above methods have low recognition accuracy in paragraph-level text recognition tasks. Summary of the Invention

[0004] This application provides a model training method, a text recognition method, and related apparatus, which can improve the recognition accuracy of paragraph-level text recognition tasks.

[0005] The first aspect of this application provides a model training method, the method comprising:

[0006] Obtain a pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks, the various pre-training tasks are used to determine the text information recognition results of different dimensions, and the pre-training samples include pre-training images and the annotation recognition results of the pre-training images in the corresponding pre-training tasks;

[0007] Based on the pre-training sample set, the text recognition model to be trained is pre-trained to obtain the pre-trained text recognition model; the text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information, and obtain the recognition result of the input image in the text information recognition task;

[0008] Obtain a fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to the paragraph text recognition task, the fine-tuning samples include fine-tuning images, and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task;

[0009] Based on the fine-tuning sample set, the pre-trained text recognition model is fine-tuned to obtain a target text recognition model for performing the paragraph text recognition task.

[0010] A second aspect of this application provides a text recognition method, the method comprising:

[0011] Obtain the image to be identified; the image to be identified includes at least one piece of text;

[0012] The target text recognition model determines each segment of text in the image to be recognized and its respective location information based on the image to be recognized; the target text recognition model is trained using the model training method described in the first aspect.

[0013] A third aspect of this application provides a model training apparatus, the apparatus comprising:

[0014] The first sample acquisition module is used to acquire a pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks, the various pre-training tasks are used to determine the text information recognition results of different dimensions, and the pre-training samples include pre-training images and the annotation recognition results of the pre-training images in the corresponding pre-training tasks;

[0015] The pre-training module is used to pre-train the text recognition model to be trained based on the pre-training sample set to obtain the pre-trained text recognition model; the text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information to obtain the recognition result of the input image in the text information recognition task;

[0016] The second sample acquisition module is used to acquire a fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to the paragraph text recognition task, and the fine-tuning samples include fine-tuning images and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task;

[0017] The fine-tuning module is used to fine-tune the pre-trained text recognition model based on the fine-tuning sample set to obtain a target text recognition model for performing the paragraph text recognition task.

[0018] A fourth aspect of this application provides a text recognition device, the device comprising:

[0019] An image acquisition module is used to acquire an image to be recognized; the image to be recognized includes at least one piece of text.

[0020] The text recognition module is used to determine, based on the image to be recognized, each segment of text in the image to be recognized and its respective position information, using a target text recognition model; the target text recognition model is trained using the model training method described in the first aspect.

[0021] A fifth aspect of this application provides a computer device, the device comprising a processor and a memory:

[0022] The memory is used to store computer programs;

[0023] The processor is configured to perform, according to the computer program, the steps of the model training method as described in the first aspect above, or the steps of the text recognition method as described in the second aspect above.

[0024] A sixth aspect of this application provides a computer-readable storage medium for storing a computer program for performing the steps of the model training method described in the first aspect above, or for performing the steps of the text recognition method described in the second aspect above.

[0025] A seventh aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the model training method described in the first aspect, or the steps of the text recognition method described in the second aspect.

[0026] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0027] This application provides a model training method, which proposes an end-to-end model for performing paragraph text recognition tasks and a training mechanism for this model. The method includes: acquiring a pre-training sample set, which includes pre-training samples corresponding to various pre-training tasks. These pre-training tasks are used to determine text information recognition results in different dimensions. The pre-training samples include pre-training images and their labeled recognition results in the corresponding pre-training tasks. Then, based on this pre-training sample set, the text recognition model to be trained is pre-trained to obtain a pre-trained text recognition model. This model is used to perform the text information recognition task indicated by the task prompts on the input image, obtaining the recognition result of the input image in the text information recognition task. In other words, by pre-training the text recognition model based on the pre-training samples corresponding to various pre-training tasks, the model learns multi-dimensional text information recognition capabilities in various pre-training tasks, thereby enabling the pre-trained text recognition model to have better generalization performance in image-text recognition tasks. Next, a fine-tuning sample set is obtained, which includes fine-tuning samples corresponding to the paragraph text recognition task. These fine-tuning samples include fine-tuned images and their labeled recognition results in the paragraph text recognition task. Further, based on this fine-tuning sample set, the pre-trained text recognition model can be fine-tuned to obtain a target text recognition model for performing the paragraph text recognition task. In other words, based on the fine-tuning samples corresponding to the paragraph text recognition task, a pre-trained text recognition model with better generalization can be fine-tuned, thereby specifically improving the recognition performance of the pre-trained text recognition model in the paragraph text recognition task, resulting in a target text recognition model specifically designed for performing the paragraph text recognition task. Through the above model training method, an end-to-end target text recognition model for performing the paragraph text recognition task can be trained. This target text recognition model can independently complete the paragraph text recognition task without cooperating with other models, reducing the influence of other models and thus improving the recognition accuracy of the paragraph text recognition task. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating an application scenario of the model training method provided in the embodiments of this application;

[0029] Figure 2 A schematic flowchart illustrating the model training method provided in this application embodiment;

[0030] Figure 3 A schematic diagram illustrating the position coordinates of a text box provided in an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the structure of the text recognition model provided in the embodiments of this application;

[0032] Figure 5 A schematic diagram of the model structure of the text recognition model provided in the embodiments of this application;

[0033] Figure 6 A schematic diagram illustrating the model training method provided in the embodiments of this application;

[0034] Figure 7 This is a schematic diagram of the structure for pre-training an image encoder provided in an embodiment of this application;

[0035] Figure 8 This is a schematic diagram of the overall structure of the model training method provided in the embodiments of this application;

[0036] Figure 9 A comparative diagram illustrating the recognition effects provided in the embodiments of this application;

[0037] Figure 10 A flowchart illustrating the text recognition method provided in this application embodiment;

[0038] Figure 11 A schematic diagram illustrating the text recognition method provided in an embodiment of this application;

[0039] Figure 12 A schematic diagram of an interface for a translation scenario provided in an embodiment of this application;

[0040] Figure 13 A schematic diagram of an interface for another translation scenario provided in an embodiment of this application;

[0041] Figure 14 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application;

[0042] Figure 15 This is a schematic diagram of the structure of the text recognition device provided in the embodiments of this application;

[0043] Figure 16 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application;

[0044] Figure 17 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0046] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] The model training method and text recognition method provided in this application can be executed by a computer device, which can be a terminal device or a server. The terminal device includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server.

[0048] It should be noted that the information, data and signals involved in the embodiments of this application are all authorized by the relevant parties or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0049] To facilitate understanding of the model training method provided in this application embodiment, the following example uses a server as the execution subject of the model training method to illustrate its application scenarios.

[0050] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the model training method provided in the embodiments of this application. For example... Figure 1 As shown, this application scenario includes a database 110 and a server 120. The server 120 can access the database 110 via the network, or the database 110 can be integrated inside the server 120.

[0051] The database 110 stores a pre-training sample set, which includes pre-training samples corresponding to various pre-training tasks. These tasks are used to determine the text information recognition results in different dimensions. The pre-training samples include pre-training images and their labeled recognition results in the corresponding pre-training tasks. The server 120 can send a data retrieval request to the database 110 to obtain the pre-training sample set from it.

[0052] After obtaining the pre-training sample set from server 120, the text recognition model to be trained can be pre-trained based on the pre-training sample set to obtain the pre-trained text recognition model. This text recognition model is used to perform text information recognition tasks based on task prompts on the input image, obtaining the recognition result of the input image in the text information recognition task. That is, by pre-training the text recognition model based on the pre-training samples corresponding to various pre-training tasks, the text recognition model learns multiple dimensions of text information recognition capabilities in various pre-training tasks, thereby giving the pre-trained text recognition model better generalization ability in image and text recognition tasks.

[0053] Furthermore, database 110 also stores a fine-tuning sample set, which includes fine-tuning samples corresponding to the paragraph text recognition task. The fine-tuning samples include fine-tuning images and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task. Server 120 can send a data retrieval request to database 110 to obtain the fine-tuning sample set from database 110.

[0054] Furthermore, after obtaining the fine-tuning sample set, server 120 can fine-tune the pre-trained text recognition model based on the fine-tuning sample set to obtain a target text recognition model for performing paragraph text recognition tasks. In other words, based on the fine-tuning samples corresponding to the paragraph text recognition task, the pre-trained text recognition model with better generalization can be fine-tuned, thereby specifically improving the recognition performance of the pre-trained text recognition model in paragraph text recognition tasks, to obtain a target text recognition model specifically designed for performing paragraph text recognition tasks.

[0055] It should be understood that Figure 1 This paper takes the example of storing the pre-training sample set and the fine-tuning sample set in the same database. In practical applications, the pre-training sample set and the fine-tuning sample set can be stored in different databases, and the pre-training samples corresponding to different pre-training tasks can also be stored in different databases. This application embodiment does not impose any limitation on the storage location of the pre-training samples and the fine-tuning samples.

[0056] It should be understood that Figure 1 The application scenarios shown are merely examples. In practical applications, the model training method provided in this application embodiment can also be applied to other scenarios. No limitations are made here on the application scenarios of the model training method provided in this application embodiment.

[0057] The model training method provided in this application will be described in detail below through method embodiments.

[0058] See Figure 2 , Figure 2 This is a flowchart illustrating the model training method provided in an embodiment of this application. For ease of description, the following description uses a server as the execution entity of this model training method. Figure 2 As shown, the model training method includes the following steps:

[0059] S201: Obtain the pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks. The various pre-training tasks are used to determine the text information recognition results of different dimensions. The pre-training samples include pre-training images and the annotation recognition results of the pre-training images in the corresponding pre-training tasks.

[0060] A pre-training sample set refers to the collection of pre-training samples used to pre-train a text recognition model. This set includes pre-training samples corresponding to various pre-training tasks. These various pre-training tasks refer to multiple task scenarios related to text recognition, each used to determine the text information recognition results in different dimensions. For example, these tasks may include, but are not limited to: determining the text contained in an input image (by line or by single character); determining the text contained in an input image by paragraph; determining the position of each paragraph in an input image; and determining the text at a given position in an input image, etc.

[0061] Pre-training samples for a pre-training task refer to the training samples used by the text recognition model to perform that pre-training task. Pre-training samples include pre-training images and the labeled recognition results of those images in the corresponding pre-training task. Pre-training images are images containing paragraph text. The labeled recognition results of the pre-training images in the corresponding pre-training task refer to the labels of the pre-training images in that task, i.e., the standard recognition results of the pre-training images in that pre-training task.

[0062] That is, for each pre-training task, the pre-training samples corresponding to the pre-training task include the pre-training image and the annotation and recognition results of the pre-training image in that pre-training task. Taking the pre-training task of determining the text contained in the input image by paragraph as an example, the pre-training samples of this pre-training task can include the pre-training image, the text of each paragraph in the pre-training image, and the position information of each paragraph.

[0063] It should be noted that the pre-training images in the pre-training samples corresponding to various pre-training tasks can be the same image or different images. In this regard, this application does not specifically limit the pre-training images in the pre-training samples, or the annotation and recognition results of the pre-training images in the corresponding pre-training tasks.

[0064] Pre-training samples can be stored in a database beforehand. The server can retrieve pre-training samples from the database to pre-train the text recognition model to be trained based on the retrieved pre-training samples.

[0065] In one possible implementation, the pre-training tasks may include at least two of the following: paragraph text recognition task, local paragraph recognition task, and paragraph position recognition task; the paragraph text recognition task is used to recognize each paragraph of text in the input image and its respective position information; the local paragraph recognition task is used to recognize the text at the paragraph position in the input image based on the given paragraph position; and the paragraph position recognition task is used to recognize the start character, end character, and position information of each paragraph of text in the input image.

[0066] Paragraph text recognition is a task used to identify the text contained within each paragraph of an input image, as well as the positional information of each paragraph. For example, the positional information of each paragraph may include the coordinates of the paragraph frame, the angle between the paragraph frame and the horizontal line of the input image, and the reading direction of the text. This application does not specifically limit the positional information of each paragraph.

[0067] Local paragraph recognition is a task that identifies the text at the given paragraph box location in an input image based on the given paragraph box coordinates.

[0068] Paragraph position recognition is a task used to identify the starting character (i.e., the first character of each paragraph), the ending character (i.e., the last character of each paragraph), and the positional information of each paragraph within an input image. For example, the positional information of each paragraph may include the coordinates of the paragraph frame, the angle between the paragraph frame and the horizontal line of the input image, and the reading direction of the text. This application does not specifically limit the positional information of each paragraph.

[0069] The pre-training sample set includes at least two of the following pre-training tasks: paragraph text recognition, local paragraph recognition, and paragraph position recognition. It should be noted that the position coordinates of the text box in each paragraph can be represented by the coordinates of the top-left and bottom-right points within that text box, as shown in the reference. Figure 3 , Figure 3 This is a schematic diagram of the position coordinates of a text box provided in an embodiment of this application. In the diagram, x1 and y1 are the coordinates of the upper left point of the text box, and x2 and y2 are the coordinates of the lower right point of the text box. Furthermore, the angle θ between the text box and the horizontal line of the input image can also be used to represent the angle of the text box.

[0070] As can be seen, the pre-training tasks in the pre-training process can include at least two of the following: paragraph text recognition tasks, local paragraph recognition tasks, and paragraph position recognition tasks. Therefore, the text recognition model to be trained can be pre-trained from different dimensions, so that the text recognition model can learn the text recognition ability at the paragraph level from different dimensions and achieve better generalization in paragraph-level text recognition tasks.

[0071] In one possible implementation, various pre-training tasks also include a global text recognition task, which is used to recognize all text in an input image without distinguishing paragraphs. Specifically, in the global text recognition task, it is not necessary to distinguish paragraphs; it is only necessary to recognize all text content in the input image, which can be done on a single character or line-by-line basis.

[0072] In other words, the pre-training tasks in the pre-training process can include at least two of the following: global text recognition tasks, paragraph text recognition tasks, local paragraph recognition tasks, and paragraph position recognition tasks. Therefore, the text recognition model to be trained can be pre-trained from different dimensions, so that the text recognition model can learn the text recognition capabilities of the global dimension and the paragraph dimension, and thus the text recognition model has better generalization.

[0073] In one possible implementation, the multiple pre-training tasks may include a global text recognition task and at least one paragraph-level text recognition task. That is, the pre-training sample set may include a first sub-sample set and a second sub-sample set. The first sub-sample set includes pre-training samples corresponding to the global text recognition task, and the second sub-sample set includes pre-training samples corresponding to at least one paragraph-level text recognition task.

[0074] The first subset refers to the set of pre-trained samples used when training the text recognition model to perform a global text recognition task. This subset includes pre-trained samples corresponding to the global text recognition task. These pre-trained samples are the training samples upon which the text recognition model bases its performance. The pre-trained samples include pre-trained images and their labeled recognition results in the global text recognition task. The labeled recognition results of the pre-trained images in the global text recognition task refer to the labels of the pre-trained images in the global text recognition task, i.e., the standard recognition results of the pre-trained images in the global text recognition task. For example, these labeled recognition results could be all the text content in the pre-trained images. The global text recognition task refers to the task of recognizing all text in an input image (either by line or by individual character).

[0075] The second subsample set refers to the set of pre-trained samples used when training the text recognition model to perform paragraph-level text recognition tasks. The second subsample set includes at least one pre-trained sample corresponding to a paragraph-level text recognition task. A paragraph-level text recognition task refers to the task of recognizing text information in an input image on a paragraph-by-paragraph basis. Specifically, it can be used to recognize the text content in an input image on a paragraph-by-paragraph basis, or it can be used to recognize the position of each paragraph in an input image, etc.

[0076] The global text recognition task identifies all text in an input image without distinguishing between paragraphs. Specifically, it can identify text in the input image at the individual character level or line level. The paragraph-level text recognition task includes at least one of the following: paragraph text recognition, local paragraph recognition, and paragraph position recognition. In other words, the paragraph-level text recognition task can be at least one of several pre-trained tasks.

[0077] It is evident that, for global text recognition tasks, pre-training the text recognition model before training enables the model to learn the ability to recognize all text in the input image without distinguishing paragraphs, thus ensuring the accuracy of global text recognition. For paragraph-level text recognition tasks, pre-training the text recognition model before training enables the model to learn the ability to recognize text at the paragraph level, which is beneficial for improving the accuracy of paragraph-level recognition.

[0078] In one possible implementation, the annotation recognition results in the pre-training samples corresponding to the paragraph text recognition task may include each paragraph of text in the pre-training image, as well as the paragraph box position information and reading direction corresponding to each paragraph.

[0079] The paragraph box position information includes the coordinates of the box's positioning point and the paragraph box angle. The coordinates of the box's positioning point are the coordinates of the positioning point of the paragraph annotation box. For example, the coordinates of the lower left and upper right points of the paragraph annotation box can be used as the coordinates of its box positioning point. The paragraph box angle is the angle between the paragraph annotation box and the horizontal line of the pre-training image. The reading direction refers to the browsing direction of each paragraph of text in the pre-training image, such as reading from left to right or from right to left. For example, the reading direction can be determined by... <direction> ${direction}< / direction> To express.

[0080] The pre-training samples for the local paragraph text recognition task also include task prompts, which include the location information of the given paragraph, such as the coordinates of the bounding box's location points and the angle of the bounding box. The labeled recognition results in the pre-training samples can include the text contained within the given bounding box in the pre-training image. In other words, the labeled recognition results in the pre-training samples for the local paragraph text recognition task are the text within the bounding box given in the task prompts in the pre-training image.

[0081] In the pre-training samples corresponding to the paragraph position recognition task, the annotation and recognition results of the pre-training images in the paragraph position recognition task can include the start and end characters contained in each paragraph in the pre-training image, as well as the coordinate information of the box positioning points corresponding to each paragraph, the paragraph box angle, and the reading direction.

[0082] As can be seen, the above content ensures that the text recognition model to be trained can learn different recognition capabilities during the training process of different pre-training tasks. In the paragraph text recognition task, it can learn to recognize each paragraph of text, as well as the coordinate information of the corresponding box positioning points, the paragraph box angle, and the reading direction. In the local paragraph recognition task, it can learn to recognize the text within a given paragraph box. In the paragraph position recognition task, it can learn to recognize the start and end characters of each paragraph of text, as well as the coordinate information of the corresponding box positioning points, the paragraph box angle, and the reading direction. This ensures that the pre-trained text recognition model can complete the recognition of different pre-training tasks and has high generalization ability.

[0083] S202: Based on the pre-training sample set, the text recognition model to be trained is pre-trained to obtain the pre-trained text recognition model; the text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information, and obtain the recognition result of the input image in the text information recognition task.

[0084] After obtaining the pre-training sample set, the text recognition model to be trained can be pre-trained based on the pre-training sample set to obtain the pre-trained text recognition model. The text recognition model to be trained refers to the training object of the pre-training. The text recognition model refers to an end-to-end model that recognizes specific text information in an input image. The text recognition model to be trained can be composed of an image encoder and a text decoder. For example, the image encoder can be the encoder in the vision model for image processing (Swin Transformer), and the text decoder can be the multilingual denoising pre-training for Neural Machine Translation (mBart). In this regard, this application does not specifically limit the model structure of the text recognition model to be trained.

[0085] In other words, pre-trained images from the pre-training sample set are input into the text recognition model to be trained. This model processes the input pre-training images and outputs the predicted recognition results for those images in the corresponding pre-training task. Then, a loss value is calculated based on the predicted recognition results and the labeled recognition results corresponding to the pre-training images. This loss value is then used to train the text recognition model. By iteratively executing the above training process using different pre-training samples from the pre-training sample set, the pre-trained text recognition model can be obtained.

[0086] It should be understood that in practical applications, a pre-training termination condition can be preset. When the training of the text recognition model to be trained reaches the pre-training termination condition, the pre-training of the text recognition model to be trained can be considered to have ended. This pre-training termination condition could be, for example, that the number of pre-training epochs of the text recognition model to be trained reaches a preset epoch threshold; or, for example, that the performance of the text recognition model is tested and found to have reached a preset performance standard (such as reaching a preset recognition accuracy); or, for example, that the performance of the text recognition model is tested and found to no longer significantly improve with further training. This embodiment of the application does not impose any limitations on the pre-training termination condition.

[0087] It should be noted that the text recognition model is used to perform text recognition tasks indicated by task prompts on the input image, and can obtain the recognition results of the input image in the text recognition task. During the pre-training stage of the text recognition model, the task prompts can be prompts corresponding to various pre-training tasks. Accordingly, the text recognition task indicated by the task prompt is the pre-training task corresponding to that task prompt. The task prompts corresponding to various pre-training tasks are usually different in application; however, for multiple pre-training samples corresponding to the same pre-training task, their associated task prompts can be the same or different. Taking a local paragraph recognition task as an example, different pre-training samples corresponding to this task may be associated with different task prompts. That is, for different pre-training samples, their associated task prompts may indicate different position information of a given paragraph box. Taking a paragraph text recognition task as another example, different pre-training samples corresponding to this task may be associated with the same task prompt. This application does not specifically limit the task prompt information.

[0088] For example, refer to Table 1, which includes task prompts and recognition results for various pre-training tasks.

[0089] Table 1

[0090]

[0091]

[0092] The task prompts for the Para-OCR task listed in Table 1 above can be obtained through "<para_ocr> The corresponding recognition result can be the text, position coordinates, angle, and reading direction of each paragraph in the input image; the task prompt information for the Para-Rec task can be expressed as "".<para_rec> <loc> x_0, y_0, x_1, y_1, angle< / loc> The text is represented by "x_0, y_0, x_1, y_1, angle", where "x_0, y_0, x_1, y_1, angle" represent the given position coordinates and angle of the paragraph box. The corresponding recognition result can be the text within the paragraph box at that position coordinate in the input image. The task prompt information for the Para-Det task can be expressed as "x_0, y_0, x_1, y_1, angle".<para_det> The corresponding recognition result can be the starting character, ending character, position coordinates, angle, and reading direction of each paragraph in the input image.

[0093] The task prompt information is input into the text recognition model, so that the text recognition model can perform the pre-training task corresponding to the task prompt information on the input image based on the task prompt information. Then, the text recognition model can output the recognition result of the input image in the pre-training task.

[0094] In one possible implementation, where the pre-training sample set includes a first subset and a second subset, the first subset includes pre-training samples corresponding to the global text recognition task, and the second subset includes pre-training samples corresponding to the paragraph-level text recognition task, then the phrase "pre-training the text recognition model to be trained based on the pre-training sample set to obtain the pre-trained text recognition model" in S202 above can include:

[0095] Based on the first subset of samples, the text recognition model to be trained is pre-trained in the first stage to obtain the intermediate text recognition model; based on the second subset of samples, the intermediate text recognition model is pre-trained in the second stage to obtain the pre-trained text recognition model.

[0096] When training based on a pre-training sample set, the text recognition model to be trained can first undergo a first-stage pre-training based on a first subset of samples. This involves inputting the pre-training images from the first subset into the text recognition model. The model processes these pre-training images and outputs a predicted recognition result for the pre-training image in the global text recognition task. Then, a loss value is calculated based on the predicted recognition result and the corresponding labeled recognition result of the pre-training image in the global text recognition task. This loss value is then used to train the text recognition model. This training process is iteratively executed using different pre-training samples from the first subset to iteratively train the text recognition model, thus obtaining an intermediate text recognition model. The intermediate text recognition model refers to the text recognition model obtained after pre-training based on the first subset of samples, which exhibits superior performance in the global text recognition task.

[0097] Then, based on the second subset of samples, a second stage of pre-training can be performed on the intermediate text recognition model to obtain a pre-trained text recognition model. Specifically, pre-trained images from the pre-training samples in the second subset can be input into the intermediate text recognition model. This model processes the input pre-training images and outputs the predicted recognition results for the corresponding pre-training tasks. Then, a loss value can be calculated based on the predicted recognition results and the labeled recognition results corresponding to the pre-training images, and this loss value can be used to train the intermediate text recognition model. By iteratively executing the above training process using different pre-training samples from the second subset, the pre-trained text recognition model can be obtained.

[0098] It should be understood that corresponding pre-training termination conditions can be set for the first and second stages of pre-training to control the termination of the first and second stages of pre-training respectively. These termination conditions may include, but are not limited to, restrictions on the number of pre-training rounds, performance restrictions on the trained text recognition model, etc.

[0099] As can be seen, the above method allows for the first stage of pre-training of the text recognition model to be trained, based on a first subset of pre-trained samples corresponding to the global text recognition task. This enables the text recognition model to learn global text recognition capabilities, resulting in an intermediate text recognition model. Then, based on a second subset of pre-trained samples corresponding to paragraph-level text recognition tasks, the intermediate text recognition model undergoes a second stage of pre-training, enabling it to learn paragraph-level text recognition capabilities. In other words, based on the intermediate text recognition model that has learned global text recognition capabilities, it further learns paragraph-level text recognition capabilities. This helps to efficiently and quickly improve the performance of the text recognition model in paragraph-level text recognition tasks.

[0100] It should be understood that in practical applications, the text recognition model to be trained can also be pre-trained without distinguishing between stages. That is, pre-training samples corresponding to different pre-training tasks can be mixed and used to pre-train the text recognition model to be trained.

[0101] In one possible implementation, the step S202 above, "pre-training the text recognition model to be trained based on the pre-trained sample set to obtain the pre-trained text recognition model," may include the following S2021-S2024 (not shown in the figure):

[0102] S2021: Input the pre-trained images from the pre-trained samples, along with task prompts indicating the pre-trained tasks corresponding to the pre-trained samples, into the text recognition model to be trained.

[0103] During pre-training based on a pre-training sample set, the text recognition model can be pre-trained using pre-training samples corresponding to each pre-training task. Specifically, the pre-training images from the pre-training samples corresponding to each pre-training task, along with task prompts indicating that pre-training task, can be input into the text recognition model to be trained. This allows the text recognition model to perform the pre-training task based on the task prompts corresponding to that pre-training task, using the pre-training images from the pre-training samples.

[0104] S2022: The pre-training image is encoded using the image encoder in the text recognition model to be trained, and the pre-training image features are obtained.

[0105] The text recognition model to be trained can consist of an image encoder and a text decoder, for example, refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of the text recognition model provided in this application embodiment. The image encoder can be the encoder in the Swin model. By encoding the pre-training image using this image encoder, the pre-training image features can be obtained. The pre-training image features are used to represent the global image information in the pre-training image. That is, the pre-training image can be input into the image encoder (vision encoder) in the text recognition model to be trained, and then the image features in the pre-training image can be extracted through the image encoder to obtain the pre-training image features.

[0106] S2023: The text decoder in the text recognition model to be trained decodes the pre-trained image based on the pre-trained image features and task prompts to obtain the predicted recognition result of the pre-trained image in the pre-training task.

[0107] Then, the pre-trained image features and task prompts can be input into the text decoder of the text recognition model to be trained. The text decoder decodes the pre-trained image based on the task prompts and pre-trained image features to obtain the predicted recognition result of the pre-trained image in the pre-training task. The predicted recognition result refers to the output result obtained by the text recognition model to be trained when it performs the corresponding pre-training task on the input pre-trained image. For example, refer to... Figure 4 , Figure 4 The text decoder uses the mBart model's decoding structure. When the pre-training task is paragraph text recognition, the task prompt information input to the text decoder can be "<para_ocr> The text decoder can perform decoding based on the task prompts and pre-trained image features to obtain the predicted recognition results of the pre-trained image in the paragraph text recognition task. <para> <text> Good afternoon...< / text> <loc> x0, y0, x1, y1, angle< / loc> ...”.

[0108] pass Figure 4 As shown in the diagram, the image encoder and text decoder can be connected through a cross-attention mechanism. That is, the pre-trained image features output by the image encoder can be fused with the features generated during the processing of the text decoder through the cross-attention mechanism. This helps the text decoder combine the pre-trained image features and task prompts to determine the prediction and recognition results.

[0109] The specific model structure of the text recognition model to be trained can be found in [reference]. Figure 5 , Figure 5 This is a schematic diagram of the model structure of the text recognition model provided in the embodiments of this application. The image encoder part is the encoder structure in the Swin model. First, the pre-training image can be input into the image encoder. The pre-training image is first converted into an embedding vector through patch partitioning. Then, the input channel of the embedding vector is adjusted through the linear embedding layer, that is, the size of the pre-training image can be adjusted from H / 4*W / 4*48 to H / 4*W / 4*C. Then, the embedding vectors corresponding to the pre-trained images can be input into four stages for progressive image feature extraction. Each stage (except the first stage) consists of patch merging and a Swing Transformer block. Patch merging is used to perform downsampling before feature extraction in each stage to reduce the resolution. The Swing Transformer block consists of a normalization layer (LayerNorm), a window attention mechanism, a shifted window attention mechanism, and a multilayer perceptron (MLP). In the Swing Transformer block, the input image features are divided into multiple non-overlapping windows, and a window attention mechanism is calculated within each window. Then, every certain number of layers, the windows are shifted, and a shifted window attention mechanism can be calculated based on the shifted windows to capture global image features. Furthermore, the output of the attention mechanism can be further processed by a multilayer perceptron to extract effective image features. Finally, the final output image features can be stabilized by a normalization layer, making the output distribution of each stage more stable, which helps to improve the convergence speed and stability of the image encoder.

[0110] After obtaining the pre-trained image features output by the image encoder, these features can be input into the text decoder. Task prompts can also be input into the text decoder for decoding. Specifically, the task prompts can be converted into corresponding embedding vectors using an output embedding layer. These embedding vectors are then input into the decoder layer. The text decoder can contain N (N is an integer greater than or equal to 1) decoder layers, each consisting of three sub-layer connections. The first sub-layer connection includes a masked multi-head attention sub-layer, a normalization layer, and a residual connection (Add&Norm). The second sub-layer connection includes a multi-head attention sub-layer, a normalization layer, and a residual connection (Add&Norm). The third sub-layer connection includes a feedforward fully connected sub-layer, a normalization layer, and a residual connection (Add&Norm). The input to the second sub-layer connection is the output of the first sub-layer connection and the pre-trained image features output by the image encoder. After multiple decoder layers perform decoding, the output data of the decoder layers can be linearly transformed by a linear layer, and the output of the linear layer can be converted into a probability distribution by an activation function layer (softmax). This allows us to obtain the predicted recognition result output by the text recognition model to be trained.

[0111] S2024: Determine the pre-training loss based on the labeled recognition results and predicted recognition results in the pre-training samples; pre-train the text recognition model to be trained based on the pre-training loss.

[0112] Pre-training loss refers to the loss value calculated during the pre-training process of a text recognition model based on pre-training samples.

[0113] After determining the predicted recognition result, the pre-training loss can be determined based on the labeled recognition result and the predicted recognition result in the pre-training samples. This pre-training loss can then be used to pre-train the text recognition model to be trained. That is, the difference between the labeled recognition result and the predicted recognition result in the pre-training task can be calculated using a loss function to obtain the loss value, i.e., the pre-training loss. Then, the text recognition model to be trained can be pre-trained based on this pre-training loss. This involves continuously adjusting the parameters of the text recognition model to optimize its recognition performance and improve its accuracy, with the goal of reducing the pre-training loss.

[0114] As can be seen, through the above method, this embodiment combines the encoder part of the SWIN model and the decoder part of the mBart model to form a text recognition model to be trained. Global image features can be extracted from the pre-training image through the image encoder. Then, the text decoder can perform decoding processing based on the pre-training image features and task prompt information. That is, the pre-training task indicated by the task prompt information and the pre-training image features can be fused to obtain the prediction and recognition result of the pre-training image in the pre-training task. Finally, the training process for the pre-training task can be completed based on the pre-training loss between the prediction and labeled recognition results, so that the text recognition model to be trained can learn the recognition ability of the pre-training task during the training process.

[0115] S203: Obtain the fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to the paragraph text recognition task, and the fine-tuning samples include fine-tuning images and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task.

[0116] The fine-tuning sample set refers to the collection of samples used to fine-tune the pre-trained text recognition model. This set includes fine-tuning samples corresponding to the paragraph text recognition task. The paragraph text recognition task refers to the task of recognizing the text information corresponding to each paragraph in an input image, such as recognizing text within a paragraph, paragraph box position, angle, and reading direction. The fine-tuning samples corresponding to the paragraph recognition task refer to the fine-tuning samples used by the text recognition model when performing the paragraph text recognition task. These fine-tuning samples include the fine-tuned image and the labeled recognition results of the fine-tuned image in the paragraph text recognition task. The fine-tuned image refers to an image containing paragraph text, which may be the same as or different from the pre-training image. This application does not specifically limit the fine-tuned image and the pre-training image. The labeled recognition results of the fine-tuned image in the paragraph text recognition task refer to the label of the fine-tuned image in that task, i.e., the standard recognition result of the fine-tuned image in the paragraph text recognition task.

[0117] The fine-tuning sample set can be pre-stored in a database. The server can retrieve the fine-tuning sample set from the database to fine-tune the pre-trained text recognition model based on the retrieved fine-tuning sample set.

[0118] In one possible implementation, in the fine-tuning samples corresponding to the paragraph text recognition task, the annotation and recognition results of the fine-tuning images may include the text content contained in each paragraph in the fine-tuning image, as well as the coordinate information of the corresponding box positioning points of each paragraph, the paragraph box angle, and the reading direction. It should be noted that, in order to ensure the accuracy of the fine-tuning samples, the annotation and recognition results of the fine-tuning images in the fine-tuning samples can usually be obtained through manual annotation.

[0119] As can be seen, the above content can ensure that the pre-trained text recognition model can specifically learn the recognition ability of paragraph text recognition task during the fine-tuning process for paragraph text recognition task, thereby improving the recognition accuracy of the target text recognition model in paragraph text recognition task.

[0120] S204: Based on the fine-tuning sample set, the pre-trained text recognition model is fine-tuned to obtain the target text recognition model for performing paragraph text recognition tasks.

[0121] After obtaining the fine-tuning sample set, the pre-trained text recognition model can be fine-tuned based on this set to obtain the target text recognition model for performing paragraph text recognition tasks. Specifically, the fine-tuned images from the fine-tuning sample set can be input into the pre-trained text recognition model. This model processes the input fine-tuned images and outputs a predicted recognition result for the fine-tuned image in the paragraph text recognition task. Then, a loss value can be calculated based on the predicted recognition result and the corresponding labeled recognition result of the fine-tuned image. This loss value is then used to fine-tune the pre-trained text recognition model. By iteratively executing the above fine-tuning process using different fine-tuned samples from the fine-tuning sample set, the pre-trained text recognition model can be iteratively fine-tuned to obtain the fine-tuned text recognition model, which is the target text recognition model for performing paragraph text recognition tasks.

[0122] It should be understood that in practical applications, a fine-tuning termination condition can be preset. When the training of the pre-trained text recognition model reaches this termination condition, the fine-tuning of the pre-trained text recognition model can be considered to have ended. This fine-tuning termination condition could be, for example, that the training epochs of the pre-trained text recognition model have reached a preset epoch threshold; or it could be that the performance of the text recognition model has been tested and found to have reached a preset performance standard (such as reaching a preset recognition accuracy); or it could be that the performance of the text recognition model no longer significantly improves with further training. This application does not impose any limitations on this fine-tuning termination condition.

[0123] For reference Figure 6 , Figure 6 This diagram illustrates the model training method provided in this application. For example, a text recognition model to be trained can be pre-trained for multiple tasks in a business scenario using a business dataset of 11 million (i.e., a pre-trained sample set of 11 million), resulting in a pre-trained text recognition model. Then, a fine-tuning sample set of 100,000 manually annotated samples can be used to fine-tune the pre-trained text recognition model based on the business scenario, thus obtaining the final target text recognition model suitable for paragraph text recognition tasks.

[0124] In one possible implementation, the "fine-tuning the pre-trained text recognition model based on the fine-tuning sample set" in S204 above may include the following S2041-S2044 (not shown in the figure):

[0125] S2041: Input the fine-tuning images from the fine-tuning samples, along with the task prompts indicating the paragraph text recognition task, into the pre-trained text recognition model.

[0126] During the fine-tuning process based on the fine-tuning sample set, the pre-trained text recognition model can be fine-tuned based on the fine-tuning samples corresponding to the paragraph text recognition task. Specifically, the fine-tuned image from the fine-tuning samples corresponding to the paragraph text recognition task, along with the task prompts indicating the paragraph text recognition task, can be input into the pre-trained text recognition model. This allows the pre-trained text recognition model to perform the paragraph text recognition task on the fine-tuned image based on the task prompts corresponding to the paragraph text recognition task.

[0127] S2042: The fine-tuned image is encoded using the image encoder in the pre-trained text recognition model to obtain the fine-tuned image features.

[0128] The model structure of the pre-trained text recognition model is the same as that of the text recognition model to be trained, and will not be repeated here. After determining the fine-tuning image, the image encoder in the pre-trained text recognition model can be used to encode the fine-tuning image, that is, the image features in the fine-tuning image can be extracted by the image encoder, thus obtaining the fine-tuning image features. The fine-tuning image features are used to represent the global image information in the fine-tuning image.

[0129] S2043: The text decoder in the pre-trained text recognition model decodes the image based on the fine-tuned image features and task prompts to obtain the predicted recognition result of the fine-tuned image in the paragraph text recognition task.

[0130] After determining the fine-tuned image features, these features and task prompts can be input into the text decoder of the pre-trained text recognition model. The text decoder then decodes the image based on the task prompts and the fine-tuned image features to obtain the predicted recognition result of the fine-tuned image in the paragraph text recognition task. The predicted recognition result refers to the output of the pre-trained text recognition model when performing the paragraph text recognition task on the input fine-tuned image features. The decoding process using the text decoder is the same as the decoding process in S2023 above, and will not be repeated here.

[0131] S2044: Determine the fine-tuning loss based on the labeled recognition results and predicted recognition results in the fine-tuning samples; fine-tune the pre-trained text recognition model based on the fine-tuning loss.

[0132] Fine-tuning loss refers to the loss value calculated during the process of fine-tuning the text recognition model based on fine-tuning samples.

[0133] After determining the predicted recognition result, the fine-tuning loss can be determined based on the labeled recognition result and the predicted recognition result in the fine-tuning sample. This fine-tuning loss can then be used to fine-tune the pre-trained text recognition model. Specifically, the difference between the labeled recognition result and the predicted recognition result in the paragraph text recognition task can be calculated using a loss function to obtain the loss value, i.e., the fine-tuning loss. Then, the pre-trained text recognition model can be fine-tuned based on this loss. This involves continuously adjusting the parameters of the pre-trained text recognition model to optimize its recognition performance and improve its accuracy, with the goal of reducing the fine-tuning loss.

[0134] As can be seen, the above method allows for the extraction of global image features from the fine-tuned image using an image encoder. Then, a text decoder can perform decoding based on the fine-tuned image features and the task prompts corresponding to the paragraph text recognition task. This allows for the fusion of the paragraph text recognition task indicated by the task prompts and the fine-tuned image features to obtain the predicted recognition result of the fine-tuned image in the paragraph text recognition task. Finally, the fine-tuning process for the paragraph text recognition task can be completed based on the fine-tuning loss between the predicted recognition result and the labeled recognition result. This enables the pre-trained text recognition model to specifically learn the recognition ability of the paragraph text recognition task, thereby improving the recognition accuracy of the target text recognition model in the paragraph text recognition task.

[0135] In the model training method provided in this application embodiment, an end-to-end model for performing paragraph text recognition tasks is provided, and a training mechanism is proposed for this model. The method includes: acquiring a pre-training sample set, which includes pre-training samples corresponding to various pre-training tasks. These pre-training tasks are used to determine text information recognition results in different dimensions. The pre-training samples include pre-training images and the labeled recognition results of the pre-training images in their respective pre-training tasks. Then, based on the pre-training sample set, the text recognition model to be trained is pre-trained to obtain a pre-trained text recognition model. This text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information, and obtain the recognition result of the input image in the text information recognition task. That is, by pre-training the text recognition model based on the pre-training samples corresponding to various pre-training tasks, the text recognition model learns multiple dimensions of text information recognition capabilities in various pre-training tasks, thereby enabling the pre-trained text recognition model to have better generalization in image-text recognition tasks. Next, a fine-tuning sample set is obtained, which includes fine-tuning samples corresponding to the paragraph text recognition task. These fine-tuning samples include fine-tuned images and their labeled recognition results in the paragraph text recognition task. Further, based on this fine-tuning sample set, the pre-trained text recognition model can be fine-tuned to obtain a target text recognition model for performing the paragraph text recognition task. In other words, based on the fine-tuning samples corresponding to the paragraph text recognition task, a pre-trained text recognition model with better generalization can be fine-tuned, thereby specifically improving the recognition performance of the pre-trained text recognition model in the paragraph text recognition task, resulting in a target text recognition model specifically designed for performing the paragraph text recognition task. Through the above model training method, an end-to-end target text recognition model for performing the paragraph text recognition task can be trained. This target text recognition model can independently complete the paragraph text recognition task without cooperating with other models, reducing the influence of other models and thus improving the recognition accuracy of the paragraph text recognition task.

[0136] In one possible implementation, before performing S202 above, the method provided in this application embodiment may further include:

[0137] Obtain a pre-trained image set; the pre-trained image set includes multiple pre-trained text images;

[0138] Based on the pre-trained image set, the image encoder is pre-trained to obtain the pre-trained image encoder;

[0139] A text recognition model to be trained is constructed using a pre-trained image encoder and text decoder.

[0140] A pre-training image set refers to the basic data used as input data when training the image encoder in a text recognition model. The pre-training image set includes multiple pre-training text images, which are images containing text, specifically images containing dense text. For example, 100 million images containing text can be obtained from a database as pre-training text images. These 100 million images may include publicly available images, as well as pre-training images corresponding to various pre-training tasks. This application does not specifically limit the pre-training image set.

[0141] After obtaining the pre-trained image set, the image encoder in the text recognition model can be pre-trained based on the pre-trained image set to obtain the pre-trained image encoder. This image encoder can adopt the encoder structure in the swim model structure.

[0142] In one possible implementation, the above-mentioned "pre-training the image encoder based on a pre-trained image set" may include:

[0143] The pre-training text image is divided into multiple image blocks, and multiple image blocks are used to form an image block sequence; at least one image block in the image block sequence is occluded to obtain the training input image block sequence.

[0144] The image encoder determines the occluded image patch features based on the training input image patch sequence; the image decoder then decodes the predicted image patch based on the occluded image patch features.

[0145] Based on the predicted image patches and the image patches occluded by the occlusion process, the image coding loss is determined; based on the image coding loss, the image encoder is pre-trained.

[0146] During the pre-training process based on the pre-trained image set, you can refer to... Figure 7 , Figure 7 This is a schematic diagram of the structure for pre-training an image encoder according to an embodiment of this application. The size of the pre-training text images in the pre-training image set can be 768*768. This application does not specifically limit the size of the pre-training text images. Each pre-training text image in the pre-training image set can be divided into multiple image blocks according to a preset size. Then, the segmented image blocks can be used to form an image block sequence. Further, at least one image block can be randomly selected from the image block sequence for occlusion processing, thus obtaining a training input image block sequence composed of occluded and unoccluded image blocks. The training input image block sequence is used as input data when training the image encoder. The occlusion processing methods may include, but are not limited to, modifying the pixel values ​​of the image blocks, and replacing image blocks with other images through image synthesis technology. This application does not specifically limit the occlusion processing methods.

[0147] After determining the training input image patch sequence, each image patch in the sequence can be image-encoded using an image encoder (Swin Endoder). This yields the corresponding image patch sequence features, which are composed of the individual image features of each image patch in the training input image patch sequence. Since the training input image patch sequence includes occluded image patches, the occluded image patch features can be determined from the image patch sequence features by referring to the position of the occluded image patches in the training input image patch sequence. Occluded image patch features refer to the output result obtained by image encoding occluded image patches using the image encoder, and they are used to characterize the image information contained in the occluded image patches.

[0148] The image encoder's process of encoding the training input image block sequence is the same as the process of encoding the pre-training image by the image encoder described above, and will not be repeated here.

[0149] Then, the occluded image patch features can be input into the image decoder for decoding, which can then produce the predicted image patch. The predicted image patch is the original image content of the predicted occluded image patch. This image decoder can use the Masked Image Modeling (MIM) head region, which is used to output the predicted image patch based on the received occluded image patch features.

[0150] Furthermore, the image coding loss can be determined based on the difference between the predicted image patch and the image patch occluded by the occlusion process. Finally, the image encoder can be pre-trained based on this image coding loss.

[0151] Image coding loss refers to the loss value calculated during the pre-training of an image encoder based on a sequence of input image blocks.

[0152] After identifying the predicted image patches, the image coding loss can be determined based on the image patches that have undergone occlusion processing (the image patches before occlusion) and the predicted image patches in the image patch sequence. That is, the loss value, i.e., the image coding loss, can be obtained by calculating the difference between the occluded image patches and the predicted image patches in the image patch sequence using a loss function. Then, the image encoder can be pre-trained based on this image coding loss. This involves optimizing the image encoder's performance by continuously adjusting its parameters with the goal of reducing the image coding loss, thereby improving the image encoder's accuracy.

[0153] It should be understood that in practical applications, a training termination condition can be preset. When the training of the image encoder reaches this termination condition, the training of the image encoder can be considered to have ended. This termination condition could be, for example, the number of training epochs for the image encoder reaching a preset epoch threshold; or it could be testing the performance of the image encoder and finding that its performance reaches a preset performance standard (such as reaching a preset accuracy); or it could be testing the performance of the image encoder and finding that its performance no longer significantly improves with further training. This application does not impose any limitations on the training termination condition in this embodiment.

[0154] As can be seen, the above method first divides the pre-training image into multiple image blocks, forming an image block sequence. Then, at least one image block can be randomly selected from this sequence for occlusion processing. The occluded and unoccluded image blocks then form the training input image block sequence. Subsequently, the image encoder can be pre-trained based on this training input image block sequence, enabling it to learn the ability to predict occluded image blocks and thus improving its image coding capability.

[0155] After pre-training the image encoder, the pre-trained image encoder and text decoder can be used to form a text recognition model to be trained, which can then be pre-trained through step S202 described above. It should be noted that the text decoder in the text recognition model to be trained can be a pre-trained decoder structure from the mBart model; that is, the mBart model is a model trained using a large number of training samples. In this embodiment, the decoding structure in the mBart model can be directly called as the text decoder in the text recognition model to be trained. Therefore, using an image encoder and text decoder with superior performance to form the text recognition model to be trained improves the pre-training efficiency of the text recognition model.

[0156] As can be seen, the above method can be used to pre-train the image encoder based on the pre-trained image set to improve the image information extraction capability of the image encoder. Thus, the pre-trained image encoder and text decoder can be combined to form a text recognition model to be trained.

[0157] Finally, the model training method provided in the embodiments of this application will be described in its entirety by way of example, which can be referred to as follows: Figure 8 , Figure 8 This is a schematic diagram of the overall structure of the model training method provided in this application embodiment. For example, 100 million images including text can be obtained from a database as pre-training text images. The image encoder is then pre-trained based on a pre-training image set composed of multiple pre-training text images to obtain a pre-trained image encoder. The pre-trained image encoder and text decoder can be combined to form a text recognition model to be trained.

[0158] Then, a public dataset of 13 million samples (the first subset of the pre-training sample set) can be obtained from the database to perform the first stage of pre-training on the text recognition model to be trained, resulting in an intermediate text recognition model. Through this first stage of pre-training, the intermediate text recognition model can learn the recognition capabilities of the global text recognition task. Furthermore, a business dataset of 11 million samples (the second subset of the pre-training sample set) can be obtained from the database to perform the second stage of pre-training on the intermediate text recognition model, resulting in the pre-trained text recognition model.

[0159] Finally, a fine-tuned dataset of 100,000 manually annotated data points can be obtained from the database. This dataset can be used to fine-tune the pre-trained text recognition model, resulting in the final target text recognition model for performing paragraph text recognition tasks.

[0160] Furthermore, this application embodiment also evaluated the recognition performance of the target text recognition model. The recognition performance (F1 score) for paragraph boxes improved from the initial 0.886 to 0.921, and the recognition performance (F1 score) for line granularity improved from the initial 0.652 to 0.676. For example, refer to Table 2, which includes the recognition results obtained by performing paragraph text recognition tasks using existing text recognition models and the recognition results obtained by performing paragraph text recognition tasks using the target text recognition model.

[0161] Table 2

[0162] Models Detect F1 Identify F1 Open source text recognition model 0.904 0.690 Target text recognition model 0.913 0.709

[0163] Therefore, the target text recognition model trained using the above model training method achieves significantly improved recognition accuracy in paragraph text recognition tasks, and can thus be used in practical applications to perform paragraph text recognition tasks. For example, refer to... Figure 9 , Figure 9 This is a comparative diagram illustrating the recognition effects provided in the embodiments of this application. Figure 9 Interface 'a' in the image shows the recognition result obtained by performing a paragraph text recognition task using a text recognition model based on existing technology. It is evident that the paragraph box position is inaccurate in this interface. Meanwhile... Figure 9 Interface b in the example shows the recognition result obtained by performing a paragraph text recognition task using the target text recognition model provided in the embodiment of this application. It can be seen that the target text recognition model trained by the embodiment of this application can accurately identify each paragraph of text in the input image and its respective position information.

[0164] This application also provides a text recognition method, which uses a target text recognition model trained by the above-described model training method to perform a paragraph text recognition task, that is, to recognize each paragraph of text in the input image and its respective position information. See also... Figure 10 , Figure 10 This is a flowchart illustrating the text recognition method provided in an embodiment of this application.

[0165] S1001: Obtain the image to be recognized; the image to be recognized includes at least one piece of text.

[0166] The image to be recognized is the object of the paragraph text recognition task. The image to be recognized is an image that includes at least one paragraph of text. In this regard, this application does not specifically limit the content of the image to be recognized.

[0167] As an example, the image to be identified can be pre-stored in a database, and the server can retrieve the image from the database.

[0168] As another example, the image to be recognized can also be an image uploaded to the server by a terminal device. For instance, if the terminal device runs an application with paragraph text recognition capabilities, the user can use this application to select an image stored locally on the terminal device to apply the paragraph text recognition function. Accordingly, in response to the above operation, the terminal device can upload the image to be recognized to the server, thereby allowing the server to obtain the user-uploaded image from the terminal device. This application does not specifically limit the method of obtaining the image to be recognized.

[0169] S1002: Using the target text recognition model, determine the text segments in the image to be recognized and their respective location information based on the image to be recognized.

[0170] The target text recognition model is a text recognition model trained using the model training method described above, and it is used to perform paragraph text recognition tasks.

[0171] For reference Figure 11 , Figure 11 This is a schematic diagram of the text recognition method provided in the embodiments of this application. After obtaining the image to be recognized, the image to be recognized can be input into the target text recognition model. By performing the paragraph text recognition task through the target text recognition model, the text of each paragraph in the image to be recognized and its respective position information can be obtained from the target text recognition model output.

[0172] The text segments in the image to be recognized, along with their respective location information, can include the position information of the paragraph boxes and the reading direction. In other words, the target text recognition model can output the coordinates of the paragraph boxes' location points, the angle of each paragraph box, and the reading direction. Therefore, when performing paragraph text recognition tasks using the target recognition model, it can not only accurately identify each text segment but also accurately identify relevant information such as the coordinates of the paragraph boxes' location points, the angle of each paragraph box, and the reading direction.

[0173] In one possible implementation, the step S1002 above, "determining each segment of text in the image to be recognized and its respective location information based on the image to be recognized using a target text recognition model," may include:

[0174] The image encoder in the target text recognition model encodes the image to be recognized, thus obtaining the features of the image to be recognized.

[0175] The text decoder in the target text recognition model decodes the text of each segment in the image to be recognized and its location information based on the features of the image to be recognized and the task prompts for the paragraph text recognition task.

[0176] It should be understood that the model structure of the target text recognition model is the same as that of the model during training. Therefore, the image encoder in the target text recognition model can be used to encode the image to be recognized, thereby obtaining the features of the image to be recognized. The features of the image to be recognized refer to the output result obtained by image encoding the image to be recognized through the image encoder, which is used to characterize the global image information of the image to be recognized. The image encoding process through the image encoder is the same as the image encoding process during the training of the target text recognition model, and will not be repeated here.

[0177] Then, the features of the image to be recognized and the task prompts corresponding to the paragraph text recognition task can be input into the text decoder in the target text recognition model for decoding processing, thereby obtaining the text segments in the image to be recognized and their respective position information. The decoding process through the text decoder is the same as the decoding process of the text decoder in the training process of the target text recognition model, and will not be repeated here.

[0178] As can be seen, the above method can first encode the image to be recognized by the image encoder in the target text recognition model to extract image features from the image to be recognized. Then, the text decoder can perform decoding based on the image features to be recognized and the task prompt information corresponding to the paragraph text recognition task, so as to accurately identify each paragraph of text in the image to be recognized and its respective position information.

[0179] As can be seen, the end-to-end target text recognition model trained by the above model training method can perform paragraph text recognition tasks. Since the target text recognition model has high recognition accuracy and can complete paragraph text recognition tasks independently without cooperating with other models, it can accurately identify each paragraph of text in the image to be recognized and its respective location information.

[0180] As an example, the text recognition method described above can be applied to translation scenarios. For instance, refer to... Figure 12 , Figure 12 This is a schematic diagram of an interface for a translation scenario provided in an embodiment of this application. A user can use a terminal device to take a picture of the content to be translated. After the terminal device captures an image containing the content to be translated, it can apply the aforementioned text recognition method to identify each segment of text in the image and its respective location information. Then, based on each segment of text in the image and its respective location information, a translation task can be performed. That is, the image captured by the terminal device can be used as the image to be recognized in the text recognition method. Then, the target text recognition model can determine each segment of text in the image to be recognized and its respective location information based on the image to be recognized. Further, the terminal device can perform a translation task based on each segment of text in the image to be recognized and its respective location information, thereby outputting the translation content corresponding to each segment of text in the image to be recognized.

[0181] As another example, see [link to relevant documentation] Figure 13 , Figure 13 This is an interface illustration for another translation scenario provided in this application embodiment. When a user wants to translate text in an image received in a session, or when a user wants to translate text in a locally stored image, they can refer to... Figure 13 The interface shown allows users to trigger a translation operation on the currently displayed image by clicking the translation control. In response, the terminal device uses a text recognition method to identify the text segments and their respective positions within the image, and then performs a translation task based on these text segments and their positions. That is, the image targeted by the triggered translation operation can be used as the image to be recognized in the text recognition method. Then, a target text recognition model can determine the text segments and their positions within the image to be recognized. Furthermore, the terminal device can perform a translation task based on the text segments and their positions within the image to be recognized, thereby outputting the translated content corresponding to each text segment. This application does not specifically limit the application scenario of the text recognition method.

[0182] Based on the model training method provided in the preceding embodiments, this application also provides a corresponding model training apparatus. The following, in conjunction with... Figure 14 To explain, Figure 14 This is a schematic diagram of the structure of the model training device 1400 provided in the embodiments of this application. The device includes:

[0183] The first sample acquisition module 1401 is used to acquire a pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks, the various pre-training tasks are used to determine the text information recognition results of different dimensions, and the pre-training samples include pre-training images and the annotation recognition results of the pre-training images in the corresponding pre-training tasks;

[0184] The pre-training module 1402 is used to pre-train the text recognition model to be trained based on the pre-training sample set to obtain the pre-trained text recognition model; the text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information to obtain the recognition result of the input image in the text information recognition task.

[0185] The second sample acquisition module 1403 is used to acquire a fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to the paragraph text recognition task, and the fine-tuning samples include fine-tuning images and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task;

[0186] The fine-tuning module 1404 is used to fine-tune the pre-trained text recognition model based on the fine-tuning sample set to obtain a target text recognition model for performing the paragraph text recognition task.

[0187] Optionally, the multiple pre-training tasks include at least two of the paragraph text recognition task, local paragraph recognition task, and paragraph position recognition task;

[0188] The paragraph text recognition task is used to identify each paragraph of text in the input image and its respective position information; the local paragraph recognition task is used to identify the text at the given paragraph position in the input image; the paragraph position recognition task is used to identify the start character, end character and position information of each paragraph of text in the input image.

[0189] Optionally, the pre-training sample set includes a first sub-sample set and a second sub-sample set, wherein the first sub-sample set includes pre-training samples corresponding to the global text recognition task, and the second sub-sample set includes pre-training samples corresponding to the paragraph-level text recognition task.

[0190] The pre-training module 1402 includes:

[0191] The first pre-training unit is used to perform a first-stage pre-training on the text recognition model to be trained based on the first subsample set to obtain an intermediate text recognition model.

[0192] The second pre-training unit is used to perform a second stage of pre-training on the intermediate text recognition model based on the second sub-sample set, so as to obtain the pre-trained text recognition model.

[0193] Optionally, the global text recognition task is used to recognize all text in the input image without distinguishing between paragraphs;

[0194] The paragraph-level text recognition task includes at least one of the paragraph text recognition task, the local paragraph recognition task, and the paragraph position recognition task; the paragraph text recognition task is used to recognize each paragraph of text in the input image and its respective position information; the local paragraph recognition task is used to recognize the text at the given paragraph position in the input image; the paragraph position recognition task is used to recognize the start character, end character, and position information of each paragraph of text in the input image.

[0195] Optionally, in the pre-training samples corresponding to the paragraph text recognition task, the annotation recognition result includes each paragraph of text in the pre-training image, as well as the paragraph box position information and reading direction corresponding to each paragraph of text. The paragraph box position information includes the coordinate information of the box positioning point and the paragraph box angle.

[0196] The pre-training samples corresponding to the local paragraph recognition task also include task prompt information for the local paragraph recognition task. The task prompt information includes the position information of a given paragraph box. The annotation recognition result in the pre-training samples includes the text within the given paragraph box in the pre-training image.

[0197] In the pre-training samples corresponding to the paragraph position recognition task, the annotation recognition results include the start and end characters of each paragraph in the pre-training image, as well as the paragraph box position information and reading direction of each paragraph.

[0198] Optionally, the pre-training module 1402 includes:

[0199] The first input unit is used to input the pre-training images in the pre-training samples and the task prompt information indicating the pre-training task corresponding to the pre-training samples into the text recognition model to be trained.

[0200] The first image encoding unit is used to encode the pre-trained image using the image encoder in the text recognition model to be trained, so as to obtain the pre-trained image features.

[0201] The first decoding unit is used to decode the pre-trained image in the pre-training task by using the text decoder in the text recognition model to be trained, based on the pre-trained image features and the task prompt information;

[0202] The first training unit is used to determine the pre-training loss based on the labeled recognition results and the predicted recognition results in the pre-training samples; and to pre-train the text recognition model to be trained based on the pre-training loss.

[0203] Optionally, in the fine-tuning sample corresponding to the paragraph text recognition task, the annotation recognition result includes each paragraph of text in the fine-tuning image, as well as the paragraph box position information and reading direction corresponding to each paragraph of text. The paragraph box position information includes the coordinate information of the box positioning point and the paragraph box angle.

[0204] Optionally, the fine-tuning module 1404 includes:

[0205] The second input unit is used to input the fine-tuning image in the fine-tuning sample and the task prompt information indicating the paragraph text recognition task into the pre-trained text recognition model.

[0206] The second image encoding unit is used to encode the fine-tuned image using the image encoder in the pre-trained text recognition model to obtain fine-tuned image features.

[0207] The second decoding unit is used to decode the fine-tuned image in the paragraph text recognition task by using the text decoder in the pre-trained text recognition model, based on the fine-tuned image features and the task prompt information.

[0208] The second training unit is used to determine the fine-tuning loss based on the labeled recognition results and the predicted recognition results in the fine-tuned samples; and to fine-tune the pre-trained text recognition model based on the fine-tuning loss.

[0209] Optionally, the device further includes:

[0210] An acquisition unit is used to acquire a pre-trained image set; the pre-trained image set includes multiple pre-trained text images;

[0211] The third pre-training unit is used to pre-train the image encoder based on the pre-training image set to obtain the pre-trained image encoder.

[0212] The combination unit is used to assemble the text recognition model to be trained using the pre-trained image encoder and text decoder.

[0213] Optionally, the third pre-training unit includes:

[0214] A segmentation unit is used to divide the pre-trained text image into multiple image blocks, and use the multiple image blocks to form an image block sequence; occlusion processing is performed on at least one image block in the image block sequence to obtain a training input image block sequence.

[0215] The third decoding unit is used to determine the occluded image patch features based on the training input image patch sequence using the image encoder; and to decode the predicted image patch based on the occluded image patch features using the image decoder.

[0216] The third training unit is used to determine the image coding loss based on the predicted image patch and the image patch occluded by the occlusion process; and to pre-train the image encoder based on the image coding loss.

[0217] Based on the text recognition method provided in the preceding embodiments, this application also provides a text recognition device. The following, in conjunction with... Figure 15 To explain, Figure 15 This is a schematic diagram of the structure of the text recognition device 1500 provided in an embodiment of this application. The device includes:

[0218] Image acquisition module 1501 is used to acquire an image to be recognized; the image to be recognized includes at least one piece of text;

[0219] The text recognition module 1502 is used to determine each segment of text in the image to be recognized and its respective position information based on the image to be recognized using a target text recognition model; the target text recognition model is trained using the model training method provided in the foregoing embodiments.

[0220] Optionally, the text recognition module 1502 includes:

[0221] The third image encoding unit is used to encode the image to be recognized using the image encoder in the target text recognition model to obtain the features of the image to be recognized.

[0222] The fourth decoding unit is used to decode each segment of text in the image to be recognized and its respective position information based on the features of the image to be recognized and the task prompt information of the paragraph text recognition task through the text decoder in the target text recognition model.

[0223] Optionally, the location information includes the paragraph box location information and reading direction of the corresponding text segment, and the paragraph box location information includes the coordinate information of the box positioning point and the paragraph box angle.

[0224] This application also provides a computer device, which may specifically be a terminal device or a server. The terminal device and server provided in this application will be described below from the perspective of hardware implementation.

[0225] See Figure 16 , Figure 16 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 16 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, in-vehicle computers, etc. Taking a computer as an example:

[0226] Figure 16 This is a block diagram illustrating a portion of the structure of a computer associated with the terminal provided in an embodiment of this application. (Reference) Figure 16 The computer includes: a radio frequency (RF) circuit 1210, a memory 1220, an input unit 1230 (including a touch panel 1231 and other input devices 1232), a display unit 1240 (including a display panel 1241), a sensor 1250, an audio circuit 1260 (connected to a speaker 1261 and a microphone 1262), a wireless fidelity (WiFi) module 1270, a processor 1280, and a power supply 1290, etc. Those skilled in the art will understand that... Figure 16 The computer architecture shown does not constitute a limitation on the computer and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0227] The memory 1220 can be used to store software programs and modules. The processor 1280 executes various computer functions and data processing by running the software programs and modules stored in the memory 1220. The memory 1220 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer (such as audio data, telephone directory, etc.). In addition, the memory 1220 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0228] The processor 1280 is the control center of the computer, connecting various parts of the computer through various interfaces and lines. It performs various computer functions and processes data by running or executing software programs and / or modules stored in the memory 1220, and by calling data stored in the memory 1220. Optionally, the processor 1280 may include one or more processing units; preferably, the processor 1280 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 1280.

[0229] In the embodiments of this application, the processor 1280 included in the terminal is used to execute the steps in the model training method described in the foregoing embodiments, or to execute the steps in the text recognition method described in the foregoing embodiments.

[0230] See Figure 17 , Figure 17 This is a schematic diagram of the structure of a server 1300 provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1322 (e.g., one or more processors) and memory 1332, and one or more storage media 1330 (e.g., one or more mass storage devices) for storing application programs 1342 or data 1344. The memory 1332 and storage media 1330 can be temporary or persistent storage. The program stored in the storage media 1330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1322 may be configured to communicate with the storage media 1330 and execute the series of instruction operations stored in the storage media 1330 on the server 1300.

[0231] Server 1300 may also include one or more power supplies 1326, one or more wired or wireless network interfaces 1350, one or more input / output interfaces 1358, and / or one or more operating systems, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0232] The steps performed by the server in the above embodiments can be based on this Figure 17 The server structure shown is as follows. The CPU 1322 is used to execute the steps in the model training methods described in the foregoing embodiments, or to execute the steps in the text recognition methods described in the foregoing embodiments.

[0233] This application also provides a computer-readable storage medium for storing a computer program that performs steps in the model training method described in the foregoing embodiments, or performs steps in the text recognition method described in the foregoing embodiments.

[0234] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in the model training method described in the foregoing embodiments, or the steps in the text recognition method described in the foregoing embodiments.

[0235] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0236] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0237] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0238] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0239] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing computer programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0240] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0241] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0242] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / para>

Claims

1. A model training method, characterized in that, The method includes: Obtain a pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks, the various pre-training tasks are used to determine the text information recognition results of different dimensions, and the pre-training samples include pre-training images and the annotation recognition results of the pre-training images in the corresponding pre-training tasks; Based on the pre-training sample set, the text recognition model to be trained is pre-trained to obtain the pre-trained text recognition model; the text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information, and obtain the recognition result of the input image in the text information recognition task; Obtain a fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to the paragraph text recognition task, the fine-tuning samples include fine-tuning images, and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task; Based on the fine-tuning sample set, the pre-trained text recognition model is fine-tuned to obtain a target text recognition model for performing the paragraph text recognition task.

2. The method according to claim 1, characterized in that, The multiple pre-training tasks include at least two of the paragraph text recognition task, local paragraph recognition task, and paragraph position recognition task; The paragraph text recognition task is used to identify each paragraph of text in the input image and its respective position information; the local paragraph recognition task is used to identify the text at the given paragraph position in the input image; the paragraph position recognition task is used to identify the start character, end character and position information of each paragraph of text in the input image.

3. The method according to claim 2, characterized in that, The various pre-training tasks also include a global text recognition task; the global text recognition task is used to recognize all text in the input image without distinguishing between paragraphs.

4. The method according to claim 1, characterized in that, The pre-training sample set includes a first sub-sample set and a second sub-sample set. The first sub-sample set includes pre-training samples corresponding to the global text recognition task, and the second sub-sample set includes pre-training samples corresponding to the paragraph-level text recognition task. The step of pre-training the text recognition model to be trained based on the pre-trained sample set to obtain the pre-trained text recognition model includes: Based on the first subset of samples, the text recognition model to be trained is pre-trained in the first stage to obtain an intermediate text recognition model. Based on the second subsample set, the intermediate text recognition model is pre-trained in the second stage to obtain the pre-trained text recognition model.

5. The method according to claim 4, characterized in that, The global text recognition task is used to recognize all text in an input image without distinguishing between paragraphs; The paragraph-level text recognition task includes at least one of the paragraph text recognition task, the local paragraph recognition task, and the paragraph position recognition task; the paragraph text recognition task is used to recognize each paragraph of text in the input image and its respective position information; the local paragraph recognition task is used to recognize the text at the given paragraph position in the input image; the paragraph position recognition task is used to recognize the start character, end character, and position information of each paragraph of text in the input image.

6. The method according to claim 2 or 5, characterized in that, In the pre-training samples corresponding to the paragraph text recognition task, the annotation recognition results include each paragraph of text in the pre-training image, as well as the paragraph box position information and reading direction corresponding to each paragraph of text. The paragraph box position information includes the coordinate information of the box positioning point and the paragraph box angle. The pre-training samples corresponding to the local paragraph recognition task also include task prompt information for the local paragraph recognition task. The task prompt information includes the position information of a given paragraph box. The annotation recognition result in the pre-training samples includes the text within the given paragraph box in the pre-training image. In the pre-training samples corresponding to the paragraph position recognition task, the annotation recognition results include the start and end characters of each paragraph in the pre-training image, as well as the paragraph box position information and reading direction of each paragraph.

7. The method according to any one of claims 1 to 6, characterized in that, The pre-training of the text recognition model to be trained based on the pre-training sample set includes: The pre-training images in the pre-training samples, along with task prompts indicating the pre-training tasks corresponding to the pre-training samples, are input into the text recognition model to be trained. The image encoder in the text recognition model to be trained is used to encode the pre-trained image to obtain pre-trained image features. The text decoder in the text recognition model to be trained decodes the pre-trained image based on the pre-trained image features and the task prompt information to obtain the predicted recognition result of the pre-trained image in the pre-training task. Based on the labeled recognition results and the predicted recognition results in the pre-training samples, the pre-training loss is determined; based on the pre-training loss, the text recognition model to be trained is pre-trained.

8. The method according to any one of claims 1 to 7, characterized in that, In the fine-tuning sample corresponding to the paragraph text recognition task, the annotation recognition result includes each paragraph of text in the fine-tuning image, as well as the paragraph box position information and reading direction corresponding to each paragraph of text. The paragraph box position information includes the coordinate information of the box positioning point and the paragraph box angle.

9. The method according to any one of claims 1 to 8, characterized in that, The fine-tuning of the pre-trained text recognition model based on the fine-tuning sample set includes: The fine-tuned image from the fine-tuned sample, along with the task prompt information indicating the paragraph text recognition task, is input into the pre-trained text recognition model. The fine-tuned image is image encoded using the image encoder in the pre-trained text recognition model to obtain the fine-tuned image features. The text decoder in the pre-trained text recognition model decodes the fine-tuned image based on the fine-tuned image features and the task prompt information to obtain the predicted recognition result of the fine-tuned image in the paragraph text recognition task. Based on the labeled recognition results and the predicted recognition results in the fine-tuned samples, the fine-tuning loss is determined; based on the fine-tuning loss, the pre-trained text recognition model is fine-tuned.

10. The method according to any one of claims 1 to 9, characterized in that, Before pre-training the text recognition model to be trained based on the pre-trained sample set to obtain the pre-trained text recognition model, the method further includes: Obtain a pre-trained image set; the pre-trained image set includes multiple pre-trained text images; Based on the pre-trained image set, the image encoder is pre-trained to obtain the pre-trained image encoder; The pre-trained image encoder and text decoder are used to form the text recognition model to be trained.

11. The method according to claim 10, characterized in that, The pre-training of the image encoder based on the pre-trained image set includes: The pre-trained text image is divided into multiple image blocks, and the multiple image blocks are used to form an image block sequence; at least one image block in the image block sequence is occluded to obtain the training input image block sequence. The image encoder determines occluded image patch features based on the training input image patch sequence; the image decoder then decodes the predicted image patch based on the occluded image patch features. Based on the predicted image patch and the image patch occluded by the occlusion process, the image coding loss is determined; based on the image coding loss, the image encoder is pre-trained.

12. A text recognition method, characterized in that, The method includes: Obtain the image to be identified; the image to be identified includes at least one piece of text; The target text recognition model determines each segment of text in the image to be recognized and its respective location information based on the image to be recognized; the target text recognition model is trained by the model training method described in any one of claims 1 to 11.

13. The method according to claim 12, characterized in that, The step of determining each segment of text and its location information in the image to be recognized using a target text recognition model includes: The image to be recognized is encoded using the image encoder in the target text recognition model to obtain the features of the image to be recognized. The text decoder in the target text recognition model decodes each segment of text in the image to be recognized and its respective position information based on the features of the image to be recognized and the task prompts for the paragraph text recognition task.

14. The method according to claim 12 or 13, characterized in that, The location information includes the paragraph box position information and reading direction of the corresponding text segment. The paragraph box position information includes the coordinate information of the box positioning point and the paragraph box angle.

15. A model training device, characterized in that, The device includes: The first sample acquisition module is used to acquire a pre-training sample set; the pre-training sample set includes pre-training samples corresponding to various pre-training tasks, the various pre-training tasks are used to determine the text information recognition results of different dimensions, and the pre-training samples include pre-training images and the annotation recognition results of the pre-training images in the corresponding pre-training tasks; The pre-training module is used to pre-train the text recognition model to be trained based on the pre-training sample set to obtain the pre-trained text recognition model; the text recognition model is used to perform the text information recognition task indicated by the task prompt information on the input image according to the task prompt information to obtain the recognition result of the input image in the text information recognition task; The second sample acquisition module is used to acquire a fine-tuning sample set; the fine-tuning sample set includes fine-tuning samples corresponding to the paragraph text recognition task, and the fine-tuning samples include fine-tuning images and the annotation and recognition results of the fine-tuning images in the paragraph text recognition task; The fine-tuning module is used to fine-tune the pre-trained text recognition model based on the fine-tuning sample set to obtain a target text recognition model for performing the paragraph text recognition task.

16. A text recognition device, characterized in that, The device includes: An image acquisition module is used to acquire an image to be recognized; the image to be recognized includes at least one piece of text. A text recognition module is used to determine, based on the image to be recognized, each segment of text in the image to be recognized and its respective position information, using a target text recognition model; the target text recognition model is trained using the model training method described in any one of claims 1 to 11.

17. A computer device, characterized in that, The device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute, according to the computer program, the model training method of any one of claims 1 to 11, or the text recognition method of any one of claims 12 to 14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, which, when executed by an electronic device, implements the model training method of any one of claims 1 to 11, or the text recognition method of any one of claims 12 to 14.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method of any one of claims 1 to 11, or the text recognition method of any one of claims 12 to 14.

Images