Image processing method and device

Abstract

The invention relates to an image processing method and device. The method comprises the following steps: acquiring a to-be-processed image; performing text detection on the to-be-processed image to obtain a plurality of text image blocks in the to-be-processed image; performing image block synthesis processing on the plurality of text image blocks to obtain a text image; and according to the text image, identifying a language corresponding to each text image block. Therefore, by means of the plurality of text image blocks in the to-be-processed image, the language corresponding to each text image block is recognized, the accuracy of language distinguishing can be improved, the languages appearing in the to-be-processed image can be distinguished, and subsequent text recognition is prevented from being affected.

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image processing method and device. Background Art

[0002] At present, some texts in different languages ​​are highly similar. If the text in the image is not distinguished by language, it will be difficult to directly recognize the text in the image through the recognition model. Therefore, before accurately recognizing the text in the image, it is necessary to distinguish the language of the text in advance, and then enter the recognition model of the corresponding language according to the language to recognize the text in the image.

[0003] In the related technology, the existing language classification models can be roughly divided into two types: one is that it can only determine the language of the image at the entire page level. When multiple languages ​​appear in the image, it is impossible to accurately determine the remaining languages, and subsequent text recognition will be affected; the other is to determine the language of the image based on a single text image block in the image. However, if there is interference or symbols in the single text image block, the accuracy of language distinction will be affected. Summary of the invention

[0004] In order to solve the above-mentioned related technologies, the existing language classification models are roughly divided into two types: one is that the language of the image can only be determined at the entire page level. When multiple languages ​​appear in the image, it is impossible to accurately determine the remaining languages, and subsequent text recognition will be affected; the other is to determine the language of the image based on a single text image block in the image, but if there is interference or symbols in the single text image block, the accuracy of language distinction is affected. The embodiment of the present application provides an image processing method, device, electronic device and storage medium. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, an image processing method is first provided, the method comprising:

[0006] Get the image to be processed;

[0007] Performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed;

[0008] Performing image block synthesis processing on a plurality of the text image blocks to obtain a text image;

[0009] According to the text image, the language corresponding to each of the text image blocks is identified.

[0010] In an optional implementation, the performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed includes:

[0011] Performing angle correction on the image to be processed;

[0012] Text detection is performed on the image to be processed that has undergone angle correction to obtain a plurality of text image blocks in the image to be processed.

[0013] In an optional implementation, performing image block synthesis processing on the plurality of text image blocks to obtain a text image includes:

[0014] For any of the text image blocks, performing angle correction on the text image block;

[0015] An image block synthesis process is performed on the plurality of angle-corrected text image blocks to obtain a text image.

[0016] In an optional implementation, performing image block synthesis processing on the plurality of text image blocks to obtain a text image includes:

[0017] For any of the text image blocks, the following processing is performed to obtain a fourth text image block:

[0018] Performing grayscale processing on the text image block to obtain a first text image block;

[0019] Scaling the first text image block in equal proportion to obtain a second text image block, wherein the height of the second text image block is a preset height threshold;

[0020] Cropping the second text image block to obtain a third text image block;

[0021] Rotating the third text image block by a preset first angle according to a preset rotation direction to obtain a fourth text image block;

[0022] Perform image block synthesis processing on the plurality of fourth text image blocks to obtain a text image.

[0023] In an optional implementation, the step of cropping the second text image block to obtain a third text image block includes:

[0024] Determining whether the length of the second text image block is greater than a preset length threshold;

[0025] When the length of the second text image block is greater than the preset length threshold, cropping the second text image block to obtain a third text image block;

[0026] The height of the third text image block is a preset height threshold, and the length of the third text image block is the preset length threshold.

[0027] In an optional embodiment, the method further comprises:

[0028] When the length of the second text image block is equal to the preset length threshold, determining the second text image block as a third text image block;

[0029] When the length of the second text image block is less than the preset length threshold, image filling processing is performed on the second text image block to obtain a third text image block.

[0030] In an optional implementation, performing image filling processing on the second text image block to obtain a fifth text image block includes:

[0031] The second text image block is subjected to image filling processing in a horizontal direction according to a preset grayscale value to obtain a third text image block.

[0032] In an optional implementation, performing image block synthesis processing on the plurality of fourth text image blocks to obtain a text image includes:

[0033] The plurality of fourth text image blocks are spliced ​​together in a preset splicing order to obtain a text image.

[0034] In an optional implementation, the identifying, based on the text image, the language corresponding to each of the text image blocks includes:

[0035] Inputting the text image into a pre-trained recognition model to obtain a predicted probability of each language corresponding to each text image block output by the pre-trained recognition model;

[0036] Determine the language with the greatest predicted probability corresponding to each of the text image blocks as the language corresponding to each of the text image blocks;

[0037] Among them, the pre-trained recognition model includes N convolutional layers, M pooling layers, a bidirectional long short-term memory network and a fully connected layer, and N and M are both positive integers.

[0038] In an optional embodiment, inputting the text image into a pre-trained recognition model to obtain the predicted probability of each language corresponding to each text image block output by the pre-trained recognition model includes:

[0039] Performing image preprocessing on the text image, wherein the image preprocessing is used to adjust the pixel value range of the text image;

[0040] The text image that has undergone image preprocessing is input into a pre-trained recognition model to obtain the predicted probability of each language corresponding to each text image block output by the pre-trained recognition model.

[0041] In an optional embodiment, inputting the text image into a pre-trained recognition model to obtain the predicted probability of each language corresponding to each text image block output by the pre-trained recognition model includes:

[0042] Inputting the text image into a pre-trained recognition model to obtain the prediction probability of each language and the prediction probability of each text direction corresponding to each text image block output by the pre-trained recognition model;

[0043] The method further comprises:

[0044] The text direction with the largest prediction probability corresponding to each of the text image blocks is determined as the text direction corresponding to each of the text image blocks.

[0045] In an optional embodiment, the method further comprises:

[0046] Determining whether the text direction is a preset text direction;

[0047] When the text direction is a preset text direction, the image to be processed is rotated by a preset second angle, and the process jumps to the step of performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed.

[0048] In an optional embodiment, before executing the method, the method further includes:

[0049] Acquire a preset sample image set, wherein the preset sample image set includes a plurality of sample images, and each sample image in the sample images of a preset first ratio corresponds to one language, and each sample image in the sample images of a preset second ratio corresponds to a plurality of different languages, and the preset first ratio is greater than the preset second ratio;

[0050] For each sample image in the preset sample image set, the following processing is performed:

[0051] Performing text detection on the sample image to obtain a plurality of sample text image blocks in the sample image;

[0052] Performing image block synthesis processing on a plurality of the sample text image blocks to obtain a sample text image;

[0053] Inputting the sample text image into a preset recognition model, and obtaining the predicted probability of each sample language corresponding to each sample text image block output by the preset recognition model;

[0054] Determining a language error according to the predicted probability of each sample language corresponding to each sample text image block and the labeled language corresponding to each sample text image block;

[0055] The preset recognition model is trained according to the language error, and when the language error meets a preset condition, the model training is stopped to obtain the pre-trained recognition model.

[0056] In an optional implementation, performing text detection on the sample image to obtain a plurality of sample text image blocks in the sample image includes:

[0057] Performing angle correction on the sample image;

[0058] Text detection is performed on the angle-corrected sample image to obtain a plurality of sample text image blocks in the sample image.

[0059] In an optional implementation, performing image block synthesis processing on a plurality of sample text image blocks to obtain a sample text image includes:

[0060] For any of the sample text image blocks, performing angle correction on the sample text image block;

[0061] An image block synthesis process is performed on a plurality of the sample text image blocks that have undergone angle correction to obtain a sample text image.

[0062] In an optional implementation, performing image block synthesis processing on a plurality of sample text image blocks to obtain a sample text image includes:

[0063] For any of the sample text image blocks, the following processing is performed to obtain a fourth sample text image block;

[0064] Performing grayscale processing on the sample text image block to obtain a first sample text image block;

[0065] Scaling the first sample text image block in equal proportion to obtain a second sample text image block, wherein the height of the second sample text image block is a preset height threshold;

[0066] Cropping the second sample text image block to obtain a third sample text image block;

[0067] Rotating the third sample text image block by a preset first angle according to a preset rotation direction to obtain a fourth sample text image block;

[0068] Perform image block synthesis processing on the plurality of fourth sample text image blocks to obtain a sample text image.

[0069] In an optional implementation, the step of cropping the second sample text image block to obtain a third sample text image block includes:

[0070] Determining whether the length of the second sample text image block is greater than a preset length threshold;

[0071] When the length of the second sample text image block is greater than a preset length threshold, cropping the second sample text image block to obtain a third sample text image block;

[0072] The height of the third sample text image block is a preset height threshold, and the length of the third sample text image block is the preset length threshold.

[0073] In an optional embodiment, the method further comprises:

[0074] When the length of the second sample text image block is equal to a preset length threshold, determining the second sample text image block as a third sample text image block;

[0075] When the length of the second sample text image block is less than a preset length threshold, image filling processing is performed on the second sample text image block to obtain a third sample text image block.

[0076] In an optional implementation, performing image filling processing on the second sample text image block to obtain a third sample text image block includes:

[0077] The second sample text image block is subjected to image filling processing in a horizontal direction according to a preset grayscale value to obtain a third sample text image block.

[0078] In an optional implementation, performing image block synthesis processing on a plurality of the fourth sample text image blocks to obtain a sample text image includes:

[0079] The plurality of fourth sample text image blocks are spliced ​​together in a preset splicing order to obtain a sample text image.

[0080] In an optional implementation, the step of inputting the sample text image into a preset recognition model and obtaining the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model includes:

[0081] Performing image preprocessing on the sample text image, wherein the image preprocessing is used to adjust the pixel value range of the sample text image;

[0082] The sample text image that has undergone image preprocessing is input into a preset recognition model to obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model.

[0083] In an optional embodiment, in the preset sample image set, each sample image in the sample images of the preset third ratio corresponds to the first text direction, each sample image in the sample images of the preset fourth ratio corresponds to the second text direction, and the preset third ratio is the same as the preset fourth ratio;

[0084] The step of inputting the sample text image into a preset recognition model and obtaining the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model comprises:

[0085] Inputting the sample text image into a preset recognition model, obtaining the prediction probability of each sample language and the prediction probability of each sample text direction corresponding to each sample text image block output by the preset recognition model;

[0086] The step of training the preset recognition model according to the language error and stopping the model training when the language error meets a preset condition to obtain the pre-trained recognition model comprises:

[0087] Determine a text direction error according to a prediction probability of each sample text direction corresponding to each sample text image block and an annotated text direction corresponding to each sample text image block, wherein the annotated text direction is the first text direction or the second text direction;

[0088] The preset recognition model is trained according to the language error and the text direction error, and when the language error and the text direction error meet preset conditions, the model training is stopped to obtain the pre-trained recognition model.

[0089] In an optional implementation, the preset recognition model is trained according to the language error and the text direction error, and when the language error and the text direction error meet preset conditions, the model training is stopped to obtain the pre-trained recognition model, including:

[0090] When the direction of the annotated text is a preset text direction, the language error is set to a preset value;

[0091] The preset recognition model is trained according to the text direction error, and when the text direction error meets a preset condition, the model training is stopped to obtain the pre-trained recognition model.

[0092] In a second aspect of the embodiments of the present application, an image processing device is further provided, the device comprising:

[0093] An image acquisition module, used for acquiring an image to be processed;

[0094] A text detection module, used to perform text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed;

[0095] An image block synthesis module, used for performing image block synthesis processing on a plurality of the text image blocks to obtain a text image;

[0096] The language recognition module is used to recognize the language corresponding to each text image block according to the text image.

[0097] In an optional implementation, the text detection module is specifically used to:

[0098] Performing angle correction on the image to be processed;

[0099] Text detection is performed on the image to be processed that has undergone angle correction to obtain a plurality of text image blocks in the image to be processed.

[0100] In an optional implementation, the image block synthesis module is specifically used to:

[0101] For any of the text image blocks, performing angle correction on the text image block;

[0102] An image block synthesis process is performed on the plurality of angle-corrected text image blocks to obtain a text image.

[0103] In an optional implementation, the image block synthesis module specifically includes:

[0104] The image block processing submodule is used to perform the following processing on any of the text image blocks to obtain a fourth text image block:

[0105] A grayscale processing submodule, used for performing grayscale processing on the text image block to obtain a first text image block;

[0106] An image block scaling submodule, used for scaling the first text image block in equal proportion to obtain a second text image block, wherein the height of the second text image block is a preset height threshold;

[0107] An image block cropping submodule, used for cropping the second text image block to obtain a third text image block;

[0108] An image block rotation submodule, configured to rotate the third text image block by a preset first angle according to a preset rotation direction to obtain a fourth text image block;

[0109] The image block synthesis submodule is used to perform image block synthesis processing on the plurality of fourth text image blocks to obtain a text image.

[0110] In an optional implementation, the image block cropping submodule is specifically used to:

[0111] Determining whether the length of the second text image block is greater than a preset length threshold;

[0112] When the length of the second text image block is greater than the preset length threshold, cropping the second text image block to obtain a third text image block;

[0113] The height of the third text image block is a preset height threshold, and the length of the third text image block is the preset length threshold.

[0114] In an optional implementation, the image block synthesis module further includes:

[0115] an image block determination submodule, configured to determine the second text image block as a third text image block when the length of the second text image block is equal to the preset length threshold;

[0116] The image block filling submodule is used to perform image filling processing on the second text image block to obtain a third text image block when the length of the second text image block is less than the preset length threshold.

[0117] In an optional implementation, the image block filling submodule is specifically used to:

[0118] The second text image block is subjected to image filling processing in a horizontal direction according to a preset grayscale value to obtain a third text image block.

[0119] In an optional implementation, the image block synthesis submodule is specifically used to:

[0120] The plurality of fourth text image blocks are spliced ​​together in a preset splicing order to obtain a text image.

[0121] In an optional implementation, the language identification module specifically includes:

[0122] A prediction probability output submodule, used to input the text image into a pre-trained recognition model to obtain the prediction probability of each language corresponding to each text image block output by the pre-trained recognition model;

[0123] A language determination submodule, used to determine the language with the greatest predicted probability corresponding to each of the text image blocks as the language corresponding to each of the text image blocks;

[0124] Among them, the pre-trained recognition model includes N convolutional layers, M pooling layers, a bidirectional long short-term memory network and a fully connected layer, and N and M are both positive integers.

[0125] In an optional implementation, the prediction probability output submodule is specifically used for:

[0126] Performing image preprocessing on the text image, wherein the image preprocessing is used to adjust the pixel value range of the text image;

[0127] The text image that has undergone image preprocessing is input into a pre-trained recognition model to obtain the predicted probability of each language corresponding to each text image block output by the pre-trained recognition model.

[0128] In an optional implementation, the prediction probability output submodule is specifically used for:

[0129] Inputting the text image into a pre-trained recognition model to obtain the prediction probability of each language and the prediction probability of each text direction corresponding to each text image block output by the pre-trained recognition model;

[0130] The language recognition module also includes:

[0131] The text direction determination submodule is used to determine the text direction with the maximum prediction probability corresponding to each text image block as the text direction corresponding to each text image block.

[0132] In an optional embodiment, the device further comprises:

[0133] A text direction determination module, used to determine whether the text direction is a preset text direction;

[0134] The jump module is used to rotate the image to be processed by a preset second angle when the text direction is a preset text direction, and jump to the step of performing text detection on the image to be processed to obtain multiple text image blocks in the image to be processed.

[0135] In an optional embodiment, the device further comprises:

[0136] An image set acquisition module, used to acquire a preset sample image set, wherein the preset sample image set includes a plurality of sample images, and each sample image in the sample images of a preset first ratio corresponds to one language, and each sample image in the sample images of a preset second ratio corresponds to a plurality of different languages, and the preset first ratio is greater than the preset second ratio;

[0137] The sample image processing module is used to perform the following processing on each sample image in the preset sample image set:

[0138] A sample image detection module, used to perform text detection on the sample image to obtain a plurality of sample text image blocks in the sample image;

[0139] A sample text image block synthesis module, used for performing image block synthesis processing on a plurality of the sample text image blocks to obtain a sample text image;

[0140] A sample language recognition module, used for inputting the sample text image into a preset recognition model, and obtaining the predicted probability of each sample language corresponding to each sample text image block output by the preset recognition model;

[0141] A language error determination module, used to determine the language error according to the predicted probability of each sample language corresponding to each sample text image block and the labeled language corresponding to each sample text image block;

[0142] The model training module is used to train the preset recognition model according to the language error, and stop the model training when the language error meets the preset conditions to obtain the pre-trained recognition model.

[0143] In an optional implementation, the sample image detection module is specifically used to:

[0144] Performing angle correction on the sample image;

[0145] Text detection is performed on the angle-corrected sample image to obtain a plurality of sample text image blocks in the sample image.

[0146] In an optional implementation, the sample text image block synthesis module is specifically used to:

[0147] For any of the sample text image blocks, performing angle correction on the sample text image block;

[0148] An image block synthesis process is performed on a plurality of the sample text image blocks that have undergone angle correction to obtain a sample text image.

[0149] In an optional implementation, the sample text image block synthesis module specifically includes:

[0150] A sample text image block processing submodule, configured to perform the following processing on any of the sample text image blocks to obtain a fourth sample text image block;

[0151] A sample text image block grayscale processing submodule, used for performing grayscale processing on the sample text image block to obtain a first sample text image block;

[0152] A sample text image block scaling submodule, used for scaling the first sample text image block in equal proportion to obtain a second sample text image block, wherein the height of the second sample text image block is a preset height threshold;

[0153] a sample text image block cropping submodule, used for cropping the second sample text image block to obtain a third sample text image block;

[0154] A sample text image block rotation submodule, used to rotate the third sample text image block by a preset first angle according to a preset rotation direction to obtain a fourth sample text image block;

[0155] The sample text image block synthesis submodule is used to perform image block synthesis processing on a plurality of the fourth sample text image blocks to obtain a sample text image.

[0156] In an optional implementation, the sample text image block cropping submodule is specifically used to:

[0157] Determining whether the length of the second sample text image block is greater than a preset length threshold;

[0158] When the length of the second sample text image block is greater than a preset length threshold, cropping the second sample text image block to obtain a third sample text image block;

[0159] The height of the third sample text image block is a preset height threshold, and the length of the third sample text image block is the preset length threshold.

[0160] In an optional implementation, the sample text image block synthesis module further includes:

[0161] a sample text image block determining submodule, configured to determine the second sample text image block as a third sample text image block when the length of the second sample text image block is equal to a preset length threshold;

[0162] The sample text image block filling submodule is used to perform image filling processing on the second sample text image block to obtain a third sample text image block when the length of the second sample text image block is less than a preset length threshold.

[0163] In an optional implementation, the sample text image block filling submodule is specifically used to:

[0164] The second sample text image block is subjected to image filling processing in a horizontal direction according to a preset grayscale value to obtain a third sample text image block.

[0165] In an optional implementation, the sample text image block synthesis submodule is specifically used for:

[0166] The plurality of fourth sample text image blocks are spliced ​​together in a preset splicing order to obtain a sample text image.

[0167] In an optional implementation, the sample language recognition module is specifically used to:

[0168] Performing image preprocessing on the sample text image, wherein the image preprocessing is used to adjust the pixel value range of the sample text image;

[0169] The sample text image that has undergone image preprocessing is input into a preset recognition model to obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model.

[0170] In an optional embodiment, in the preset sample image set, each sample image in the sample images of the preset third ratio corresponds to the first text direction, each sample image in the sample images of the preset fourth ratio corresponds to the second text direction, and the preset third ratio is the same as the preset fourth ratio;

[0171] The sample language recognition module is specifically used for:

[0172] Inputting the sample text image into a preset recognition model, obtaining the prediction probability of each sample language and the prediction probability of each sample text direction corresponding to each sample text image block output by the preset recognition model;

[0173] The model training module specifically includes:

[0174] A text direction error determination submodule, used to determine a text direction error according to a prediction probability of each sample text direction corresponding to each sample text image block and an annotated text direction corresponding to each sample text image block, wherein the annotated text direction is the first text direction or the second text direction;

[0175] The model training submodule is used to train the preset recognition model according to the language error and the text direction error, and stop the model training when the language error and the text direction error meet the preset conditions to obtain the pre-trained recognition model.

[0176] In an optional embodiment, the model training submodule is specifically used for:

[0177] When the direction of the annotated text is a preset text direction, the language error is set to a preset value;

[0178] The preset recognition model is trained according to the text direction error, and when the text direction error meets a preset condition, the model training is stopped to obtain the pre-trained recognition model.

[0179] In a third aspect of the embodiments of the present application, there is further provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0180] Memory, used to store computer programs;

[0181] The processor is used to implement any image processing method described in the first aspect when executing the program stored in the memory.

[0182] In a fourth aspect of the embodiments of the present application, a storage medium is further provided, wherein instructions are stored in the storage medium, and when the storage medium is run on a computer, the computer executes the image processing method described in any one of the first aspects above.

[0183] In a fifth aspect of the embodiments of the present application, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned image processing methods.

[0184] The technical solution provided by the embodiment of the present application obtains an image to be processed, performs text detection on the image to be processed, obtains multiple text image blocks in the image to be processed, performs image block synthesis processing on the multiple text image blocks, obtains a text image, and identifies the language corresponding to each text image block based on the text image. By performing text detection on the image to be processed, multiple text image blocks in the image to be processed are obtained, and image block synthesis processing is performed to obtain a text image, thereby identifying the language corresponding to each text image block based on the text image. In this way, with the help of multiple text image blocks in the image to be processed, the language corresponding to each text image block can be identified, which can improve the accuracy of language distinction, and can distinguish the language appearing in the image to be processed to avoid affecting subsequent text recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0185] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0186] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0187] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0188] Figure 1 A schematic diagram of an implementation flow of an image processing method shown in an embodiment of the present application;

[0189] Figure 2 A schematic diagram of an image to be processed shown in an embodiment of the present application;

[0190] Figure 3 It is a schematic diagram of an implementation flow of another image processing method shown in an embodiment of the present application;

[0191] Figure 4 A schematic diagram of a text image shown in an embodiment of the present application;

[0192] Figure 5 A schematic diagram of a text recognition result of an image to be processed shown in an embodiment of the present application;

[0193] Figure 6 A schematic diagram of an implementation flow of a model training method shown in an embodiment of the present application;

[0194] Figure 7 A schematic diagram of the structure of an image processing device shown in an embodiment of the present application;

[0195] Figure 8 It is a schematic diagram of the structure of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0196] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0197] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0198] like Figure 1 FIG. 1 is a schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0199] S101, obtaining an image to be processed.

[0200] In an embodiment of the present application, an image to be processed can be obtained, wherein the image to be processed includes text, and the language of the text can be any current language, such as simplified Chinese, traditional Chinese, etc., which can include one or more languages.

[0201] For example, Figure 2 The image shown can be used as the image to be processed, thereby obtaining the image to be processed, wherein, Figure 2 The image shown includes text, and the language of the text is Sanskrit.

[0202] It should be noted that, for the image to be processed, the text included therein may correspond to a single language, such as simplified Chinese, or may correspond to different languages, such as simplified Chinese and English, etc., and this embodiment of the present application does not limit this.

[0203] S102, performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed.

[0204] In an embodiment of the present application, for an image to be processed, text detection may be performed on the image to be processed to obtain multiple text image blocks in the image to be processed, wherein the text image block includes text and is a part of the image to be processed, which may be referred to as a text strip.

[0205] Among them, text detection can be performed on the image to be processed through a text detection algorithm to locate multiple text areas in the image to be processed, and for each text area, the image to be processed is cropped to obtain a corresponding text image block, so that multiple text image blocks can be obtained.

[0206] It should be noted that, for the text detection algorithm, reference can be made to the existing text detection algorithm, and the embodiments of the present application will not be described in detail here.

[0207] S103, performing image block synthesis processing on multiple text image blocks to obtain a text image.

[0208] In an embodiment of the present application, for multiple text image blocks, image block synthesis processing can be performed on the multiple text image blocks to obtain a text image, which means integrating the multiple text image blocks into a complete text image.

[0209] It should be noted that the text image is integrated from multiple text image blocks, including the text corresponding to each text image block, and generally does not contain non-text content, which is not limited in the embodiments of the present application.

[0210] S104: Identify the language corresponding to each text image block according to the text image.

[0211] In the embodiment of the present application, for a text image, the language corresponding to each text image block can be identified based on the text image, which means that the language corresponding to each text image block is identified with the help of multiple text image blocks.

[0212] It should be noted that the language corresponding to each text image block can be the same, for example Figure 2 As shown, the language corresponding to each text image block is identified as Sanskrit. Of course, the language corresponding to each text image block may also be different, and this embodiment of the present application does not limit this.

[0213] Through the above description of the technical solution provided by the embodiment of the present application, text detection is performed on the image to be processed to obtain multiple text image blocks in the image to be processed, and image block synthesis processing is performed to obtain a text image, so that the language corresponding to each text image block is identified based on the text image. In this way, with the help of multiple text image blocks in the image to be processed, the language corresponding to each text image block is identified, which can improve the accuracy of language distinction, and can distinguish the language appearing in the image to be processed to avoid affecting subsequent text recognition.

[0214] In addition, for the text direction in the image, without considering the text tilt, there are still four directions: upward, downward, left, and right. How to distinguish upward and downward is still a technical difficulty. The existing solution separates the text direction and language classification models, which not only increases the model size, but also increases the time consumption of recognition.

[0215] To this end, before accurately identifying the text in the image, the embodiment of the present application can use a model to screen and distinguish the languages ​​of multiple text image blocks in the image, and at the same time determine the orientation of the text, thereby improving the accuracy of text recognition. Figure 3 FIG. 1 is a schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0216] S301, obtaining an image to be processed.

[0217] In the embodiment of the present application, this step is similar to the above-mentioned step S101, and the embodiment of the present application will not be described one by one here.

[0218] S302, performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed.

[0219] In the embodiment of the present application, for the image to be processed, text detection is performed on the image to be processed to obtain multiple text image blocks in the image to be processed. The image to be processed needs to be angle corrected, and the text orientation involved in the image to be processed after angle correction is only in two directions: upward and downward.

[0220] This means that the image to be processed is firstly angle-corrected, wherein the algorithm used for the angle correction can refer to an existing algorithm, and then text detection is performed on the image to be processed after the angle correction to obtain a plurality of text image blocks in the image to be processed.

[0221] S303: For any text image block, perform the following processing to obtain a fourth text image block.

[0222] In the embodiment of the present application, for any text image block, the following processing is performed to obtain a fourth text image block, which means that for each text image block, steps S304 to S307 are performed to obtain a corresponding fourth text image block.

[0223] In addition, it should be noted that in the above step S302, the image to be processed does not need to be subjected to angle correction. Instead, angle correction is performed for each text image block after text detection.

[0224] Based on this, for any text image block, the text image block is angle corrected, wherein the algorithm used for the angle correction can refer to the existing algorithm, and then multiple text image blocks that have undergone angle correction are subjected to image block synthesis processing to obtain a text image, which means that for any text image block that has undergone angle correction, steps S304 to S307 are executed to obtain the corresponding fourth text image block, and then multiple fourth text image blocks are subjected to image block synthesis processing to obtain a text image.

[0225] S304, graying the text image block to obtain a first text image block.

[0226] In the embodiment of the present application, for a text image block, the text image block may be grayed to obtain a first text image block, where the first text image block is a grayscale text image block.

[0227] S305 , scaling the first text image block in equal proportion to obtain a second text image block, wherein the height of the second text image block is a preset height threshold.

[0228] In an embodiment of the present application, for a first text image block, the first text image block can be scaled proportionally to obtain a second text image block, wherein the height of the second text image block is a preset height threshold, which means that the first text image block is scaled proportionally to a fixed height.

[0229] For example, for a first text image block, the first text image block may be scaled proportionally to obtain a second text image block, and the height of the second text image block is 32, which means that the first text image block is scaled proportionally to a fixed height H=32.

[0230] S306: crop the second text image block to obtain a third text image block.

[0231] In the embodiment of the present application, the second text image block may be cropped to obtain a third text image block. Here, the second text image block needs to be cropped to a fixed size to obtain the third text image block.

[0232] To this end, it can be determined whether the length of the second text image block is greater than a preset length threshold. If the length of the second text image block is greater than the preset length threshold, the second text image block is cropped to obtain a third text image block. If the length of the second text image block is equal to the preset length threshold, the second text image block is determined as the third text image block. If the length of the second text image block is less than the preset length threshold, the second text image block is image-filled to obtain a third text image block. The height of the third text image block is the preset height threshold, and the length of the third text image block is the preset length threshold.

[0233] For example, it is determined whether the length of the second text image block is greater than 256. If the length of the second text image block is greater than 256, the second text image block is cropped to obtain a third text image block, where the height of the third text image block is 32 and the length is 256. If the length of the second text image block is equal to 256, the second text image block is determined as the third text image block. If the length of the second text image block is less than 256, the second text image block is subjected to image filling processing to obtain a third text image block.

[0234] The second text image block can be horizontally filled with a preset grayscale value (e.g., 255) to obtain a third text image block. This means that for the second text image block, the third text image block is cropped to a fixed size (e.g., 32 in height and 256 in length), and the second text image block whose length is less than 256 is filled with pixels, such as 255, to obtain the third text image block.

[0235] S307: Rotate the third text image block by a preset first angle according to a preset rotation direction to obtain a fourth text image block.

[0236] In the embodiment of the present application, for the third text image block, the third text image block can be rotated by a preset first angle according to a preset rotation direction to obtain a fourth text image block. For example, the third image block is rotated 90 degrees clockwise to obtain a fourth text image block. Here, the third image block is rotated 90 degrees clockwise to meet the requirements of the subsequent model for the input image.

[0237] It should be noted that for any text image block, it is grayscaled, proportionally scaled to a fixed height (for example, H=32), and cropped to a fixed size (for example, height 32, length 256). The insufficient length is filled with pixels, for example, 255, and then rotated 90 degrees clockwise to obtain the final fourth text image block.

[0238] S308, performing image block synthesis processing on the plurality of fourth text image blocks to obtain a text image.

[0239] In the embodiment of the present application, for the plurality of fourth text image blocks, the plurality of fourth text image blocks may be subjected to image block synthesis processing to obtain a text image, wherein the plurality of fourth text image blocks may be spliced ​​in a preset splicing order to obtain a text image.

[0240] For example, assuming there are m fourth text image blocks, the m fourth text image blocks are spliced ​​in a splicing order from left to right to obtain a text image of size (256, 32*m), which is the input of the model in the embodiment of the present application.

[0241] For example, Figure 2 The image to be processed shown in FIG. 1 is detected by text to obtain 16 text image blocks, wherein each of the first 12 lines is a separate text image block, and the 13th line is divided into 4 text image blocks. After the above steps S304 to S307, 16 fourth text image blocks are obtained. These 16 fourth text image blocks are spliced ​​from left to right to obtain a completed text image, as shown in FIG. Figure 4 shown.

[0242] S309, inputting the text image into the pre-trained recognition model to obtain the prediction probability of each language corresponding to each text image block and the prediction probability of each text direction output by the pre-trained recognition model.

[0243] S310: Determine the language with the highest prediction probability corresponding to each text image block as the language corresponding to each text image block.

[0244] S311 , determining the text direction with the largest prediction probability corresponding to each text image block as the text direction corresponding to each text image block.

[0245] In an embodiment of the present application, for a text image, the text image can be input into a pre-trained recognition model to obtain the predicted probability of each language corresponding to each text image block output by the pre-trained recognition model, and the language with the largest predicted probability corresponding to each text image block is determined as the language corresponding to each text image block.

[0246] Among them, the text direction can also be obtained at the same time as the language, that is, the text image is input into the pre-trained recognition model, and the prediction probability of each language and each text direction corresponding to each text image block output by the pre-trained recognition model are obtained, and the text direction with the largest prediction probability corresponding to each text image block is determined as the text direction corresponding to each text image block.

[0247] For text images, they can be preprocessed and then input into a pretrained recognition model, that is, the text images are preprocessed, wherein the image preprocessing is used to adjust the pixel value range of the text image, and the text images after image preprocessing are input into the pretrained recognition model to obtain the prediction probability of each language corresponding to each text image block output by the pretrained recognition model, as well as the prediction probability of each text direction.

[0248] For example, for a text image, its pixel value range is 0 to 255. The text image is preprocessed (similar to normalization processing), and its pixel value range is transformed to -1 to 1. The text image after image preprocessing is input into the pre-trained recognition model to obtain the prediction probability of each language corresponding to each text image block output by the pre-trained recognition model, as well as the prediction probability of each text direction.

[0249] It should be noted that the output of the pre-trained recognition model is usually expressed as m*(C+2), where m refers to the number of text image blocks, and m specifically refers to m text image blocks such as text image block 1, text image block 2, text image block 3, ..., text image block m, C refers to the number of language categories, and C specifically refers to C languages ​​such as language C1, language C2, language C3, ..., language Cc, 2 represents the number of text orientation categories, and 2 specifically refers to 2 text orientations such as 0 or 1, where 0 indicates that the image to be processed is placed forward, that is, the text orientation is upward, and 1 indicates that the image to be processed is inverted, that is, the text orientation is downward. Therefore, the output of the pre-trained recognition model can be specifically represented by an m-order matrix, as shown in Table 1 below, where each row represents the prediction probability of each language corresponding to a text image block, and the prediction probability of each text direction.

[0250]

[0251] Table 1

[0252] It can be seen from Table 1 above that m*(C+2) can be specifically represented by an m-order matrix, where each row represents the prediction probability of each language corresponding to a text image block, and the prediction probability of each text direction. Therefore, according to m*(C+2), the prediction probability of each language corresponding to each text image block, and the prediction probability of each text direction can be known, so that the language with the largest prediction probability corresponding to each text image block is determined as the language corresponding to each text image block, and the text direction with the largest prediction probability corresponding to each text image block is determined as the text direction corresponding to each text image block.

[0253] For the pre-trained recognition model, it usually includes N convolutional layers, M pooling layers, a bidirectional long short-term memory network and a fully connected layer, where N and M are both positive integers, and there is also an activation function in each convolutional layer. As shown in Table 2 below, a pre-trained recognition model shown in an embodiment of the present application includes 6 convolutional layers, 4 pooling layers, a bidirectional long short-term memory network and a fully connected layer.

[0254] Layer Name Convolution kernel size, step size and number Conv0 (convolutional layer 0) + Relu0 (activation function 0) k = 3*3s = 1 / 16 MaxPool0 (Pooling layer 0) k = 2 * 2s = 2 Conv1 (convolutional layer 1) + Relu1 (activation function 1) k = 3*3s = 1 / 32 MaxPool1 (Pooling layer 1) k = 2 * 2s = 2 Conv2 (convolutional layer 2) + Relu2 (activation function 2) k = 3*3s = 1 / 64 Conv3 (convolution layer 3) + Relu3 (activation function 3) k = 3*3s = 1 / 64 MaxPool2 (Pooling layer 2) k = 2 * 2s = 2 Conv4 (convolutional layer 4) + Relu4 (activation function 4) k = 3*3s = 1 / 128 Conv5 (convolution layer 5) + Relu5 (activation function 5) k = 3*3s = 1 / 128 MaxPool3 (Pooling layer 3) k = 2 * 2s = 2 Conv6 (convolutional layer 6) + Relu6 (activation function 6) k = 2*2s = 2*1 / 128 MaxPool4 (Pooling layer 4) k=8*2s=1*2 Bidirectional LSTM (Bidirectional Long Short-Term Memory Network) Linear (fully connected layer)

[0255] Table 2

[0256] For the above pre-trained recognition model, the 6 convolutional layers and 4 pooling layers are responsible for feature extraction. The downsampling ratio has been specially designed to be (256, 32). The bidirectional long short-term memory network can make full use of the features between different text image blocks and compare their features to improve the accuracy of language judgment. The fully connected layer can output the predicted probability of each language corresponding to each text image block, as well as the predicted probability of each text direction. The volume of the above pre-trained recognition model is only 2M, and the inference speed is very fast.

[0257] In addition, in the embodiment of the present application, for the text direction, if the text direction is positive, that is, upward, no flipping is required. If the text direction is inverted, that is, downward, it needs to be rotated 180 degrees and then the reasoning process of the pre-trained recognition model is performed again. Based on this, it is determined whether the text direction is the preset text direction; if the text direction is the preset text direction, the image to be processed is rotated by the preset second angle, and the process jumps to step S302 to perform the reasoning process of the pre-trained recognition model again.

[0258] Among them, by executing the inference process of the pre-trained recognition model again, the predicted probability of each language corresponding to each text image block can be obtained again, and the language with the largest predicted probability corresponding to each text image block is determined as the language corresponding to each text image block, which can be compared with the language corresponding to each text image block obtained for the first time. If the languages ​​recognized twice are consistent, no special processing is required. If the languages ​​recognized twice are inconsistent, the predicted probability of the language corresponding to each text image block obtained for the first time and the predicted probability of the language corresponding to each text image block obtained again can be compared, and the language with the largest predicted probability can be selected as the language corresponding to each text image block.

[0259] After the above processing, the language of each text image block in the image to be processed can be known, and then the recognition model of the corresponding language can be entered according to the language to recognize the text in the image to be processed. The recognition result is as follows: Figure 5 shown.

[0260] Through the above description of the technical solution provided by the embodiment of the present application, text detection is performed on the image to be processed to obtain multiple text image blocks in the image to be processed, and grayscale processing, proportional scaling, cropping, rotation and other operations are performed on any text image block to obtain a corresponding fourth text image block, and image block synthesis processing is performed on the multiple fourth text image blocks to obtain a text image, and the text image is input into a pre-trained recognition model to obtain the prediction probability of each language corresponding to each text image block and the prediction probability of each text direction output by the pre-trained recognition model, and the language with the largest prediction probability corresponding to each text image block is determined as the language corresponding to each text image block, and the text direction with the largest prediction probability corresponding to each text image block is determined as the text direction corresponding to each text image block.

[0261] In this way, with the help of multiple text image blocks in the image to be processed, the language corresponding to each text image block is identified, which can improve the accuracy of language distinction, and can distinguish the language appearing in the image to be processed to avoid affecting subsequent text recognition. In addition, with the help of multiple text image blocks in the image to be processed, the text direction corresponding to each text image block is identified, and the orientation of the text can be determined while screening and distinguishing the languages.

[0262] In an embodiment of the present application, a model training method is provided, such as Figure 6 As shown, it is a schematic diagram of an implementation flow of a model training method provided in an embodiment of the present application. The method is applied to an electronic device and may specifically include the following steps:

[0263] S601, obtaining a preset sample image set.

[0264] In the embodiment of the present application, a sample image set is pre-constructed. Since the image input to the model is composed of multiple sample text image blocks, the languages ​​of the sample text image blocks in the same sample image can be matched in a certain proportion, so the pre-constructed sample image set needs to meet certain requirements: the sample image set contains multiple sample images, and each sample image in the sample images of the preset first proportion corresponds to one language, and each sample image in the sample images of the preset second proportion corresponds to multiple different languages, and the preset first proportion is greater than the preset second proportion.

[0265] For example, for a preset sample image set, it contains multiple sample images, among which 70% of the sample images each correspond to one language, and 30% of the sample images each correspond to multiple different languages. In this way, there is a probability of 0.7 that sample text image blocks of the same language are extracted to form the input of a model, and there is another probability of 0.3 that sample text image blocks of different languages ​​are extracted to form the input of a model.

[0266] In addition, for the preset sample image set, there are certain requirements for the text direction. In this preset sample image set, each sample image in the sample images of the preset third ratio corresponds to the first text direction (for example, upward), and each sample image in the sample images of the preset fourth ratio corresponds to the second text direction (for example, downward), and the preset third ratio is the same as the preset fourth ratio.

[0267] For example, in the preset sample image set, the text direction of each sample image in 50% of the sample images is upward, and the text direction of each sample image in 50% of the sample images is downward. In this way, for the input composed of sample text image blocks, there is a 0.5 probability that the text direction is downward, and there is a 0.5 probability that the text direction is upward.

[0268] S602: For each sample image in the preset sample image set, the following processing is performed.

[0269] In the embodiment of the present application, the following processing is performed for each sample image in the preset sample image set, that is, steps S603 to S607 are performed.

[0270] S603: Perform text detection on the sample image to obtain a plurality of sample text image blocks in the sample image.

[0271] In the embodiment of the present application, for a sample image, text detection is performed on the sample image to obtain a plurality of sample text image blocks in the sample image, wherein the sample text image block contains text, such as a line of text, which is a part of the sample image.

[0272] Among them, text detection is performed on the sample image through a text detection algorithm to locate multiple sample text areas in the sample image, and for each sample text area, the sample image is cropped to obtain a corresponding sample text image block, so that multiple sample text image blocks can be obtained.

[0273] In addition, in the embodiment of the present application, for the sample image, the sample image needs to be angle-corrected, so that the text orientation involved in the sample image is only in two directions, upward and downward. This means that the sample image is first angle-corrected, and then text detection is performed on the angle-corrected sample image to obtain multiple sample text image blocks in the sample image.

[0274] S604, performing image block synthesis processing on multiple sample text image blocks to obtain a sample text image.

[0275] In an embodiment of the present application, multiple sample text image blocks are subjected to image block synthesis processing to obtain a sample text image. Among them, for any sample text image block, the following processing is performed to obtain a fourth sample text image block: grayscale processing is performed on the sample text image block to obtain a first sample text image block; the first sample text image block is scaled proportionally to obtain a second sample text image block, wherein the height of the second sample text image block is a preset height threshold; the second sample text image block is cropped to obtain a third sample text image block; the third sample text image block is rotated by a preset first angle according to a preset rotation direction to obtain a fourth sample text image block; multiple fourth sample text image blocks are subjected to image block synthesis processing to obtain a sample text image.

[0276] It should be noted that in the above step S603, the sample image does not need to be angle corrected, but after text detection, angle correction is performed for each sample text image block. Based on this, for any sample text image block, the sample text image block is angle corrected; multiple sample text image blocks that have undergone angle correction are subjected to image block synthesis processing to obtain a sample text image. Therefore, for any sample text image block that has undergone angle correction, the following processing is performed to obtain a fourth sample text image block: grayscale processing is performed on the sample text image block that has undergone angle correction to obtain a first sample text image block; the first sample text image block is scaled proportionally to obtain a second sample text image block, wherein the height of the second sample text image block is a preset height threshold; the second sample text image block is cropped to obtain a third sample text image block; the third sample text image block is rotated by a preset first angle according to a preset rotation direction to obtain a fourth sample text image block; multiple fourth sample text image blocks are subjected to image block synthesis processing to obtain a sample text image.

[0277] In an embodiment of the present application, the second sample text image block is cropped to obtain a third sample text image block, and the second sample text image block needs to be cropped to a fixed size. Therefore, it is determined whether the length of the second sample text image block is greater than a preset length threshold; when the length of the second sample text image block is greater than the preset length threshold, the second sample text image block is cropped to obtain a third sample text image block; when the length of the second sample text image block is equal to the preset length threshold, the second sample text image block is determined as the third sample text image block; when the length of the second sample text image block is less than the preset length threshold, the second sample text image block is image-filled to obtain a third sample text image block. Among them, the height of the third sample text image block is the preset height threshold, and the length of the third sample text image block is the preset length threshold.

[0278] For the filling process of the second sample text image block, the second sample text image block is usually filled with images in the horizontal direction according to a preset grayscale value to obtain a third sample text image block. For example, the second sample text image block is filled with images in the horizontal direction with 255 pixels to obtain a third sample text image block, which means that the second sample text image block is cropped to a third sample text image block of a fixed size (e.g., 32 in height and 256 in length), and the second sample text image block whose length is less than 256 is filled with pixels, such as 255, to obtain the third sample text image block.

[0279] In the embodiment of the present application, the synthesis processing of multiple fourth sample text image blocks is usually a splicing processing, so that the multiple fourth sample text image blocks are spliced ​​according to a preset splicing order to obtain a sample text image. For example, assuming that there are n fourth sample text image blocks, the n fourth sample text image blocks are spliced ​​according to the splicing order from left to right to obtain a sample text image of size (256, 32*n), which is the input of the model in the embodiment of the present application.

[0280] S605: Input the sample text image into a preset recognition model, and obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model.

[0281] In the embodiment of the present application, for the above sample image, the sample text image can be input into a preset recognition model to obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model. Wherein, the sample text direction can also be obtained while obtaining the sample language. For this purpose, the sample text image is input into a preset recognition model to obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model and the prediction probability of each sample text direction.

[0282] The sample text image can be subjected to image preprocessing and then input into a preset recognition model, that is, the sample text image is subjected to image preprocessing, wherein the image preprocessing is used to adjust the pixel value range of the sample text image, and the sample text image after image preprocessing is input into the preset recognition model to obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model, as well as the prediction probability of each sample text direction.

[0283] S606: Determine the language error according to the predicted probability of each sample language corresponding to each sample text image block and the labeled language corresponding to each sample text image block.

[0284] In an embodiment of the present application, for each sample text image block, there is a corresponding prediction probability of each sample language (i.e., the predicted language) and a corresponding annotated language (i.e., the actual language), thereby the language error can be determined based on the prediction probability of each sample language corresponding to each sample text image block and the annotated language corresponding to each sample text image block.

[0285] S607, training a preset recognition model according to the language error, and stopping the model training when the language error meets a preset condition to obtain a pre-trained recognition model.

[0286] In an embodiment of the present application, for language errors, a preset recognition model can be trained based on the language errors, and when the language error meets a preset condition, the model training is stopped to obtain a pre-trained recognition model.

[0287] Among them, for each sample text image block, there is a corresponding prediction probability of each sample text direction (i.e., the predicted text direction) and a corresponding annotated text direction (i.e., the actual text direction). Therefore, according to the prediction probability of each sample text direction corresponding to each sample text image block and the annotated text direction corresponding to each sample text image block, the text direction error is determined, and the annotated text direction is the first text direction or the second text direction.

[0288] Therefore, for the training of the preset recognition model, the preset recognition model can be trained according to the language error and the text direction error, and when the language error and the text direction error meet the preset conditions, the model training is stopped to obtain the pre-trained recognition model.

[0289] Among them, for the direction of the annotated text, when it is in the downward direction, it means that the sample image is flipped 180 degrees. At this time, the language error is set to 0 and does not participate in back propagation. Based on this, when the direction of the annotated text is the preset text direction, the language error is set to the preset value; according to the text direction error, the preset recognition model is trained, and when the text direction error meets the preset conditions, the model training is stopped to obtain the pre-trained recognition model.

[0290] For language error, it can be determined by the following formula:

[0291]

[0292] Where C is the set of sample languages ​​(i.e., the number of language categories), y ic is a symbolic function. If the annotated language (i.e., actual language) of sample text image block i (i.e., each sample text image block) is equal to sample language c (i.e., each sample language in C), then y ic If it is 1, otherwise it is 0.ic is the predicted probability that the language of sample text image block i (i.e., each sample text image block) belongs to sample language c (each sample language in C), d u is a symbolic function, and the text orientation of each sample text image block i is upward, then d i Take 0, the text direction of each sample text image block i is downward, then d i Take 1, so through d i , set the language error to 0 and do not participate in back propagation.

[0293] For the text direction error, it can be determined by the following formula:

[0294]

[0295] Where D is the set of sample text directions (i.e., the number of text direction categories), y ud is a sign function. If the annotated text direction (i.e., actual text direction) of sample text image block i (i.e., each sample text image block) is equal to the sample text direction d (each sample direction in D), then y id Take 1, otherwise y id Take 0, p id is the predicted probability that the text direction of sample text image block i (ie, each sample text image block) belongs to sample text direction d (each sample direction in D).

[0296] The preset recognition model is trained according to the language error and the text direction error. Specifically, the loss value is calculated according to the language error and the text direction error, and the preset recognition model is trained according to the loss value. When the loss value converges or the model training times reaches a certain number, the model training can be stopped.

[0297]

[0298] Among them, m is the number of sample text image blocks, and L is the loss value.

[0299] Corresponding to the above method embodiment, the present application embodiment also provides an image processing device, such as Figure 7 As shown, the device may include: an image acquisition module 710 , a text detection module 720 , an image block synthesis module 730 , and a language recognition module 740 .

[0300] An image acquisition module 710 is used to acquire an image to be processed;

[0301] A text detection module 720 is used to perform text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed;

[0302] An image block synthesis module 730 is used to perform image block synthesis processing on the plurality of text image blocks to obtain a text image;

[0303] The language identification module 740 is used to identify the language corresponding to each of the text image blocks according to the text image.

[0304] In an optional implementation, the text detection module is specifically used to:

[0305] Performing angle correction on the image to be processed;

[0306] Text detection is performed on the image to be processed that has undergone angle correction to obtain a plurality of text image blocks in the image to be processed.

[0307] In an optional implementation, the image block synthesis module is specifically used to:

[0308] For any of the text image blocks, performing angle correction on the text image block;

[0309] An image block synthesis process is performed on the plurality of angle-corrected text image blocks to obtain a text image.

[0310] In an optional implementation, the image block synthesis module specifically includes:

[0311] The image block processing submodule is used to perform the following processing on any of the text image blocks to obtain a fourth text image block:

[0312] A grayscale processing submodule, used for performing grayscale processing on the text image block to obtain a first text image block;

[0313] An image block scaling submodule, used for scaling the first text image block in equal proportion to obtain a second text image block, wherein the height of the second text image block is a preset height threshold;

[0314] An image block cropping submodule, used for cropping the second text image block to obtain a third text image block;

[0315] An image block rotation submodule, configured to rotate the third text image block by a preset first angle according to a preset rotation direction to obtain a fourth text image block;

[0316] The image block synthesis submodule is used to perform image block synthesis processing on the plurality of fourth text image blocks to obtain a text image.

[0317] In an optional implementation, the image block cropping submodule is specifically used to:

[0318] Determining whether the length of the second text image block is greater than a preset length threshold;

[0319] When the length of the second text image block is greater than the preset length threshold, cropping the second text image block to obtain a third text image block;

[0320] The height of the third text image block is a preset height threshold, and the length of the third text image block is the preset length threshold.

[0321] In an optional implementation, the image block synthesis module further includes:

[0322] an image block determination submodule, configured to determine the second text image block as a third text image block when the length of the second text image block is equal to the preset length threshold;

[0323] The image block filling submodule is used to perform image filling processing on the second text image block to obtain a third text image block when the length of the second text image block is less than the preset length threshold.

[0324] In an optional implementation, the image block filling submodule is specifically used to:

[0325] The second text image block is subjected to image filling processing in a horizontal direction according to a preset grayscale value to obtain a third text image block.

[0326] In an optional implementation, the image block synthesis submodule is specifically used to:

[0327] The plurality of fourth text image blocks are spliced ​​together in a preset splicing order to obtain a text image.

[0328] In an optional implementation, the language identification module specifically includes:

[0329] A prediction probability output submodule, used to input the text image into a pre-trained recognition model to obtain the prediction probability of each language corresponding to each text image block output by the pre-trained recognition model;

[0330] A language determination submodule, used to determine the language with the greatest predicted probability corresponding to each of the text image blocks as the language corresponding to each of the text image blocks;

[0331] Among them, the pre-trained recognition model includes N convolutional layers, M pooling layers, a bidirectional long short-term memory network and a fully connected layer, and N and M are both positive integers.

[0332] In an optional implementation, the prediction probability output submodule is specifically used for:

[0333] Performing image preprocessing on the text image, wherein the image preprocessing is used to adjust the pixel value range of the text image;

[0334] The text image that has undergone image preprocessing is input into a pre-trained recognition model to obtain the predicted probability of each language corresponding to each text image block output by the pre-trained recognition model.

[0335] In an optional implementation, the prediction probability output submodule is specifically used for:

[0336] Inputting the text image into a pre-trained recognition model to obtain the prediction probability of each language and the prediction probability of each text direction corresponding to each text image block output by the pre-trained recognition model;

[0337] The language recognition module also includes:

[0338] The text direction determination submodule is used to determine the text direction with the maximum prediction probability corresponding to each text image block as the text direction corresponding to each text image block.

[0339] In an optional embodiment, the device further comprises:

[0340] A text direction determination module, used to determine whether the text direction is a preset text direction;

[0341] The jump module is used to rotate the image to be processed by a preset second angle when the text direction is a preset text direction, and jump to the step of performing text detection on the image to be processed to obtain multiple text image blocks in the image to be processed.

[0342] In an optional embodiment, the device further comprises:

[0343] An image set acquisition module, used to acquire a preset sample image set, wherein the preset sample image set includes a plurality of sample images, and each sample image in the sample images of a preset first ratio corresponds to one language, and each sample image in the sample images of a preset second ratio corresponds to a plurality of different languages, and the preset first ratio is greater than the preset second ratio;

[0344] The sample image processing module is used to perform the following processing on each sample image in the preset sample image set:

[0345] A sample image detection module, used to perform text detection on the sample image to obtain a plurality of sample text image blocks in the sample image;

[0346] A sample text image block synthesis module, used for performing image block synthesis processing on a plurality of the sample text image blocks to obtain a sample text image;

[0347] A sample language recognition module, used for inputting the sample text image into a preset recognition model, and obtaining the predicted probability of each sample language corresponding to each sample text image block output by the preset recognition model;

[0348] A language error determination module, used to determine the language error according to the predicted probability of each sample language corresponding to each sample text image block and the labeled language corresponding to each sample text image block;

[0349] The model training module is used to train the preset recognition model according to the language error, and stop the model training when the language error meets the preset conditions to obtain the pre-trained recognition model.

[0350] In an optional implementation, the sample image detection module is specifically used to:

[0351] Performing angle correction on the sample image;

[0352] Text detection is performed on the angle-corrected sample image to obtain a plurality of sample text image blocks in the sample image.

[0353] In an optional implementation, the sample text image block synthesis module is specifically used to:

[0354] For any of the sample text image blocks, performing angle correction on the sample text image block;

[0355] An image block synthesis process is performed on a plurality of the sample text image blocks that have undergone angle correction to obtain a sample text image.

[0356] In an optional implementation, the sample text image block synthesis module specifically includes:

[0357] A sample text image block processing submodule, configured to perform the following processing on any of the sample text image blocks to obtain a fourth sample text image block;

[0358] A sample text image block grayscale processing submodule, used for performing grayscale processing on the sample text image block to obtain a first sample text image block;

[0359] A sample text image block scaling submodule, used for scaling the first sample text image block in equal proportion to obtain a second sample text image block, wherein the height of the second sample text image block is a preset height threshold;

[0360] a sample text image block cropping submodule, used for cropping the second sample text image block to obtain a third sample text image block;

[0361] A sample text image block rotation submodule, used to rotate the third sample text image block by a preset first angle according to a preset rotation direction to obtain a fourth sample text image block;

[0362] The sample text image block synthesis submodule is used to perform image block synthesis processing on a plurality of the fourth sample text image blocks to obtain a sample text image.

[0363] In an optional implementation, the sample text image block cropping submodule is specifically used to:

[0364] Determining whether the length of the second sample text image block is greater than a preset length threshold;

[0365] When the length of the second sample text image block is greater than a preset length threshold, cropping the second sample text image block to obtain a third sample text image block;

[0366] The height of the third sample text image block is a preset height threshold, and the length of the third sample text image block is the preset length threshold.

[0367] In an optional implementation, the sample text image block synthesis module further includes:

[0368] a sample text image block determining submodule, configured to determine the second sample text image block as a third sample text image block when the length of the second sample text image block is equal to a preset length threshold;

[0369] The sample text image block filling submodule is used to perform image filling processing on the second sample text image block to obtain a third sample text image block when the length of the second sample text image block is less than a preset length threshold.

[0370] In an optional implementation, the sample text image block filling submodule is specifically used to:

[0371] The second sample text image block is subjected to image filling processing in a horizontal direction according to a preset grayscale value to obtain a third sample text image block.

[0372] In an optional implementation, the sample text image block synthesis submodule is specifically used for:

[0373] The plurality of fourth sample text image blocks are spliced ​​together in a preset splicing order to obtain a sample text image.

[0374] In an optional implementation, the sample language recognition module is specifically used to:

[0375] Performing image preprocessing on the sample text image, wherein the image preprocessing is used to adjust the pixel value range of the sample text image;

[0376] The sample text image that has undergone image preprocessing is input into a preset recognition model to obtain the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model.

[0377] In an optional embodiment, in the preset sample image set, each sample image in the sample images of the preset third ratio corresponds to the first text direction, each sample image in the sample images of the preset fourth ratio corresponds to the second text direction, and the preset third ratio is the same as the preset fourth ratio;

[0378] The sample language recognition module is specifically used for:

[0379] Inputting the sample text image into a preset recognition model, obtaining the prediction probability of each sample language and the prediction probability of each sample text direction corresponding to each sample text image block output by the preset recognition model;

[0380] The model training module specifically includes:

[0381] A text direction error determination submodule, used to determine a text direction error according to a prediction probability of each sample text direction corresponding to each sample text image block and an annotated text direction corresponding to each sample text image block, wherein the annotated text direction is the first text direction or the second text direction;

[0382] The model training submodule is used to train the preset recognition model according to the language error and the text direction error, and stop the model training when the language error and the text direction error meet the preset conditions to obtain the pre-trained recognition model.

[0383] In an optional embodiment, the model training submodule is specifically used for:

[0384] When the direction of the annotated text is a preset text direction, the language error is set to a preset value;

[0385] The preset recognition model is trained according to the text direction error, and when the text direction error meets a preset condition, the model training is stopped to obtain the pre-trained recognition model.

[0386] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor 81, a communication interface 82, a memory 83 and a communication bus 84, wherein the processor 81, the communication interface 82, and the memory 83 communicate with each other through the communication bus 84.

[0387] A memory 83, for storing computer programs;

[0388] The processor 81 is used to execute the program stored in the memory 83 to implement the following steps:

[0389] Acquire an image to be processed; perform text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed; perform image block synthesis processing on the plurality of text image blocks to obtain a text image; and identify the language corresponding to each text image block based on the text image.

[0390] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0391] The communication interface is used for communication between the above electronic device and other devices.

[0392] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0393] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0394] In another embodiment provided in the present application, a storage medium is provided. The storage medium stores instructions. When the instructions are executed on a computer, the computer executes the image processing method described in any one of the above embodiments.

[0395] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the image processing method described in any one of the above embodiments.

[0396] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium, or transmitted from one storage medium to another storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0397] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0398] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0399] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Get the image to be processed; Performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed; Performing image block synthesis processing on a plurality of the text image blocks to obtain a text image; According to the text image, the language corresponding to each of the text image blocks is identified.

2. The method according to claim 1, characterized in that The performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed includes: Performing angle correction on the image to be processed; Text detection is performed on the image to be processed that has undergone angle correction to obtain a plurality of text image blocks in the image to be processed.

3. The method according to claim 1, characterized in that The step of performing image block synthesis processing on the plurality of text image blocks to obtain a text image comprises: For any of the text image blocks, the following processing is performed to obtain a fourth text image block: Performing grayscale processing on the text image block to obtain a first text image block; Scaling the first text image block in equal proportion to obtain a second text image block, wherein the height of the second text image block is a preset height threshold; Cropping the second text image block to obtain a third text image block; Rotating the third text image block by a preset first angle according to a preset rotation direction to obtain a fourth text image block; Perform image block synthesis processing on the plurality of fourth text image blocks to obtain a text image.

4. The method according to claim 3, characterized in that: The step of cropping the second text image block to obtain a third text image block includes: Determining whether the length of the second text image block is greater than a preset length threshold; When the length of the second text image block is greater than the preset length threshold, cropping the second text image block to obtain a third text image block; Wherein, the height of the third text image block is a preset height threshold, and the length of the third text image block is the preset length threshold; When the length of the second text image block is equal to the preset length threshold, determining the second text image block as a third text image block; When the length of the second text image block is less than the preset length threshold, performing image filling processing on the second text image block to obtain a third text image block; The step of performing image block synthesis processing on the plurality of fourth text image blocks to obtain a text image comprises: The plurality of fourth text image blocks are spliced ​​together in a preset splicing order to obtain a text image.

5. The method according to claim 1, characterized in that The step of identifying the language corresponding to each text image block according to the text image includes: Inputting the text image into a pre-trained recognition model to obtain a predicted probability of each language corresponding to each text image block output by the pre-trained recognition model; Determine the language with the greatest predicted probability corresponding to each of the text image blocks as the language corresponding to each of the text image blocks; Among them, the pre-trained recognition model includes N convolutional layers, M pooling layers, a bidirectional long short-term memory network and a fully connected layer, and N and M are both positive integers.

6. The method according to claim 5, characterized in that The step of inputting the text image into a pre-trained recognition model to obtain a prediction probability of each language corresponding to each text image block output by the pre-trained recognition model includes: Inputting the text image into a pre-trained recognition model to obtain the prediction probability of each language and the prediction probability of each text direction corresponding to each text image block output by the pre-trained recognition model; The method further comprises: Determine the text direction with the largest prediction probability corresponding to each of the text image blocks as the text direction corresponding to each of the text image blocks; Determining whether the text direction is a preset text direction; When the text direction is a preset text direction, the image to be processed is rotated by a preset second angle, and the process jumps to the step of performing text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed.

7. The method according to claim 6, characterized in that Before executing the method, the method further includes: Acquire a preset sample image set, wherein the preset sample image set includes a plurality of sample images, and each sample image in the sample images of a preset first ratio corresponds to one language, and each sample image in the sample images of a preset second ratio corresponds to a plurality of different languages, and the preset first ratio is greater than the preset second ratio; For each of the sample images in the preset sample image set, the following processing is performed: Performing text detection on the sample image to obtain a plurality of sample text image blocks in the sample image; Performing image block synthesis processing on a plurality of the sample text image blocks to obtain a sample text image; Inputting the sample text image into a preset recognition model, and obtaining the predicted probability of each sample language corresponding to each sample text image block output by the preset recognition model; Determining a language error according to the predicted probability of each sample language corresponding to each sample text image block and the labeled language corresponding to each sample text image block; The preset recognition model is trained according to the language error, and when the language error meets a preset condition, the model training is stopped to obtain the pre-trained recognition model.

8. The method according to claim 7, characterized in that In the preset sample image set, each sample image in the sample images of the preset third ratio corresponds to the first text direction, each sample image in the sample images of the preset fourth ratio corresponds to the second text direction, and the preset third ratio is the same as the preset fourth ratio; The step of inputting the sample text image into a preset recognition model and obtaining the prediction probability of each sample language corresponding to each sample text image block output by the preset recognition model comprises: Inputting the sample text image into a preset recognition model, obtaining the prediction probability of each sample language and the prediction probability of each sample text direction corresponding to each sample text image block output by the preset recognition model; The step of training the preset recognition model according to the language error and stopping the model training when the language error meets a preset condition to obtain the pre-trained recognition model comprises: Determine a text direction error according to a prediction probability of each sample text direction corresponding to each sample text image block and an annotated text direction corresponding to each sample text image block, wherein the annotated text direction is the first text direction or the second text direction; The preset recognition model is trained according to the language error and the text direction error, and when the language error and the text direction error meet preset conditions, the model training is stopped to obtain the pre-trained recognition model.

9. The method according to claim 8, characterized in that The method of training the preset recognition model according to the language error and the text direction error, and stopping the model training when the language error and the text direction error meet preset conditions to obtain the pre-trained recognition model includes: When the direction of the annotated text is a preset text direction, the language error is set to a preset value; The preset recognition model is trained according to the text direction error, and when the text direction error meets a preset condition, the model training is stopped to obtain the pre-trained recognition model.

10. An image processing device, characterized in that: The device comprises: An image acquisition module, used for acquiring an image to be processed; A text detection module, used to perform text detection on the image to be processed to obtain a plurality of text image blocks in the image to be processed; An image block synthesis module, used for performing image block synthesis processing on a plurality of the text image blocks to obtain a text image; The language recognition module is used to recognize the language corresponding to each text image block according to the text image.

Images