Sample image generation, model training, character recognition method, device and medium

Abstract

The present application relates to the technical field of image processing, and in particular to a sample image generation method, model training, character recognition method, device and medium, the generation method comprising obtaining a target text and style features under a target scene and a first sample character, the target text and style features being obtained by text style encoding on a real character image under the target scene; processing the first sample character and the target text and style features based on a target diffusion model to obtain a sample character image under the target scene, the target diffusion model being used to migrate the font style of the first sample character to the real character image to generate the sample character image; and splicing the sample character image with a background image under the target scene to obtain a sample image. The font style of the first sample character is migrated to the real character image by using the target diffusion model, style uniformity is achieved, and a realistic sample image is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a sample image generation method, a model training method, a character recognition method, a device and a medium. BACKGROUND

[0002] Optical character recognition (OCR) has many applications in the rapidly developing field of deep learning applications at the present stage, and has excellent performance in document and certificate recognition and other applications. In the field of machine vision, character recognition is required in different scenarios. In the case of sufficient sample data, the two-stage model of the trained character recognition model can well recognize the text content on the picture. Based on this, sample data affects the accuracy of the character recognition model.

[0003] However, in some application scenarios, the amount of sample data is small, and the realism of the sample data obtained through some data enhancement processing is low. For example, character recognition on clothes. In order to increase the amount of sample data, the existing scheme is to directly paste characters on the body. Even if some text image augmentation transformations are added, it is difficult to make it look like the characters on the clothes rather than pasted later, thereby resulting in low realism of the sample image. SUMMARY

[0004] Therefore, the embodiments of the present application provide a sample image generation method, a model training method, a character recognition method, a device and a medium to solve the problem of low realism of the sample image.

[0005] According to a first aspect, the embodiments of the present application provide a sample image generation method, comprising:

[0006] obtaining a target text and a style feature in a target scene and a first sample character, wherein the target text and the style feature are obtained by text style encoding on a real character image in the target scene;

[0007] processing the first sample character and the target text and style feature based on a target diffusion model to obtain a sample character image in the target scene, wherein the target diffusion model is used to migrate the font style of the first sample character to the real character image to generate the sample character image;

[0008] splicing the sample character image and a background image in the target scene to obtain a sample image.

[0009] The sample image generation method provided by the embodiment of the present application uses the target text and style features obtained from the real character image in the target scene as the condition of the target diffusion model, uses the target diffusion model to migrate the font style of the first sample character to the real character image, so as to ensure that the text in the generated sample character image and the image can realize style unification and be closer to the real text style of the target scene, and then splicing the sample character image and the background image in the target scene can obtain a realistic sample character image.

[0010] In some embodiments, the target text and style features in the target scene are obtained, including:

[0011] The real character image is obtained.

[0012] Image feature extraction is performed on the real character image to obtain image style encoding.

[0013] Text feature extraction is performed on the text content in the real character image to obtain text encoding.

[0014] The image style encoding and the text encoding are fused to obtain the target text and style features.

[0015] The sample image generation method provided by the embodiment of the present application extracts image features and text features from the real character image respectively, and then fuses the two, so that the obtained target text and style features include both image features and text features, improving the reliability of the obtained target text and style features.

[0016] In some embodiments, the image style encoding and the text encoding are fused to obtain the target text and style features, including:

[0017] Attention processing is performed on the image style encoding and the text encoding to obtain the attention between the image style encoding and the text encoding.

[0018] The attention and the text encoding are fused, and the fusion result is processed through a feedforward network to obtain the target text and style features.

[0019] The sample image generation method provided by the embodiment of the present application fuses image style encoding and text encoding through attention, so that different text contents pay attention to different parts of the given image style encoding, improves the reliability and authenticity of the obtained fusion result, and then the rich semantic features can be extracted through the processing of the feedforward network, further ensuring the authenticity of the real character image text style encoding, so that when the target text and style features are taken as the input conditions of the target diffusion model, the target diffusion model can obtain more real sample images through diffusion.

[0020] In some embodiments, the splicing the sample character image and the background image in the target scene to obtain a sample image comprises:

[0021] Obtaining a background image in the target scene, and identifying a region of interest of the background image to obtain a region of interest image;

[0022] Rotating the sample character image at an arbitrary angle to obtain a rotated sample character image;

[0023] Splicing the rotated sample character image and the region of interest image to obtain the sample image.

[0024] The sample image generation method provided by the embodiment of the present application identifies the region of interest to facilitate subsequent splicing of the sample character image to the region of interest to be close to the use in the target scene; at the same time, the sample character image is rotated at an arbitrary angle, so that a large number of sample images can be generated, thereby enriching the number of sample images.

[0025] In some embodiments, the training method of the target diffusion model comprises:

[0026] Obtaining a second sample character and text and style features of a sample image in the target scene;

[0027] Generating a noise image based on the second sample character in a forward diffusion process of a preset diffusion model;

[0028] Training a reverse diffusion process of the preset diffusion model based on the noise image and the text and style features of the sample image in the target scene to determine the target diffusion model.

[0029] The sample image generation method provided by the embodiment of the present application trains the reverse diffusion process of the preset diffusion model with the second sample character as supervision and the text and style features of the sample image in the target scene as conditions, which can ensure that the obtained target diffusion model can generate images that fit the target scene.

[0030] According to a second aspect, the embodiments of the present application further provide a training method of a character detection model, comprising:

[0031] obtaining a sample image, the sample image being obtained according to the sample image generation method in the first aspect of the present application or any of the embodiments of the first aspect;

[0032] obtaining label data of the sample image, the label data comprising position information of Chinese text content of the sample image and a target rotation angle;

[0033] inputting the sample image into the character detection model to obtain predicted position information of the Chinese text content of the sample image and a predicted rotation angle;

[0034] updating parameters of the character detection model based on the predicted position information, the predicted rotation angle and the label data to obtain a target character detection model.

[0035] The training method of the character detection model provided by the embodiments of the present application is based on a large number of realistic sample images obtained in a target scene, and the training of the character detection model is performed, so that the accuracy of the obtained target character detection model can be improved. Moreover, the target character detection model obtained by training also outputs a rotation angle of the text content, so that the detection accuracy of the target character detection model is improved. Furthermore, the detected text content can be rotated by using the rotation angle during subsequent character recognition, so that the accuracy of character recognition is further improved.

[0036] According to a third aspect, the embodiments of the present application further provide a training method of a character recognition model, comprising:

[0037] obtaining a sample image, the sample image being obtained according to the sample image generation method in the first aspect of the present application or any of the embodiments of the first aspect;

[0038] inputting the sample image into a target character detection model to obtain position and rotation angle of text content of the sample image, the target character detection model being obtained by training according to the training method of the character detection model in the second aspect of the present application;

[0039] rotating the text content in the sample image by using the position and the rotation angle of the text content to obtain target text content;

[0040] updating parameters of the character recognition model based on the target text content and a text label of the sample image to obtain a target character recognition model.

[0041] The training method of the character recognition model provided in the embodiment of the present application rotates and corrects the text content by using the predicted position and rotation angle before inputting the detected text line into the character recognition model, so as to ensure that the angle of the obtained target text content is uniform, and then the parameters of the character recognition model are updated by using the target text content with uniform angle, thereby further improving the accuracy of the target character recognition model obtained by training.

[0042] According to a fourth aspect, the embodiment of the present application also provides a character recognition method, comprising:

[0043] obtaining an image to be processed under a target scene;

[0044] inputting the image to be processed into a target character detection model to obtain the position and rotation angle of the text content of the image to be processed, wherein the target character detection model is obtained by training according to the training method of the character detection model of the second aspect of the present application;

[0045] rotating the text content of the image to be processed by using the position and rotation angle of the text content of the image to be processed to obtain text content to be recognized;

[0046] inputting the text content to be recognized into a target character recognition model to obtain a character recognition result of the image to be processed, wherein the target character recognition model is obtained by training according to the training method of the character recognition model of the third aspect of the present application.

[0047] The character recognition method provided in the embodiment of the present application has high character detection and character recognition accuracy because the target character detection model and the target character recognition model are obtained by training using a large number of realistic sample images, and the character recognition method can obtain a more accurate character recognition result by using the target character detection model and the target character recognition model to recognize characters in the image to be processed.

[0048] According to a fifth aspect, the embodiment of the present application also provides a sample image generation module, comprising:

[0049] a first obtaining module configured to obtain a target text and style feature and a font style of a first sample character under a target scene, wherein the target text and style feature is obtained by performing text style encoding on a real character image under the target scene;

[0050] a style processing module configured to process the font style of the first sample character and the target text and style feature based on a target diffusion model to obtain a sample character image under the target scene, wherein the target diffusion model is used to migrate the font style of the first sample character to the real character image to generate the sample character image;

[0051] The splicing module is configured to splice the sample character image and the background image in the target scene to obtain a sample image.

[0052] According to a sixth aspect, the embodiments of the present application further provide a training device of a character detection model, comprising:

[0053] The second acquisition module is configured to acquire a sample image, wherein the sample image is obtained according to the sample image generation method in the first aspect of the present application or any implementation manner of the first aspect.

[0054] The third acquisition module is configured to acquire label data of the sample image, wherein the label data comprises position information of Chinese text content of the sample image and a target rotation angle.

[0055] The first prediction module is configured to input the sample image into the character detection model to obtain predicted position information of Chinese text content of the sample image and a predicted rotation angle.

[0056] The first update module is configured to update parameters of the character detection model based on the predicted position information, the predicted rotation angle and the label data, so as to obtain a target character detection model.

[0057] According to a seventh aspect, the embodiments of the present application further provide a training device of a character recognition model, comprising:

[0058] The fourth acquisition module is configured to acquire a sample image, wherein the sample image is obtained according to the sample image generation method in the first aspect of the present application or any implementation manner of the first aspect.

[0059] The first detection module is configured to input the sample image into a target character detection model to obtain position and rotation angle of Chinese text content of the sample image, wherein the target character detection model is obtained by training the character detection model according to the training method of the character detection model in the second aspect of the present application.

[0060] The first rotation module is configured to rotate the text content in the sample image by using the position of the text content and the rotation angle, so as to obtain target text content.

[0061] The second update module is configured to update parameters of the character recognition model based on the target text content and a text label of the sample image, so as to obtain a target character recognition model.

[0062] According to an eighth aspect, the embodiments of the present application further provide a character recognition module, comprising:

[0063] The fifth acquisition module is configured to acquire an image to be processed in a target scene.

[0064] a second detection module, configured to input the to-be-processed image into a target character detection model to obtain a position and a rotation angle of Chinese text content of the to-be-processed image, the target character detection model being obtained by training according to the training method of the character detection model of the second aspect of the present application;

[0065] a second rotation module, configured to rotate the Chinese text content of the to-be-processed image according to the position and the rotation angle of the Chinese text content of the to-be-processed image to obtain to-be-recognized text content;

[0066] a recognition module, configured to input the to-be-recognized text content into a target character recognition model to obtain a character recognition result of the to-be-processed image, the target character recognition model being obtained by training according to the training method of the character recognition model of the third aspect of the present application.

[0067] According to a ninth aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, which are in communication connection with each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the sample image generation method in the first aspect or any one of the implementation manners of the first aspect, or executes the training method of the character detection model in the second aspect, or executes the training method of the character recognition model in the third aspect, or executes the character recognition method in the fourth aspect.

[0068] According to a tenth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer instructions, and the computer instructions are used to make the computer execute the sample image generation method in the first aspect or any one of the implementation manners of the first aspect, or execute the training method of the character detection model in the second aspect, or execute the training method of the character recognition model in the third aspect, or execute the character recognition method in the fourth aspect.

[0069] It should be noted that the corresponding beneficial effects of the sample image generation apparatus, the character detection model training apparatus, the character recognition model training apparatus, the character recognition apparatus, the electronic device and the computer readable storage medium provided by the embodiments of the present application are described in the corresponding beneficial effect description of the sample image generation method, the character detection model training method, the character recognition model training method and the character recognition method, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0070] In order to make the technical solutions in the specific embodiments or prior art of the present application clearer, the accompanying drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort on the basis of these accompanying drawings.

[0071] Figure 1 is a flow chart of a sample image generation method according to an embodiment of the present application;

[0072] Figure 2 is a flow chart of a sample image generation method according to an embodiment of the present application;

[0073] Figure 3 is a schematic diagram of character encoding according to an embodiment of the present application;

[0074] Figure 4 is a structural schematic diagram of a reverse diffusion process according to an embodiment of the present application;

[0075] Figure 5 is a flow chart of a character detection model training method according to an embodiment of the present application;

[0076] Figure 6 is a structural schematic diagram of a character detection model according to an embodiment of the present application;

[0077] Figure 7 is a flow chart of a character recognition model training method according to an embodiment of the present application;

[0078] Figure 8 is a flow chart of a character recognition method according to an embodiment of the present application;

[0079] Figure 9 is a structural block diagram of a sample image generation device according to an embodiment of the present application;

[0080] Figure 10 is a structural block diagram of a character detection model training device according to an embodiment of the present application;

[0081] Figure 11 is a structural block diagram of a character recognition model training device according to an embodiment of the present application;

[0082] Figure 12 is a structural block diagram of a character recognition device according to an embodiment of the present application;

[0083] Figure 13 is a hardware structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0085] As described above, the character printed image generated by using the simple character style cannot be well fused with the background image under the target scene, which causes great negative influence on character recognition. In the case of a small amount of samples, the related art generally uses a generation model to expand the sample data set. The generation model mostly uses an adversarial neural model, but it is difficult to train to reach a balance due to the adversality of the adversarial neural model. Based on this, the method for generating a sample image provided in the embodiments of the present application expands the sample data set based on a target diffusion model in the case of a small amount of sample data sets, and improves the fidelity of the obtained sample image on this basis. Based on the generated sample image, the embodiments of the present application further provide a method for training a character detection model, which is used to train a target character detection model for detecting the position and rotation angle of text content. Further, based on the sample character image, the embodiments of the present application further provide a method for training a character recognition model, which is used to train a target character recognition model for character recognition.

[0086] Based on the trained target character recognition model, the character recognition method provided in the embodiments of the present application can be applied to character recognition on clothes, character recognition on a banner, and the like. The specific application scenarios of character recognition are set according to actual needs, and are not limited herein.

[0087] According to the embodiments of the present application, a method for generating a sample image, a method for training a character recognition model, and a character recognition method are provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0088] In the present embodiment, a method for generating a sample image is provided, which can be used in electronic devices such as computers, servers, etc. Figure 1 The flowchart of the method for generating a sample image according to the embodiments of the present application is shown in FIG. 1, which includes the following steps: Figure 1

[0089] S11, obtaining a target text and style features under a target scene and a first sample character.​

[0090] The target text and style feature are obtained by text style encoding on the real character image in the target scene.

[0091] As described above, the target scene is set according to actual needs. For example, the trained character recognition model is used to recognize characters on clothes, and accordingly, the real character image in the target scene is a real character image on clothes. The character image on clothes is cut out from the real scene collected by the image collection device to obtain the real character image in the target scene.

[0092] The first sample character is a network printed character of different styles, or other forms of characters, etc. The form and source of the character are not limited here, and are set according to actual needs.

[0093] The target text and style feature in the target scene are obtained by a text style encoder. The input of the text style encoder is a character image, and the output is the text and style feature of the character image. The text style encoder is used to extract the text and style feature of the input character image. Based on this, the style encoder is constructed based on a feature extraction model. The feature extraction model here includes but is not limited to MobileNetV2 model, ResNet or VGG, etc.

[0094] S12, processing the first sample character and the target text and style feature based on a target diffusion model to obtain a sample character image in the target scene.

[0095] The target diffusion model is used to migrate the font style of the first sample character to the real character image to generate a sample character image.

[0096] The target diffusion model is used to realize the migration of the font style. After obtaining the first sample character and the target text and style feature, it is input into the target diffusion model to obtain a sample character image in the target scene. Specifically, the target diffusion model migrates the font style of the first sample character to the real character image to generate a sample character image. Taking the target scene as an example of character recognition on clothes, the font style in the first sample character is migrated to the actual personnel clothing scene based on the target diffusion model to generate a realistic sample character image.

[0097] S13, splicing the sample character image with a background image in the target scene to obtain a sample image.

[0098] The background image in the target scene is the background of the character image. The sample character image obtained in the above S12 is only a sample character image that fuses the text and style features of the target scene. In order to obtain a specific sample image, the sample character image needs to be fused into the background of the target scene. Specifically, the sample character image is spliced with the background image in the target scene. The splicing can be placing the sample character image at any position of the background image, or placing the sample character image at any position of the background image at any angle, and the like. After the splicing processing, a sample image including the background of the target scene and the sample character is obtained.

[0099] The number of generated sample images is not limited here, and can be set according to actual needs.

[0100] The sample image generation method provided in this embodiment uses the target text and style features obtained from the real character image in the target scene as the condition of the target diffusion model, and uses the target diffusion model to migrate the font style of the first sample character into the real character image, so as to ensure that the text and image in the generated sample character image can achieve style unification and be closer to the real text style of the target scene. Splicing the sample character image with the background image in the target scene can obtain a realistic sample character image. Using this method can generate a large number of realistic sample images in the target scene. On this basis, training the preset character recognition model can improve the accuracy of the obtained target character recognition model.

[0101] In this embodiment, a sample image generation method is provided, which can be used in electronic devices such as computers, servers, and the like, Figure 2 is a flowchart of a character recognition model training method according to an embodiment of the present application, as shown in Figure 2 The flowchart includes the following steps:

[0102] S21, obtaining a target text and style feature in a target scene and a font style of a first sample character.

[0103] The target text and style feature is obtained by text style encoding of a real character image in a target scene.

[0104] Specifically, the above S21 includes:

[0105] S211, obtaining a real character image.

[0106] The real character image is obtained by cutting out a character image from an image collected in a target scene. The specific cutting method is not limited here.

[0107] S212, performing image feature extraction on the real character image to obtain image style encoding.

[0108] The image feature extraction can be implemented by an image feature extraction model or by an image processing method. Taking the feature extraction model as an example, the real character image is input into the feature extraction model, and the output of the model is the image style code. For example, the local feature or key point feature of the input real character image is extracted to obtain the image style code.

[0109] S213, text feature extraction is performed on the text content in the real character image to obtain a text code.

[0110] The text content is a text line in the real character image, and the text feature extraction is performed on the text line to obtain a text code. The text feature extraction can also be implemented by a text feature extraction model, which includes but is not limited to bag-of-words feature extraction, TF-IDF (inverse document frequency) feature extraction, or word vector-based feature extraction, etc.

[0111] S214, the image style code and the text code are fused to obtain a target text and style feature.

[0112] The fusion mode of the image style code and the text code includes but is not limited to splicing of two encoding vectors or weighted sum of two encoding vectors, etc. After the fusion processing, the target text and style feature are obtained.

[0113] In some embodiments, the above S214 includes:

[0114] (1) attention processing is performed on the image style code and the text code to obtain the attention between the image style code and the text code.

[0115] (2) the attention is fused with the text code, and the fusion result is processed by a feedforward network to obtain a target text and style feature.

[0116] The attention processing is implemented based on a multi-head attention module, the input is the image style code and the text code, and the output is the attention between the image style code and the text code, so that different text characters pay attention to different parts of the given style sample. After obtaining the attention between the two, the attention is fused with the text code to obtain a fusion result, and then the fusion result is processed by a feedforward network to obtain a target text and style feature. The fusion of the attention and the text code can be stacking in the multi-dimensional matrix channel direction. The feedforward network integrates the image features of the character region and the semantic features of the content of the region characters.

[0117] The fusion of image style encoding and text encoding is realized by attention, so that different text contents pay attention to different parts of the given image style encoding, improving the reliability and authenticity of the obtained fusion result. On this basis, the rich semantic features can be extracted through the processing of the feedforward network, further ensuring the authenticity of the real character image text style encoding, so that when the target text and style features are taken as the input conditions of the target diffusion model, the target diffusion model can obtain more real sample images through diffusion.

[0118] As a specific application example, the character recognition on the clothes is taken as an example. Figure 3 A specific application example of text style encoding is shown. Specifically, as shown in Figure 3 The above-mentioned manner of determining the target text and style features includes: cutting out the character image on the clothes from the real scene collected by the camera, inputting the character image into the MobileNetV2 model pre-trained on Imagenet, discarding the top fully connected layer, and using the average pooling layer to extract local features as image style encoding. Since the MobileNetV2 model here is essentially a classification model, a fully connected layer is connected at the end of the model to realize the classification function. In this embodiment, only the model is needed as a feature extractor, so the fully connected layer is removed; that is, the network structure before the fully connected layer of the classification module is the feature extraction module, and the purpose of adding the average pooling is to filter the features. Of course, in addition to the average pooling, a maximum pooling layer or the like can also be used, which is not limited herein. Further, the current text line is encoded using an embedding layer according to the text content in the character image, and the characters are encoded in the order of English and numbers, which is used to represent the text encoding. The extracted image style encoding and text encoding are input into the multi-head attention layer, and the attention between the text encoding and the extracted features is calculated, so that different text characters pay attention to different parts of the given style sample. Finally, the output of the multi-head attention mechanism is added to the text encoding, and then a feedforward network is used to obtain the final target text and style feature output.

[0119] S215, obtaining a first sample character.

[0120] S22, processing the first sample character, the target text and the style features based on a target diffusion model to obtain a sample character image in a target scene.

[0121] The target diffusion model is used to migrate the font style of the first sample character to the real character image to generate the sample character image.

[0122] As described above, the target diffusion model is used to realize font style migration, and the output is a sample character image.

[0123] In some embodiments, the training method of the target diffusion model comprises:

[0124] (1) obtaining the second sample character and the text and style features of the sample image under the target scene.

[0125] (2) generating a noise image based on the second sample character during the forward diffusion process of the preset diffusion model.

[0126] (3) training the reverse diffusion process of the preset diffusion model based on the noise image and the text and style features of the sample image under the target scene to determine the target diffusion model.

[0127] The second sample character is a randomly generated image, for example, a second sample character with random length, random color and random size is generated as an original image using the opencv tool. The text and style features of the sample image under the target scene are obtained in a manner similar to the above-described manner of obtaining the target text and style features, and will not be described here.

[0128] For the preset diffusion model, it includes a forward diffusion process and a reverse diffusion process, wherein the forward diffusion process is a process for changing the distribution of the original image into a standard Gaussian distribution, and the reverse diffusion process is used to generate character images under the target scene. Taking character recognition on clothes as an example, in the forward diffusion process, noise is randomly added to the original image. For the noise mechanism β1,…β T , β t = 0.02+Exponential(1×10 -5 ,0.4), wherein Exponential(1×10 -5 ,0.4) represents a floating point number between log(1×10 -5 ) and log(0.4). After T iterations, the distribution of the original image is finally changed into a standard Gaussian distribution. Wherein β i represents the noise added in the i-th iteration (i=1, 2, …, T), and the specific value of T is set according to actual requirements, which is not limited here.

[0129] The reverse diffusion process is implemented based on the UNet model as described in Figure 4 Figure 4 ​As shown, the UNet model is composed of down-sampling blocks and up-sampling blocks, and uses long-range convolutional skip connections, which mainly use two types of blocks, convolutional blocks and attention blocks. The convolutional block is composed of 3 convolutional layers and a convolutional skip connection and applies conditional affine transformation to the output of each convolutional layer, and the scale and bias of each conditional affine transformation are parameterized by the output of the fully connected layer. The attention block is composed of 2 multi-head attention layers and a feedforward network. The first attention layer performs attention between the text sequence latent variable of the input noisy image and the text and style features of the sample image, while the second attention layer performs self-attention. Conditional affine transformation is performed after each attention layer and the feedforward network using layer normalization. As shown in the figure, Figure 4 As shown, the input of the attention block includes two parts, one part is the original image used as supervision, and the other part is the text and style features of the sample image in the target scene, that is, the inverse process of the diffusion model is trained with the text and style features of the sample image in the real clothing scene as the condition and the noisy image as the supervision.

[0130] In the inverse process, the noisy image of a forward process output is sampled from the spherical Gaussian function to obtain a noisy image of a long Markov chain, and the noisy image in the long Markov chain is denoised using the conditional latent variable containing the semantic of the clothing character (i.e., the text and style features of the sample image), to generate a character image in the style of the clothing scene.

[0131] Training the inverse diffusion process of the preset diffusion model with the second sample character as supervision and the text and style features of the sample image in the target scene as condition can ensure that the obtained target diffusion model can generate images that fit the target scene.

[0132] S23, splicing the sample character image and the background image in the target scene to obtain a sample image.

[0133] Specifically, the above S23 includes:

[0134] S231, obtaining a background image in a target scene, and identifying a region of interest of the background image to obtain a region of interest image.

[0135] When splicing the sample character image into the background image, it is generally spliced into the region of interest in the background image. For example, the character on the clothing is generally the upper body of a person, based on which the upper body of the person needs to be identified from the collected background image, and at this time the upper body of the person is the region of interest in the background image.

[0136] In the identification of the region of interest of the background image, the position of the region of interest is determined, and then a local image is cut out from the background image using the position of the region of interest to obtain a region of interest image.

[0137] S232, rotate the sample character image at an arbitrary angle to obtain a rotated sample character image.

[0138] The rotation angle of the sample character image is set according to actual needs. After rotating the sample character image at an arbitrary angle, a rotated sample character image is obtained. In order to facilitate the recording of the rotation angle, the rotation angle is defined as the acute angle between the short side of the rotation target frame and the positive direction of the x-axis, wherein the counterclockwise direction is designated as the positive angle and the clockwise direction is the negative angle. Therefore, the angle range is [-90, 90).

[0139] S233, splicing the rotated sample character image and the region of interest image to obtain a sample image.

[0140] The splicing method is to paste the rotated sample character image in the region of interest image to obtain a sample image.

[0141] Further, in order to obtain a large number of sample images, a large number of first sample characters with different styles can be used as input of the target diffusion model, real scene images under the target scene are collected to obtain target text and style features of the real scene images, and the target text and style features are used as conditions of the target diffusion model to generate a large number of sample character images, thereby generating a large number of sample images.

[0142] The sample image generation method provided in the embodiment extracts image features and text features from real character images respectively, and then fuses the two, so that the obtained target text and style features include both image features and text features, improving the reliability of the obtained target text and style features. The region of interest is identified to facilitate subsequent splicing of the sample character image to the region of interest to approach the use under the target scene; at the same time, the sample character image is rotated at an arbitrary angle, so that a large number of sample images can be generated, thereby enriching the number of sample images.

[0143] In the embodiment, a character detection model training method is provided, which can be used in electronic devices such as computers, servers, etc. Figure 5 is a flowchart of the character detection model training method according to the embodiment of the present application, as shown in Figure 5 The flowchart includes the following steps:

[0144] S31, obtaining a sample image.

[0145] The sample image is obtained according to the sample image generation method of any one of the above. For the generation of the sample image, please refer to the description above, which will not be repeated here.

[0146] S32, obtaining label data of the sample image.

[0147] The label data includes position information of the text content in the sample image and a target rotation angle.

[0148] The position information of the text content and the target rotation angle are recorded when the sample character image is spliced with the background image in the target scene.

[0149] S33, inputting the sample image into the character detection model to obtain predicted position information of the text content in the sample image and a predicted rotation angle.

[0150] The input of the character detection model is the sample image, and the output is the predicted position information of the text line in the sample image and the predicted rotation angle. The predicted position information can be represented in the form of a center point, length and width, or in the form of the upper left corner coordinates and the lower right corner coordinates of the predicted box, and the like, and is not limited in particular.

[0151] S34, updating the parameters of the character detection model based on the predicted position information, the predicted rotation angle and the label data to determine a target character detection model.

[0152] As a specific application example of the character detection model of the present embodiment, resnet-18 is used as the backbone network to build the character detection model, and four output feature map branches are connected after it, which are the target center point heat map, the sampling offset, the size of the target box and the rotation angle of the target box. The specific position of the target is predicted by training the four branches. Specifically, the obtained sample image is scaled to 3*320*192 resolution size as the input of the character detection model, and the output center point heat map, offset feature map, target box size feature map and angle feature map are obtained. Combined with the label of the sample image, the loss value between the label and the output of the character detection model is iteratively updated, and the parameters of the character detection model are trained to update the parameters of the character detection model, and finally the target character detection unit is determined.

[0153] Specifically, for the feature map size output by the backbone network, the heat map true value is calculated by the Gaussian kernel. When the position of the enumeration block and the real center key point coordinates are close to coincide, the Gaussian kernel output value is close to 1; when the position of the enumeration block and the real center key point are very different, the Gaussian kernel output value is close to 0. The output size of the center point heat map is one fourth of the original image size. Since the detection target is only one category of character box, the output heat map size is 80*48*1. In the training process, the center point heat map is trained by focal loss. In the inference process, the center point heat map is calculated by 3*3 maximum pooling to obtain the center point that meets the detection threshold.

[0154] Since the image is down-sampled by R=4, the offset of the center point in the x-axis and y-axis is calculated according to the feature map down-sampling rate, a loss function about the offset is set, so that the trained network can compensate the center point offset value and correct the position of the detection frame. The branch outputs a feature map with a size of 80*48*2, which is used to predict the offset in the x-axis and y-axis. The offset value is trained by L1 loss.

[0155] The size feature map output by the character detection model has a size of 80*48*2, and the two channels correspond to the prediction of the height and width, respectively. The predicted height and width are mapped to the original image size, and the branch is trained by L1 loss, so that the height and width predicted by the character detection model gradually converge to the original image character frame.

[0156] The character detection model adds an angle information detection head after the feature extractor of the backbone network. The feature map has a size of 80*48*1 and is used to predict the rotation angle of the target frame. The target angle error is fed back to the character detection model, so that the character detection model can learn the angle information of the target.

[0157] The training method of the character detection model provided in the embodiment is based on a large number of realistic sample images obtained in a target scene, and the training of the character detection model can improve the accuracy of the obtained target character detection model. The target character detection model obtained by training also outputs the rotation angle of the text content, which improves the detection accuracy of the target character detection model. In addition, the detected text content can be rotated using the rotation angle during subsequent character recognition to further improve the accuracy of character recognition.

[0158] In the embodiment, a training method of a character recognition model is provided, which can be used in electronic devices such as computers, servers, etc. Figure 6 is a flowchart of the training method of the character recognition model according to an embodiment of the present application, as shown in Figure 6 The flowchart includes the following steps:

[0159] S41, obtaining a sample image.

[0160] The sample image is obtained by the sample image generation method of any one of the above. For the generation of the sample image, please refer to the description above, which will not be repeated here.

[0161] S42, inputting the sample image into a target character detection model to obtain the position and rotation angle of the text content of the sample image.

[0162] The target character detection model is trained by the training method of the character detection model described above. For the specific details of the target character detection model, please refer to the description above, which will not be repeated here.

[0163] S43, rotate the text content in the sample image by using the position of the text content and the rotation angle to obtain target text content.

[0164] The rotation manner of the text content can be to extract a text line by using the position of the text content, and then rotate the text line by using the rotation angle to obtain the target text content. Alternatively, the text content can be trapezoidal corrected by using the position of the text content and the rotation angle. Specifically, a perspective transformation matrix is calculated by using the position and the rotation angle, and then the text content in the sample image is perspective transformed by using the perspective transformation matrix to obtain the target text content.

[0165] Of course, other manners can also be adopted to rotate the text content to obtain the target text content on the basis of the position of the text content and the rotation angle, which are not limited herein.

[0166] S44, update the parameters of the character recognition model based on the target text content and the text label of the sample image to obtain a target character recognition model.

[0167] The character recognition model can be a pre-trained character recognition model, for example, it can be a pre-trained English character and number character classifier for recognizing English and numbers. In the training process, the parameters of the character recognition model are adjusted to fix the parameters of the character recognition model. The parameter adjustment process can be to scale the generated sample image to 3*32*384 resolution according to the original image width and height ratio. If the width of the original image is less than 384 after scaling, the three channels are filled with the mean value of the three channels, and if the width is greater than 384, the image is truncated to 384 width and sent into the pre-trained character recognition model in batches. The character recognition model is built with resnet18 as the backbone network, and the 52 English uppercase and lowercase letters and 10 numbers are classified by using the softmax activation function. The CTClOSS for text classification is used for training, and the repeated characters are set as blank. As a class, the final classification feature map is 1*48*63, and the CTClOSS is calculated on this feature map. The network parameters are continuously adjusted by using the Adam optimizer to fit the correct string, so as to fix the parameters of the character recognition model and obtain the pre-trained character recognition model. Further, the pre-trained character recognition model is applied to the target scene, the sample image is detected by using the target character detection model, the text content is rotated based on the position and rotation angle of the text content, the loss of the target text content and the sample image text label is calculated by using the pre-trained character recognition model, and the parameters of the pre-trained character recognition model are updated based on the loss to obtain the target character recognition model, so that the target character recognition model is more suitable for character / text recognition in the target scene.

[0168] The training method of the character recognition model provided in the embodiment is used for rotating and correcting the text content by using the predicted position and rotation angle before the detected text line is input into the character recognition model, so as to ensure that the angles of the obtained target text content are uniform, and then the parameters of the character recognition model are updated by using the target text content with uniform angles, thereby further improving the accuracy of the obtained target character recognition model.

[0169] A character recognition method is provided in the embodiment, which can be used for electronic devices such as computers, servers, mobile terminals and the like. Figure 8 The flowchart of the character recognition method according to the embodiment of the present application is shown in FIG. 3, which includes the following steps: Figure 8

[0170] S51, obtaining an image to be processed in a target scene.

[0171] The image to be processed is an image in the target scene, which can be obtained by a collection device in the target scene, or stored in an electronic device, etc., and the source thereof is not limited herein.

[0172] S52, inputting the image to be processed into a target character detection model to obtain the position and rotation angle of the text content of the image to be processed.

[0173] The target character detection model is obtained by using the training method of the character detection model described above. The specific structure of the target character detection model is described above and will not be repeated herein.

[0174] S53, rotating the text content of the image to be processed by using the position and rotation angle of the text content of the image to be processed to obtain text content to be recognized.

[0175] The specific rotation process is similar to the rotation process in S43 above, and will not be repeated herein.

[0176] S54, inputting the text content to be recognized into a target character recognition model to obtain a character recognition result of the image to be processed.

[0177] The target character recognition model is obtained by using the training method of the character recognition model described above.

[0178] ​In some embodiments, if the trained target character recognition model can only recognize English characters and numbers, the target character recognition model can detect Chinese characters, resulting in misclassification of the target character recognition model, i.e., outputting a long English and number mixed code. Based on this, a proportion threshold of the number of Chinese and English occurrences is set. Since the prediction score of the Chinese character is generally low, when the ratio of the number of characters below the classification score threshold to the length of the string is greater than the set proportion threshold, the string is considered to be Chinese and is not output. Otherwise, the character recognition result is output.

[0179] Since the rotation correction is dependent on the output of the target character detection model and is not independent of the rotation detection, the text content is rotated to obtain the corrected to-be-recognized text content, and then character recognition is performed, thereby improving the accuracy of the character recognition result.

[0180] The character recognition method provided in this embodiment has high character detection and character recognition accuracy since the target character detection model and the target character recognition model are trained using a large number of realistic sample images. The target character detection model and the target character recognition model are used to perform character recognition on the to-be-processed image, and an accurate character recognition result can be obtained.

[0181] In this embodiment, a sample image generation device, a character detection model training device, a character recognition model training device, and a character recognition device are also provided. The devices are used to implement the above-described embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware or a combination of software and hardware can also be implemented and conceived.

[0182] The embodiment provides a sample image generation device, as shown in Figure 9 The device includes:

[0183] The first acquisition module 61 is configured to acquire target text and style features in a target scene and first sample characters, wherein the target text and style features are obtained by performing text style encoding on real character images in the target scene.

[0184] The style processing module 62 is configured to process the first sample characters and the target text and style features based on a target diffusion model to obtain sample character images in the target scene, wherein the target diffusion model is used to migrate the font style of the first sample characters to the real character images to generate the sample character images.

[0185] The splicing module 63 is configured to splice the sample character image and the background image in the target scene to obtain a sample image.

[0186] In some embodiments, the first acquisition module 61 comprises:

[0187] The first acquisition unit is configured to acquire the real character image.

[0188] The first feature extraction unit is configured to perform image feature extraction on the real character image to obtain an image style code.

[0189] The second feature extraction unit is configured to perform text feature extraction on the text content in the real character image to obtain a text code.

[0190] The fusion unit is configured to fuse the image style code and the text code to obtain the target text and style feature.

[0191] In some embodiments, the fusion unit comprises:

[0192] The attention processing subunit is configured to perform attention processing on the image style code and the text code to obtain attention between the image style code and the text code.

[0193] The fusion subunit is configured to fuse the attention and the text code, and perform processing on the fusion result through a feedforward network to obtain the target text and style feature.

[0194] In some embodiments, the splicing module 63 comprises:

[0195] The second acquisition unit is configured to acquire the background image in the target scene, and identify a region of interest in the background image to obtain a region of interest image.

[0196] The first rotation unit is configured to rotate the sample character image by an arbitrary angle to obtain a rotated sample character image.

[0197] The splicing unit is configured to splice the rotated sample character image and the region of interest image to obtain the sample image.

[0198] In some embodiments, the training module of the target diffusion model comprises:

[0199] The fourth acquisition unit is configured to acquire a text and style feature of a second sample character and a sample image in the target scene.

[0200] The generation unit is configured to generate a noise image based on the second sample character in a forward diffusion process of a preset diffusion model.

[0201] a training unit configured to train a reverse diffusion process of the preset diffusion model based on the noise image and the text and style features of the sample image in the target scene, to determine the target diffusion model.

[0202] The embodiment provides a character detection model training device, as shown in the accompanying drawings, which comprises: Figure 10

[0203] The second acquisition module 71 is configured to acquire a sample image, wherein the sample image is obtained according to the sample image generation method in the first aspect of the present application or any one of the implementation manners of the first aspect.

[0204] The third acquisition module 72 is configured to acquire label data of the sample image, wherein the label data comprises position information of text content of the sample image and a target rotation angle.

[0205] The first prediction module 73 is configured to input the sample image into the character detection model, to obtain predicted position information of the text content of the sample image and a predicted rotation angle.

[0206] The first update module 74 is configured to update parameters of the character detection model based on the predicted position information, the predicted rotation angle and the label data, to obtain a target character detection model.

[0207] The embodiment provides a character recognition model training device, as shown in the accompanying drawings, which comprises: Figure 11

[0208] The fourth acquisition module 81 is configured to acquire a sample image, wherein the sample image is obtained according to the sample image generation method in the first aspect of the present application or any one of the implementation manners of the first aspect.

[0209] The first detection module 82 is configured to input the sample image into a target character detection model, to obtain position and rotation angle of text content of the sample image, wherein the target character detection model is obtained by training according to the character detection model training method in the second aspect of the present application.

[0210] The first rotation module 83 is configured to rotate the text content in the sample image by using the position of the text content and the rotation angle, to obtain target text content.

[0211] The second update module 84 is configured to update parameters of a character recognition model based on the target text content and a text label of the sample image, to obtain a target character recognition model.

[0212] The embodiment provides a character recognition device, as shown in the accompanying drawings, which comprises: Figure 12 ​​As shown, comprising:

[0213] A fifth acquisition module 91 is configured to acquire a to-be-processed image in a target scene;

[0214] A second detection module 92 is configured to input the to-be-processed image into a target character detection model to obtain a position and a rotation angle of Chinese text content of the to-be-processed image, wherein the target character detection model is obtained by training the character detection model according to the training method of the character detection model in the second aspect of the present application.

[0215] A second rotation module 93 is configured to rotate the Chinese text content of the to-be-processed image according to the position and the rotation angle of the Chinese text content of the to-be-processed image to obtain to-be-recognized text content.

[0216] An identification module 94 is configured to input the to-be-recognized text content into a target character recognition model to obtain a character recognition result of the to-be-processed image, wherein the target character recognition model is obtained by training the character recognition model according to the training method of the character recognition model in the third aspect of the present application.

[0217] The training device of the character recognition model and the character recognition device in the embodiment are presented in the form of functional units, and the units herein refer to ASIC circuits, processors and memories for executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0218] Further function descriptions of the above modules are the same as those of the corresponding embodiments, and will not be described here.

[0219] The embodiment of the present application also provides an electronic device having the above Figure 9 the sample image generation device shown in the embodiment, or Figure 10 the character detection model training device shown in the embodiment, or Figure 11 the character recognition model training device shown in the embodiment, or Figure 12 the character recognition device shown in the embodiment.

[0220] Please refer to Figure 13 , Figure 13 is a structural schematic diagram of a terminal provided by an optional embodiment of the present application, as shown in Figure 13As shown, the terminal can include at least one processor 101, such as a CPU (Central Processing Unit), at least one communication interface 103, a memory 104, and at least one communication bus 102. The communication bus 102 is used to realize the connection and communication between the components. The communication interface 103 can include a display, a keyboard, and can also include a standard wired interface and a wireless interface. The memory 104 can be a high-speed RAM memory, or a non-volatile memory such as at least one disk memory. The memory 104 can also be at least one storage device located away from the aforementioned processor 101. The processor 101 can be combined with Figure 9 or Figure 10 or Figure 11 or Figure 12 The described apparatus, the memory 104 stores an application program, and the processor 101 invokes the program code stored in the memory 104 to execute any of the above method steps.

[0221] The communication bus 102 can be a PCI (peripheral component interconnect) bus or an EISA (extended industry standard architecture) bus, etc. The communication bus 102 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 13 Only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0222] The memory 104 can include volatile memory such as RAM, and can also include non-volatile memory such as flash memory, a hard disk (HDD) or a solid-state disk (SSD), and can also include a combination of the above types of memory.

[0223] The processor 101 can be a central processing unit (CPU), a network processor (NP), or a combination of the CPU and the NP.

[0224] The processor 101 can further include a hardware chip. The hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0225] Optionally, the memory 104 is further configured to store program instructions. The processor 101 can invoke the program instructions to implement the sample image generation method, the character detection model training method, the character recognition model training method, or the character recognition method as shown in any of the embodiments of the present application.

[0226] The embodiments of the present application also provide a non-transitory computer storage medium storing computer executable instructions. The computer executable instructions can execute the sample image generation method, the character detection model training method, the character recognition model training method, or the character recognition method in any of the method embodiments. The storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or the like. The storage medium can also include a combination of the above-mentioned storage media.

[0227] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes are intended to fall within the scope of the appended claims.

Claims

1. A method for generating a sample image, characterized in that, include: The target text and style features in the target scene, as well as the first sample character, are obtained by text style encoding of the real character image in the target scene. The first sample character, the target text, and style features are processed based on the target diffusion model to obtain a sample character image in the target scene. The target diffusion model is used to transfer the font style of the first sample character to the real character image to generate the sample character image. The sample character image is stitched together with the background image of the target scene to obtain the sample image; The process of obtaining target text and style features in a target scene includes: acquiring the real character image; extracting image features from the real character image to obtain an image style code; extracting text features from the text content in the real character image to obtain a text code; performing attention processing on the image style code and the text code to obtain an attention between the image style code and the text code; fusing the attention with the text code, and processing the fusion result through a feedforward network to obtain the target text and style features. The training method for the target diffusion model includes: acquiring second sample characters and text and style features of sample images in the target scene; generating a noise image based on the second sample characters during the forward diffusion process of the preset diffusion model; using the noise image as supervision and the text and style features of the sample images in the target scene as conditions to train the reverse diffusion process of the preset diffusion model to determine the target diffusion model.

2. The method according to claim 1, characterized in that, The step of concatenating the sample character image with the background image of the target scene to obtain the sample image includes: Acquire the background image of the target scene, and identify the region of interest in the background image to obtain the region of interest image; The sample character image is rotated at an arbitrary angle to obtain the rotated sample character image; The rotated sample character image is stitched together with the region of interest image to obtain the sample image.

3. A training method for a character detection model, characterized in that, include: Obtain a sample image, wherein the sample image is obtained by the sample image generation method according to any one of claims 1-2; Obtain the label data of the sample image, the label data including the position information of the text content in the sample image and the target rotation angle; The sample image is input into the character detection model to obtain the predicted position information and predicted rotation angle of the text content in the sample image; Based on the predicted position information, the predicted rotation angle, and the label data, the parameters of the character detection model are updated to obtain the target character detection model.

4. A training method for a character recognition model, characterized in that, include: Obtain a sample image, wherein the sample image is obtained by the sample image generation method according to any one of claims 1-2; The sample image is input into the target character detection model to obtain the position and rotation angle of the text content in the sample image. The target character detection model is trained by the training method of the character detection model according to claim 3. The target text content is obtained by rotating the text content in the sample image using the position of the text content and the rotation angle. Based on the target text content and the text labels of the sample images, the parameters of the character recognition model are updated to obtain the target character recognition model.

5. A character recognition method, characterized in that, include: Acquire the image to be processed in the target scene; The image to be processed is input into the target character detection model to obtain the position and rotation angle of the text content in the image to be processed. The target character detection model is trained by the training method of the character detection model according to claim 3. The text content in the image to be processed is rotated using the position and rotation angle of the text content in the image to be processed, and the text content to be recognized is obtained. The text content to be recognized is input into the target character recognition model to obtain the character recognition result of the image to be processed. The target character recognition model is trained by the training method of the character recognition model according to claim 4.

6. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the sample image generation method of any one of claims 1-2, or to perform the character detection model training method of claim 3, or to perform the character recognition model training method of claim 4, or to perform the character recognition method of claim 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the sample image generation method according to any one of claims 1-2, or to perform the character detection model training method according to claim 3, or to perform the character recognition model training method according to claim 4, or to perform the character recognition method according to claim 5.

Images