A method, device and electronic device for generating sample images

Abstract

The present application discloses a method, device and electronic device for generating a sample image. First, a character sequence in a first image is determined based on a character recognition model. Then, for the first character in the character sequence, a second character corresponding to the first character is selected from a character database, and the second character is superimposed in the area corresponding to the first character in the first image to obtain a second image. Then, the second image is input into a pre-trained image generation model, and a sample image having the same background as the first image and containing only the second character in the area corresponding to the first character is generated based on the image generation model. And the label information of the sample image is determined according to the first position information of the second character and the second character in the sample image. The difficulty of collecting new sample images is reduced, the problem of low efficiency of manual annotation is solved, and the accuracy of label information is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a sample image generation method, device and electronic device. Background Art

[0002] In the field of computer vision technology, character recognition models are increasingly widely used. Character recognition models are used to identify the position and content of characters in an image. After the character recognition model is trained, in order to improve the recognition ability of the character recognition model, it is generally necessary to collect new sample images containing characters and manually annotate the character content in the sample images to enrich the training set. Then, the character recognition model is optimized and trained based on the new training set to obtain a character recognition model with better recognition ability.

[0003] The existing solutions for enriching training sets have the following problems: first, it is difficult to collect new sample images. In some scenarios, the number of sample images is prioritized, and it is difficult to obtain a large number of sample images, which makes it difficult to enrich the training set and the efficiency of manual labeling is low; second, the character content in the manually labeled sample images may be wrong. For example, the number 0 in the image may be mistakenly labeled as the letter O, which will make the label information of the sample image inaccurate, thereby affecting the accuracy of subsequent character recognition model optimization training. Summary of the invention

[0004] The present application provides a sample image generation method, device and electronic device to solve the problems in the prior art that it is difficult to enrich the training set, the efficiency is low, and the manual annotation of images results in poor accuracy of the label information of the sample images.

[0005] In a first aspect, the present application provides a sample image generation method, the method comprising:

[0006] Acquire a first image, and determine a character sequence in the first image based on a character recognition model;

[0007] Determine a first character in the character sequence, select a second character corresponding to the first character from a character database, and superimpose the second character on a region corresponding to the first character in the first image to obtain a second image;

[0008] The second image is input into a pre-trained image generation model, and a sample image is generated based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; and the label information corresponding to the sample image is determined based on the second character and the first position information of the second character in the sample image.

[0009] The above technical solution has the following advantages or beneficial effects:

[0010] In the present application, firstly, the character sequence in the first image is determined based on the character recognition model, and then for the first character in the character sequence, the second character corresponding to the first character is selected from the character database, and the second character is superimposed in the area corresponding to the first character in the first image to obtain the second image; then the second image is input into the pre-trained image generation model, and a sample image with the same background as the first image and only containing the second character in the area corresponding to the first character is generated based on the image generation model; and the label information of the sample image is determined according to the first position information of the second character and the second character in the sample image. The present application provides a solution for generating a new sample image by replacing characters in the first image, and determining the label information of the sample image at the same time. Compared with the prior art of collecting new sample images containing characters and manually annotating the character content in the sample images, the solution of enriching the training set reduces the difficulty of collecting new sample images, solves the problem of low efficiency of manual annotation, and improves the accuracy of label information, thereby providing a basis for subsequent accurate optimization and training of the character recognition model.

[0011] In an optional embodiment, determining the first character in the character sequence includes:

[0012] Receive a character area selection instruction, obtain second position information of the selection area carried in the character area selection instruction, and determine the character in the character sequence located in the selection area as the first character based on the third position information and the second position information of each character in the character sequence.

[0013] The above technical solution has the following advantages or beneficial effects:

[0014] In order to improve the flexibility of character replacement, the present application can determine that the character in the area corresponding to the character area selection instruction is the first character based on the received character area selection instruction, and then replace the first character. Specifically, the character area selection instruction can be an instruction to select an area in the first image. The position information of the selection is the second position information of the selection area carried in the character area selection instruction. Then, the third position information of each character in the character sequence is obtained, and based on the third position information of each character and the second position information of the selection area, the character in the character sequence that is located in the selection area can be determined, and the character is determined as the first character. Thereby, the specified character is selected for replacement according to the character area selection instruction, and the flexibility of character replacement is improved.

[0015] In an optional embodiment, selecting a second character corresponding to the first character from a character database includes:

[0016] A character selection instruction for replacing the first character is received, a replacement character in the character database carried in the character selection instruction is obtained, and the replacement character is determined to be a second character corresponding to the first character.

[0017] The above technical solution has the following advantages or beneficial effects:

[0018] In order to improve the flexibility of character replacement, the present application can specify which character to replace the first character with, that is, carry the second character used to replace the first character in the character selection instruction. Specifically, the replacement character in the character database carried in the character selection instruction used to replace the first character is determined to be the second character corresponding to the first character, and then the subsequent step of superimposing the specified second character in the area corresponding to the first character in the first image to obtain the second image is performed. This improves the flexibility of character replacement and can adapt to user needs.

[0019] In an optional embodiment, selecting a second character corresponding to the first character from a character database includes:

[0020] Determine a similar character set and a non-similar character set corresponding to the first character in the character database; select a second character corresponding to the first character from the similar character set or the non-similar character set according to a preset selection probability; wherein the probability of selecting from the similar character set is greater than the probability of selecting from the non-similar character set.

[0021] The above technical solution has the following advantages or beneficial effects:

[0022] In order to improve the flexibility of character replacement, the present application pre-stores a similar character set and a non-similar character set corresponding to each character. After determining the first character, the similar character set and the non-similar character set corresponding to the first character in the character database are determined. Then, the second character corresponding to the first character is selected from the similar character set or the non-similar character set according to the preset selection probability, and the probability of selecting from the similar character set is greater than the probability of selecting from the non-similar character set. On the one hand, the automation of the second character selection can be realized, and on the other hand, the greater probability of selecting the second character from the similar character set ensures the similarity of the sample images generated subsequently, which can improve the accuracy of the optimized training of the character recognition model based on the generated sample images and improve the performance of the optimized trained character recognition model.

[0023] In an optional embodiment, superimposing the second character in a region corresponding to the first character in the first image to obtain the second image includes:

[0024] Obtain pixel values ​​of each pixel point contained in the first character in the first image, and determine the average pixel value of the pixel values ​​of each pixel point; superimpose the second character in the area corresponding to the first character in the first image, and set the pixel values ​​of each pixel point contained in the second character to the average pixel value, to obtain the second image.

[0025] The above technical solution has the following advantages or beneficial effects:

[0026] In order to ensure that the color of the characters in the generated sample image is close to the color of the characters in the first image, and thus the color of the characters in the generated sample image is close to the color of the characters in the first image, the character contents in the sample image and the first image are different. In this way, the performance of the optimized and trained character recognition model can be further improved by optimizing and training the character recognition model based on the sample image. Specifically, the pixel values ​​of each pixel point contained in the first character in the first image are obtained, and then the average pixel value of the pixel values ​​of each pixel point is determined; the second character is superimposed on the area corresponding to the first character in the first image, and the pixel values ​​of each pixel point contained in the second character are set to the average pixel value to obtain the second image. In this way, the color of the characters in the second image is close to the color of the characters in the first image, and when the sample image is generated based on the second image, the color of the characters in the sample image can be close to the color of the characters in the first image.

[0027] In an optional embodiment, the training process of the image generation model includes:

[0028] Acquire a first sample image, determine a sample character sequence in the first sample image based on the character recognition model; determine a first sample character in the sample character sequence, select a second sample character corresponding to the first sample character from the character database, and superimpose the second sample character on a region corresponding to the first sample character in the first sample image to obtain a second sample image;

[0029] Inputting a second sample image and a corresponding label image into an image generation model to be trained; wherein the label image is an image with the same background as the first sample image and only contains the second sample characters in the area corresponding to the first sample characters;

[0030] A predicted image is determined based on the image generation model, and a loss value of the image generation model is determined according to the predicted image and the label image; and the image generation model is trained according to the loss value.

[0031] The above technical solution has the following advantages or beneficial effects:

[0032] In the present application, when training an image generation model, first obtain a first sample image, then determine a first sample character in the first sample image based on a character recognition model, then select a second sample character corresponding to the first sample character, and superimpose the second sample character in the area corresponding to the first sample character in the first sample image to obtain a second sample image. Then input the second sample image into the image generation model to be trained, determine a predicted image based on the image generation model; determine a loss value based on the predicted image and the label image, and then train the image generation model based on the loss value. The image generation model trained in this way has the ability to generate a new image based on the image superimposed with the input characters.

[0033] In an optional embodiment, determining the loss value of the image generation model according to the predicted image and the label image includes:

[0034] Determine a first loss value according to the predicted image and the label image;

[0035] Inputting the predicted image into the image generation model, performing inverse feature extraction on the predicted image based on the image generation model to obtain a sample predicted image corresponding to the predicted image; determining a second loss value according to the sample predicted image and the second sample image;

[0036] A loss value of the image generation model is determined according to the first loss value and the second loss value.

[0037] The above technical solution has the following advantages or beneficial effects:

[0038] In order to improve the performance of the image generation model, the present application refers to the loss value determined based on the predicted image and the label image as the first loss value. The predicted image is input into the image generation model, and the predicted image is subjected to reverse feature extraction based on the image generation model to obtain a sample predicted image corresponding to the predicted image. Optionally, the second sample image is input into the image generation model, and the image generation model can determine the predicted image by calculating the feature matrix and the second sample image; the predicted image is input into the image generation model, and the sample predicted image can be determined by calculating the inverse matrix of the feature matrix and the predicted image. Then, the loss value of the sample predicted image and the second sample image is determined as the second loss value. According to the first loss value and the second loss value, the loss value of the image generation model is determined. Then, the image generation model is trained according to the loss value. Thereby, the performance of the image generation model can be improved.

[0039] In an optional embodiment, determining the first loss value according to the predicted image and the label image includes:

[0040] Determine, based on a first feature extraction network in the image generation model, a first prediction feature vector corresponding to the predicted image and a first label feature vector corresponding to the label image; determine, based on an adversarial network in the image generation model, a first sub-loss value of the first prediction feature vector and the first label feature vector;

[0041] Based on the second feature extraction network in the image generation model, determine a second prediction feature vector corresponding to the background area sub-image in the predicted image and a second label feature vector corresponding to the background area sub-image in the label image; determine a second sub-loss value according to the second prediction feature vector and the second label feature vector;

[0042] Determine, based on a character recognition network in the image generation model, a predicted character corresponding to the predicted image, and determine a third sub-loss value according to the predicted character and a label character corresponding to the label image;

[0043] The first loss value is determined according to the first sub-loss value, the second sub-loss value and the third sub-loss value.

[0044] The above technical solution has the following advantages or beneficial effects:

[0045] In order to further improve the performance of the image generation model, the present application performs overall feature extraction on the predicted image and the label image based on the first feature extraction network to obtain the first predicted feature vector and the first label feature vector, and then determines the first sub-loss value of the first predicted feature vector and the first label feature vector. Based on the second feature extraction network, feature extraction is performed on the background area sub-image in the predicted image and the background area sub-image in the label image to obtain the second predicted feature vector and the second label feature vector, and then the second sub-loss value of the second predicted feature vector and the second label feature vector is determined; the background area sub-image refers to the part of the image except the area where the character is located. Based on the character recognition network in the image generation model, the predicted character corresponding to the predicted image is determined, and the third sub-loss value is determined according to the predicted character and the label character corresponding to the label image. Finally, the first loss value is determined according to the first sub-loss value, the second sub-loss value and the third sub-loss value. According to the first loss value and the second loss value, the loss value of the image generation model is determined. Then, the image generation model is trained according to the loss value. Thereby, the performance of the image generation model can be further improved.

[0046] In a second aspect, the present application provides a sample image generating device, the device comprising:

[0047] A determination module, configured to acquire a first image and determine a character sequence in the first image based on a character recognition model;

[0048] a superposition module, configured to determine a first character in the character sequence, select a second character corresponding to the first character from a character database, and superimpose the second character on a region corresponding to the first character in the first image to obtain a second image;

[0049] A generation module is used to input the second image into a pre-trained image generation model, generate a sample image based on the image generation model, wherein the sample image is an image with the same background as the first image and only contains the second character in the area corresponding to the first character; and determine the label information corresponding to the sample image based on the second character and the first position information of the second character in the sample image.

[0050] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0051] Memory, used to store computer programs;

[0052] The processor is used to implement the method when executing the program stored in the memory.

[0053] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the described method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 A schematic diagram of the first sample image generation process provided in this application;

[0056] Figure 2 A schematic diagram of the first image provided for this application;

[0057] Figure 3 A schematic diagram of selecting the first character provided by this application;

[0058] Figure 4 A second image schematic diagram provided for this application;

[0059] Figure 5 Schematic diagram of sample images provided for this application;

[0060] Figure 6 A schematic diagram of the second sample image generation process provided in this application;

[0061] Figure 7 A schematic diagram of the third sample image generation process provided for this application;

[0062] Figure 8 A schematic diagram of the fourth sample image generation process provided in this application;

[0063] Fig. 9 A schematic diagram of the training process of the image generation model provided for this application;

[0064] Fig.10 A schematic diagram of the process of determining the loss value of the image generation model provided in this application;

[0065] Fig.11 A schematic diagram of a process for determining a first loss value provided in this application;

[0066] Fig.12 Sample image generation flow chart provided for this application;

[0067] Fig.13 A diagram of the training architecture of the image generation model provided for this application;

[0068] Fig.14 A schematic diagram of the structure of a sample image generating device provided in this application;

[0069] Fig.15 This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0070] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0071] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

[0072] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0073] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0074] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0076] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0077] Figure 1 The first sample image generation process provided in this application is a schematic diagram, and the process includes the following steps:

[0078] S101: Acquire a first image, and determine a character sequence in the first image based on a character recognition model;

[0079] S102: determining a first character in the character sequence, selecting a second character corresponding to the first character from a character database, and superimposing the second character on a region corresponding to the first character in the first image to obtain a second image;

[0080] S103: Input the second image into a pre-trained image generation model, and generate a sample image based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; determine the label information corresponding to the sample image according to the second character and the first position information of the second character in the sample image.

[0081] The sample image generation method provided in the present application is applied to an electronic device, which may be a PC, a computer, a server, or the like.

[0082] The electronic device acquires a first image, which may be an image containing characters, and inputs the first image into a character recognition model, and a character sequence in the first image may be determined based on the character recognition model. Figure 2 The first image schematic diagram provided for this application is Figure 2 The "J5GXA" in the image is the character in the image. The first image is input into the character recognition model, and the character sequence "J5GXA" in the first image can be determined based on the character recognition model. It should be noted that in order to improve the accuracy of recognizing the character sequence, the first image can be first input into the character segmentation model, and the regional sub-image where the characters are located in the first image can be determined based on the character segmentation model. Then, the regional sub-image where the characters are located is input into the character recognition model, and the character sequence in the first image can be determined based on the character recognition model.

[0083] After determining the character sequence in the first image, the first character in the character sequence is determined. Optionally, each character in the character sequence can be used as the first character. Alternatively, the first character can be determined by the user through a box selection operation, that is, the character within the box selection range corresponding to the user's box selection operation is used as the first character. Figure 3 This is a schematic diagram of selecting the first character provided by this application. Figure 3 "XA" selected in the middle box are respectively used as the first character. For each first character, select the second character corresponding to the first character from the character database. Optionally, the character database generally contains numeric characters, uppercase characters, lowercase characters, etc. When selecting the second character corresponding to the first character from the character database, for each first character, a second character can be randomly selected from the character database as the second character corresponding to the first character. Or the second character corresponding to the first character can be specified by the user. For example, for the first character "X", the user can specify that the second character is "A". Select the second character corresponding to the first character from the character database, and superimpose the second character on the area corresponding to the first character in the first image to obtain the second image. The area corresponding to the first character refers to the smallest rectangular area containing the first character. Superimpose the second character on the area corresponding to the first character in the first image. Optionally, the area sub-image where the second character is located can account for 60%-90% of the area sub-image corresponding to the first character. Figure 4 This is a second image schematic diagram provided by this application. Figure 4 As shown, the second character corresponding to the first character "X" is "U", and the second character corresponding to the first character "A" is "B".

[0084] The second image is input into a pre-trained image generation model, and a sample image is generated based on the image generation model. Figure 5 The sample image diagram provided for this application is as follows: Figure 5 As shown, the character sequence in the sample image is "J5GUB". The sample image has the same background as the first image. For the part where the characters are replaced, the area corresponding to the first character only contains the image of the second character. For the part where the characters are not replaced, the original characters in the first image are displayed. In addition, the label information corresponding to the sample image can be determined according to the second character and the first position information of the second character in the sample image. It should be noted that if there are unreplaced characters in the sample image, the label information corresponding to the sample image can be determined according to the unreplaced characters, the second character and the first position information of each character in the sample image in the sample image. The label information includes character content and character position. Among them, for the unreplaced characters in the sample image, when the character sequence in the first image is determined based on the character recognition model, the content and position of the unreplaced characters are determined. For the replaced characters in the sample image, the second character is superimposed in the area corresponding to the first character in the first image. After obtaining the second image, the position of the second character in the second image can be determined, and the position can be used as the position of the second character in the sample image. The position and content of the second character are also determined. Therefore, the label information corresponding to the sample image can be determined. That is, the content and position of each character in the sample image are determined.

[0085] The tag information is, for example:

[0086] "Transcription": "J", "points": [(158,54), (108,54), (108,82]), (158,82)], "difficult": false);

[0087] "Transcription": "5", "points": [(159, 80), (107, 80), (159, 107]), (107, 107)], "difficult": false);

[0088] "Transcription": "G", "points": [(159, 109), (109, 109), (109, 135]), (159, 135)], "difficult": false);

[0089] "Transcription": "U", "points": [(159, 135), (111, 135), (111, 161]), (159, 161)], "difficult": false);

[0090] "Transcription": "B", "points": [(159, 161), (111, 161), (111, 188]), (159, 188)], "difficult": false).

[0091] Subsequently, the character recognition model can be optimized and trained based on the sample images and the corresponding label information to obtain a character recognition model with better performance.

[0092] In the present application, firstly, the character sequence in the first image is determined based on the character recognition model, and then for the first character in the character sequence, the second character corresponding to the first character is selected from the character database, and the second character is superimposed in the area corresponding to the first character in the first image to obtain the second image; then the second image is input into the pre-trained image generation model, and a sample image with the same background as the first image and only containing the second character in the area corresponding to the first character is generated based on the image generation model; and the label information of the sample image is determined according to the first position information of the second character and the second character in the sample image. The present application provides a solution for generating a new sample image by replacing characters in the first image, and determining the label information of the sample image at the same time. Compared with the prior art of collecting new sample images containing characters and manually annotating the character content in the sample images, the solution of enriching the training set reduces the difficulty of collecting new sample images, solves the problem of low efficiency of manual annotation, and improves the accuracy of label information, thereby providing a basis for subsequent accurate optimization and training of the character recognition model.

[0093] Figure 6 A schematic diagram of a second sample image generation process provided in this application, the process includes the following steps:

[0094] S201: Acquire a first image, and determine a character sequence in the first image based on a character recognition model;

[0095] S202: receiving a character area selection instruction, obtaining second position information of the selection area carried in the character area selection instruction, determining a character in the character sequence located in the selection area as a first character according to the third position information and the second position information of each character in the character sequence; selecting a second character corresponding to the first character from a character database, and superimposing the second character in the area corresponding to the first character in the first image to obtain a second image;

[0096] S203: Input the second image into a pre-trained image generation model, and generate a sample image based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; determine the label information corresponding to the sample image based on the second character and the first position information of the second character in the sample image.

[0097] After the electronic device acquires the first image, the first image can be displayed in the display interface of the electronic device. Then the user inputs a character area selection instruction to the electronic device in the display interface. The electronic device receives the character area selection instruction, and acquires the second position information of the selection area carried in the character area selection instruction. The second position information of the selection area carried in the character area selection instruction is the second position information of the area included in the user's box selection operation in the display interface. When determining the character sequence in the first image based on the character recognition model, the third position information of each character in the character sequence can be determined. According to the second position information of the selection area and the third position information of each character, the characters located in the selection area can be determined, and the characters located in the selection area are determined as the first characters.

[0098] In order to improve the flexibility of character replacement, the present application can determine that the character in the area corresponding to the character area selection instruction is the first character based on the received character area selection instruction, and then replace the first character. Specifically, the character area selection instruction can be an instruction to select an area in the first image. The position information of the selection is the second position information of the selection area carried in the character area selection instruction. Then, the third position information of each character in the character sequence is obtained, and based on the third position information of each character and the second position information of the selection area, the character in the character sequence that is located in the selection area can be determined, and the character is determined as the first character. Thereby, the specified character is selected for replacement according to the character area selection instruction, and the flexibility of character replacement is improved.

[0099] Figure 7 A schematic diagram of a third sample image generation process provided in this application, the process includes the following steps:

[0100] S301: Acquire a first image, and determine a character sequence in the first image based on a character recognition model;

[0101] S302: determining a first character in the character sequence, receiving a character selection instruction for replacing the first character, obtaining a replacement character in the character database carried in the character selection instruction, determining the replacement character as a second character corresponding to the first character, and superimposing the second character in a region corresponding to the first character in the first image to obtain a second image;

[0102] S303: Input the second image into a pre-trained image generation model, and generate a sample image based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; determine the label information corresponding to the sample image based on the second character and the first position information of the second character in the sample image.

[0103] In order to improve the flexibility of character replacement, the present application can specify which character to replace the first character with, that is, carry the second character used to replace the first character in the character selection instruction. Specifically, the replacement character in the character database carried in the character selection instruction used to replace the first character is determined to be the second character corresponding to the first character, and then the subsequent step of superimposing the specified second character in the area corresponding to the first character in the first image to obtain the second image is performed. This improves the flexibility of character replacement and can adapt to user needs.

[0104] Figure 8 A schematic diagram of a fourth sample image generation process provided in this application, the process comprising the following steps:

[0105] S401: Acquire a first image, and determine a character sequence in the first image based on a character recognition model;

[0106] S402: Determine a first character in a character sequence, determine a similar character set and a non-similar character set corresponding to the first character in a character database; select a second character corresponding to the first character from the similar character set or the non-similar character set according to a preset selection probability; wherein the probability of selecting from the similar character set is greater than the probability of selecting from the non-similar character set; and superimpose the second character on a region corresponding to the first character in the first image to obtain a second image;

[0107] S403: Input the second image into a pre-trained image generation model, and generate a sample image based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; determine the label information corresponding to the sample image based on the second character and the first position information of the second character in the sample image.

[0108] For example, for the first character "B", its corresponding similar character set is "890OGDQ". The set of characters other than the characters in the similar character set is the non-similar character set corresponding to the first character "B". The probability of selecting from the similar character set is, for example, 80%, and the probability of selecting from the non-similar character set is, for example, 20%.

[0109] In order to improve the flexibility of character replacement, the present application pre-stores a similar character set and a non-similar character set corresponding to each character. After determining the first character, the similar character set and the non-similar character set corresponding to the first character in the character database are determined. Then, the second character corresponding to the first character is selected from the similar character set or the non-similar character set according to the preset selection probability, and the probability of selecting from the similar character set is greater than the probability of selecting from the non-similar character set. On the one hand, the automation of the second character selection can be realized, and on the other hand, the greater probability of selecting the second character from the similar character set ensures the similarity of the sample images generated subsequently, which can improve the accuracy of the optimized training of the character recognition model based on the generated sample images and improve the performance of the optimized trained character recognition model.

[0110] In an optional embodiment, superimposing the second character in a region corresponding to the first character in the first image to obtain the second image includes:

[0111] Obtain pixel values ​​of each pixel point contained in the first character in the first image, and determine the average pixel value of the pixel values ​​of each pixel point; superimpose the second character in the area corresponding to the first character in the first image, and set the pixel values ​​of each pixel point contained in the second character to the average pixel value, to obtain the second image.

[0112] In order to ensure that the color of the characters in the generated sample image is close to the color of the characters in the first image, and thus the color of the characters in the generated sample image is close to the color of the characters in the first image, the character contents in the sample image and the first image are different. In this way, the performance of the optimized and trained character recognition model can be further improved by optimizing and training the character recognition model based on the sample image. Specifically, the pixel values ​​of each pixel point contained in the first character in the first image are obtained, and then the average pixel value of the pixel values ​​of each pixel point is determined; the second character is superimposed on the area corresponding to the first character in the first image, and the pixel values ​​of each pixel point contained in the second character are set to the average pixel value to obtain the second image. In this way, the color of the characters in the second image is close to the color of the characters in the first image, and when the sample image is generated based on the second image, the color of the characters in the sample image can be close to the color of the characters in the first image.

[0113] Fig. 9 The training process diagram of the image generation model provided in this application includes the following steps:

[0114] S501: Acquire a first sample image, determine a sample character sequence in the first sample image based on the character recognition model; determine a first sample character in the sample character sequence, select a second sample character corresponding to the first sample character from the character database, and superimpose the second sample character on a region corresponding to the first sample character in the first sample image to obtain a second sample image;

[0115] S502: Inputting a second sample image and a corresponding label image into an image generation model to be trained; wherein the label image is an image with the same background as the first sample image and only contains the second sample characters in the area corresponding to the first sample characters;

[0116] S503: Determine a predicted image based on the image generation model, determine a loss value of the image generation model according to the predicted image and the label image; and train the image generation model according to the loss value.

[0117] In the present application, when training an image generation model, first obtain a first sample image, then determine a first sample character in the first sample image based on a character recognition model, then select a second sample character corresponding to the first sample character, and superimpose the second sample character in the area corresponding to the first sample character in the first sample image to obtain a second sample image. Then input the second sample image into the image generation model to be trained, determine a predicted image based on the image generation model; determine a loss value based on the predicted image and the label image, and then train the image generation model based on the loss value. The image generation model trained in this way has the ability to generate a new image based on the image superimposed with the input characters.

[0118] Fig.10 A schematic diagram of a process for determining the loss value of an image generation model provided in this application includes the following steps:

[0119] S601: Determine a first loss value according to the predicted image and the label image;

[0120] S602: inputting the predicted image into the image generation model, performing inverse feature extraction on the predicted image based on the image generation model to obtain a sample predicted image corresponding to the predicted image; determining a second loss value according to the sample predicted image and the second sample image;

[0121] S603: Determine a loss value of the image generation model according to the first loss value and the second loss value.

[0122] In order to improve the performance of the image generation model, the present application refers to the loss value determined based on the predicted image and the label image as the first loss value. The predicted image is input into the image generation model, and the predicted image is subjected to reverse feature extraction based on the image generation model to obtain a sample predicted image corresponding to the predicted image. Optionally, the second sample image is input into the image generation model, and the image generation model can determine the predicted image by calculating the feature matrix and the second sample image; the predicted image is input into the image generation model, and the sample predicted image can be determined by calculating the inverse matrix of the feature matrix and the predicted image. Then, the loss value of the sample predicted image and the second sample image is determined as the second loss value. According to the first loss value and the second loss value, the loss value of the image generation model is determined. Then, the image generation model is trained according to the loss value. Thereby, the performance of the image generation model can be improved.

[0123] Fig.11 The process diagram for determining the first loss value provided in this application includes the following steps:

[0124] S701: Determine, based on a first feature extraction network in the image generation model, a first prediction feature vector corresponding to the predicted image and a first label feature vector corresponding to the label image; determine, based on an adversarial network in the image generation model, a first sub-loss value of the first prediction feature vector and the first label feature vector;

[0125] S702: Determine, based on a second feature extraction network in the image generation model, a second prediction feature vector corresponding to the background area sub-image in the predicted image and a second label feature vector corresponding to the background area sub-image in the label image; determine a second sub-loss value according to the second prediction feature vector and the second label feature vector;

[0126] S703: Determine a predicted character corresponding to the predicted image based on a character recognition network in the image generation model, and determine a third sub-loss value according to the predicted character and a label character corresponding to the label image;

[0127] S704: Determine the first loss value according to the first sub-loss value, the second sub-loss value and the third sub-loss value.

[0128] In order to further improve the performance of the image generation model, the present application performs overall feature extraction on the predicted image and the label image based on the first feature extraction network to obtain the first predicted feature vector and the first label feature vector, and then determines the first sub-loss value of the first predicted feature vector and the first label feature vector. Based on the second feature extraction network, feature extraction is performed on the background area sub-image in the predicted image and the background area sub-image in the label image to obtain the second predicted feature vector and the second label feature vector, and then the second sub-loss value of the second predicted feature vector and the second label feature vector is determined; the background area sub-image refers to the part of the image except the area where the character is located. Based on the character recognition network in the image generation model, the predicted character corresponding to the predicted image is determined, and the third sub-loss value is determined according to the predicted character and the label character corresponding to the label image. Finally, the first loss value is determined according to the first sub-loss value, the second sub-loss value and the third sub-loss value. According to the first loss value and the second loss value, the loss value of the image generation model is determined. Then, the image generation model is trained according to the loss value. Thereby, the performance of the image generation model can be further improved.

[0129] The sample image generation method provided in this application has reliable characteristics:

[0130] Real-time: This application makes subsequent logical judgments based on the inference results of the deep learning model. Each model can process the images to be inferred from the optical inspection AOI device within a given time period, which meets the real-time characteristics of the trustworthy feature.

[0131] Reproducibility: The same image can be transmitted multiple times to the same batch of models for inference. The same results can be obtained by inference multiple times in different operating environments (different platforms, different inference times, and different times), which meets the reproducibility feature of the trustworthiness feature.

[0132] Controllability: This application can be turned on or off by the user at any time during use. The importance of the inference results and the degree of adoption can be intervened by the user, which conforms to the controllability feature of the trustworthy characteristic.

[0133] Based on a large number of different backgrounds, this application quickly generates a variety of sample images of different characters and corresponding label information based on specified preset strategies. The label information includes "character position" and "generated character content", and the label information corresponds one-to-one to the generated sample image.

[0134] The characters in the sample images generated by the present application are authentic, imitating the original character format and the background erasing effect is coherent, reducing the interference of irrelevant factors.

[0135] In the related art, the two-step operation of erasing first and then generating will lead to insufficient authenticity of the subsequently generated image if the erasing effect is not good enough. The present application integrates the two steps into one step. Each image is erased and new characters are generated in the same image generation model. There is no interference between the generation tasks of two adjacent images. In the process of batch generation, some images may not meet the requirements. In this case, images that meet the requirements can be manually selected to avoid batch quality issues.

[0136] The sample image generation method provided in this application aims to achieve accurate erasure of specified characters in an image containing characters, and on this basis automatically generate new characters with the same font style as the characters before erasure, supporting users to specify the generated content.

[0137] Input an image with characters, identify the content of the characters on the image surface, generate new character content according to preset rules, keep the image background unchanged, generate an image with new character content, extract the generated character content and position information, and save it in the annotation file to achieve the synchronous generation of sample images and annotation files.

[0138] In this application, the character area selection instruction can be used to achieve the erasure of all characters on the image, the erasure of characters at a specified position, the erasure of a specified number of characters, the erasure of characters of specified letters or numbers, the erasure of characters of specified contrast or coverage, etc. At the same time, it can also achieve image generation at a specified position, character generation of the same style as the erased characters, character generation of a specified number of digits, a specified number of numbers or letters, a specified number / letter ratio, starting or ending with a fixed number, and random generation with a fixed regular expression rule, etc.

[0139] Fig.12 The sample image generation flow chart provided for this application is as follows: Fig.12 As shown, input image A, detect whether there are characters in image A through character segmentation algorithm, if not, then generate any character of any font style at the specified position of image A, obtain sample image, and annotate the content and position of the generated character, and obtain the label corresponding to the sample image. If yes, determine the character sequence in image A based on the character recognition model; determine the first character to be replaced according to the manually input character area selection instruction; select the second character corresponding to the first character, and superimpose the second character in the area corresponding to the first character in image A to obtain image B; input image B into the image generation model, and generate sample image C and corresponding label information based on the image generation model.

[0140] Fig.13 The training architecture diagram of the image generation model provided for this application is as follows: Fig.13As shown, the image to be trained is input into the image generation model, and the predicted image is output based on the generator in the image generation model. The adversarial loss (first sub-loss value) of the predicted image and the label image is determined based on the discriminator; the text recognition loss (third sub-loss value) is determined based on the text recognition module OCR. According to the adversarial loss and text recognition loss, the parameters of the generator and the text recognition module OCR are updated by back propagation. In addition, the identity mapping loss (second sub-loss value) and the cycle consistency loss (second loss value) are determined. According to the adversarial loss, text recognition loss, identity mapping loss and cycle consistency loss, the overall loss is determined; according to the overall loss, the parameters of the generator are updated by back propagation, and finally the trained image generation model is obtained.

[0141] Fig.14 This is a schematic diagram of the structure of a sample image generation device provided in this application, the device comprising:

[0142] A determination module 141, configured to acquire a first image and determine a character sequence in the first image based on a character recognition model;

[0143] A superimposition module 142, configured to determine a first character in the character sequence, select a second character corresponding to the first character from a character database, and superimpose the second character on a region corresponding to the first character in the first image to obtain a second image;

[0144] A generation module 143 is used to input the second image into a pre-trained image generation model, generate a sample image based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; and determine the label information corresponding to the sample image based on the second character and the first position information of the second character in the sample image.

[0145] The superposition module 142 is specifically used to receive a character area selection instruction, obtain the second position information of the selection area carried in the character area selection instruction, and determine the character in the character sequence located in the selection area as the first character based on the third position information and the second position information of each character in the character sequence.

[0146] The superposition module 142 is specifically configured to receive a character selection instruction for replacing the first character, obtain a replacement character from the character database carried in the character selection instruction, and determine the replacement character as a second character corresponding to the first character.

[0147] The superposition module 142 is specifically used to determine a similar character set and a non-similar character set corresponding to the first character in the character database; select a second character corresponding to the first character from the similar character set or the non-similar character set according to a preset selection probability; wherein the probability of selecting from the similar character set is greater than the probability of selecting from the non-similar character set.

[0148] The overlay module 142 is specifically used to obtain the pixel value of each pixel point contained in the first character in the first image, and determine the average pixel value of the pixel values ​​of each pixel point; overlay the second character in the area corresponding to the first character in the first image, and set the pixel value of each pixel point contained in the second character to the average pixel value to obtain the second image.

[0149] The device also includes:

[0150] The training module 144 is used to obtain a first sample image, determine a sample character sequence in the first sample image based on the character recognition model; determine a first sample character in the sample character sequence, select a second sample character corresponding to the first sample character from the character database, and superimpose the second sample character on the area corresponding to the first sample character in the first sample image to obtain a second sample image; input the second sample image and the corresponding label image into the image generation model to be trained; wherein the label image is an image with the same background as the first sample image and only contains the second sample character in the area corresponding to the first sample character; determine a predicted image based on the image generation model, and determine a loss value of the image generation model according to the predicted image and the label image; and train the image generation model according to the loss value.

[0151] The training module 144 is specifically used to determine a first loss value based on the predicted image and the label image; input the predicted image into the image generation model, perform inverse feature extraction on the predicted image based on the image generation model, and obtain a sample predicted image corresponding to the predicted image; determine a second loss value based on the sample predicted image and the second sample image; and determine the loss value of the image generation model based on the first loss value and the second loss value.

[0152] The training module 144 is specifically used to determine the first predicted feature vector corresponding to the predicted image and the first label feature vector corresponding to the label image based on the first feature extraction network in the image generation model; determine the first sub-loss value of the first predicted feature vector and the first label feature vector based on the adversarial network in the image generation model; determine the second predicted feature vector corresponding to the background area sub-image in the predicted image and the second label feature vector corresponding to the background area sub-image in the label image based on the second feature extraction network in the image generation model; determine the second sub-loss value based on the second predicted feature vector and the second label feature vector; determine the predicted character corresponding to the predicted image based on the character recognition network in the image generation model, and determine the third sub-loss value based on the predicted character and the label character corresponding to the label image; determine the first loss value based on the first sub-loss value, the second sub-loss value and the third sub-loss value.

[0153] The present application also provides an electronic device, such as Fig.15 As shown, it includes: a processor 151, a communication interface 152, a memory 153 and a communication bus 154, wherein the processor 151, the communication interface 152, and the memory 153 communicate with each other through the communication bus 154;

[0154] The memory 153 stores a computer program, and when the program is executed by the processor 151, the processor 151 executes any one of the above method steps.

[0155] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0156] The communication interface 152 is used for communication between the above electronic device and other devices.

[0157] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0158] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0159] The present application also provides a computer storage readable storage medium, wherein the computer readable storage medium stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0160] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A sample image generation method, characterized in that: The method comprises: Acquire a first image, and determine a character sequence in the first image based on a character recognition model; Determine a first character in the character sequence, select a second character corresponding to the first character from a character database, and superimpose the second character on a region corresponding to the first character in the first image to obtain a second image; The second image is input into a pre-trained image generation model, and a sample image is generated based on the image generation model, wherein the sample image has the same background as the first image and only contains the second character in the area corresponding to the first character; and the label information corresponding to the sample image is determined based on the second character and the first position information of the second character in the sample image.

2. The method according to claim 1, characterized in that Determining a first character in the character sequence includes: Receive a character area selection instruction, obtain second position information of the selection area carried in the character area selection instruction, and determine the character in the character sequence located in the selection area as the first character based on the third position information and the second position information of each character in the character sequence.

3. The method according to claim 1, characterized in that The selecting a second character corresponding to the first character from a character database comprises: A character selection instruction for replacing the first character is received, a replacement character in the character database carried in the character selection instruction is obtained, and the replacement character is determined to be a second character corresponding to the first character.

4. The method according to claim 1, characterized in that The selecting a second character corresponding to the first character from a character database comprises: Determine a similar character set and a non-similar character set corresponding to the first character in the character database; select a second character corresponding to the first character from the similar character set or the non-similar character set according to a preset selection probability; wherein the probability of selecting from the similar character set is greater than the probability of selecting from the non-similar character set.

5. The method according to claim 1, characterized in that The second character is superimposed on a region corresponding to the first character in the first image to obtain the second image, comprising: Obtain pixel values ​​of each pixel point contained in the first character in the first image, and determine the average pixel value of the pixel values ​​of each pixel point; superimpose the second character in the area corresponding to the first character in the first image, and set the pixel values ​​of each pixel point contained in the second character to the average pixel value, to obtain the second image.

6. The method according to claim 1, characterized in that The training process of the image generation model includes: Acquire a first sample image, determine a sample character sequence in the first sample image based on the character recognition model; determine a first sample character in the sample character sequence, select a second sample character corresponding to the first sample character from the character database, and superimpose the second sample character on a region corresponding to the first sample character in the first sample image to obtain a second sample image; Inputting a second sample image and a corresponding label image into an image generation model to be trained; wherein the label image is an image with the same background as the first sample image and only contains the second sample characters in the area corresponding to the first sample characters; A predicted image is determined based on the image generation model, and a loss value of the image generation model is determined according to the predicted image and the label image; and the image generation model is trained according to the loss value.

7. The method according to claim 6, characterized in that Determining the loss value of the image generation model according to the predicted image and the label image includes: Determine a first loss value according to the predicted image and the label image; Inputting the predicted image into the image generation model, performing inverse feature extraction on the predicted image based on the image generation model to obtain a sample predicted image corresponding to the predicted image; determining a second loss value according to the sample predicted image and the second sample image; A loss value of the image generation model is determined according to the first loss value and the second loss value.

8. The method according to claim 7, characterized in that Determining a first loss value according to the predicted image and the label image includes: Determine, based on a first feature extraction network in the image generation model, a first prediction feature vector corresponding to the predicted image and a first label feature vector corresponding to the label image; determine, based on an adversarial network in the image generation model, a first sub-loss value of the first prediction feature vector and the first label feature vector; Based on the second feature extraction network in the image generation model, determine a second prediction feature vector corresponding to the background area sub-image in the predicted image and a second label feature vector corresponding to the background area sub-image in the label image; determine a second sub-loss value according to the second prediction feature vector and the second label feature vector; Determine, based on a character recognition network in the image generation model, a predicted character corresponding to the predicted image, and determine a third sub-loss value according to the predicted character and a label character corresponding to the label image; The first loss value is determined according to the first sub-loss value, the second sub-loss value and the third sub-loss value.

9. A sample image generating device, characterized in that: The device comprises: A determination module, configured to acquire a first image and determine a character sequence in the first image based on a character recognition model; a superposition module, configured to determine a first character in the character sequence, select a second character corresponding to the first character from a character database, and superimpose the second character on a region corresponding to the first character in the first image to obtain a second image; A generation module is used to input the second image into a pre-trained image generation model, generate a sample image based on the image generation model, wherein the sample image is an image with the same background as the first image and only contains the second character in the area corresponding to the first character; and determine the label information corresponding to the sample image based on the second character and the first position information of the second character in the sample image.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method according to any one of claims 1 to 8 when executing a program stored in a memory.

Images