Augmentation method, device, equipment and medium for magnetic code data in a ticket

Abstract

The disclosure provides an augmentation method for magnetic code data in a bill, which can be applied to the field of artificial intelligence technology. The method comprises: extracting image content in a magnetic code area from an original bill image in an original sample set to generate a magnetic code strip image; replacing the image content in the magnetic code area in the original bill image with pixels in the background in the magnetic code area of the original bill image to generate a background check image; converting the magnetic code strip image into a first to-be-fused image with a transparent background; fusing the image content of the first to-be-fused image into the magnetic code area in the background check image to obtain a first augmented image; and expanding the first augmented image to the original sample set. The disclosure also provides an augmentation device for magnetic code data in a bill, equipment, a storage medium and a program product.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to a method, device, equipment, medium and program product for augmenting magnetic code data in a bill. BACKGROUND

[0002] In the process of identifying the image of a bill, it is often necessary to first segment the bill image, for example, to segment different regions such as table regions, logo identification regions, seal regions, magnetic code regions, etc., and then train corresponding identification models (such as optical character recognition models, etc.) for different regions to identify the information of characters, icons or authenticity in them. In terms of segmenting the magnetic code region, since there are special non-universal characters or non-universal fonts in the magnetic code of some checks, it is often necessary to conduct targeted training to improve the ability of the model to accurately segment the magnetic code region. Then, the real samples of a type of bills with magnetic codes are usually few, and training with a small data set will cause overfitting, making it difficult for the model to accurately identify the magnetic code region in the check image. In order to increase the accuracy of the model in identifying the magnetic code region in the bill image, the data augmentation method can be used to increase the bill with magnetic code, thereby increasing the sample data in the training.

[0003] In the existing data augmentation method for the magnetic code in the bill image, the magnetic code region in the bill image is removed to retain the check background, and then a plurality of fragment images are generated according to the feature information of the magnetic code region, which are pasted to the check background to generate a data set. However, the image after pasting the fragment image leaves processing marks at the pasting boundary, resulting in a generated bill image that is significantly different from the real bill image, affecting the generalization ability of the model. SUMMARY

[0004] In view of the above problems, the present disclosure provides a method, device, equipment, medium and program product for augmenting magnetic code data in a bill, which can avoid the disadvantage that pasting a fragment image to a background image will leave marks, and make the generated bill image with magnetic code closer to the real bill image.

[0005] In a first aspect, an augmentation method for magnetic code data in a bill is provided. The method comprises: obtaining an original sample set, the original sample set comprising at least one original bill image and original position annotation information of a magnetic code region in the original bill image; based on the original position annotation information, extracting image content in the magnetic code region from the original bill image to generate a magnetic code bar image; replacing the image content in the magnetic code region in the original bill image with background pixels to generate a background check image; wherein the background pixels are pixels in the background of the magnetic code region in the original bill image; converting the magnetic code bar image into a first to-be-fused image with a transparent background; fusing the image content of the first to-be-fused image into the background check image to obtain a first augmented image; and expanding the first augmented image to the original sample set.

[0006] According to an embodiment of the present disclosure, the method further comprises: based on the fusion position of the image content of the first to-be-fused image in the first augmented image, obtaining first position annotation information of the magnetic code region in the first augmented image. The expanding the first augmented image to the original sample set further comprises: expanding the original sample set with the first augmented image and the first position annotation information.

[0007] According to an embodiment of the present disclosure, the method further comprises: based on characters randomly read from a character library and character layout settings, generating a magnetic code bar string, wherein the character library stores a set of characters used to generate a magnetic code; inserting the magnetic code bar string into the background check image according to a randomly generated offset value that satisfies a predetermined condition to generate a second augmented image, wherein the offset value comprises up, down, left and right offset values when the magnetic code bar string is inserted into the background check image; the predetermined condition is a value range of the up, down, left and right offset values; and expanding the second augmented image to the original sample set.

[0008] According to an embodiment of the present disclosure, the method further comprises: based on the offset value, determining second position annotation information of the magnetic code region in the second augmented image; and the expanding the second augmented image to the original sample set comprises: expanding the original sample set with the second augmented image and the second position annotation information.

[0009] According to an embodiment of the present disclosure, the converting the magnetic code bar image into a first to-be-fused image with a transparent background comprises: using a clustering algorithm to extract pixels where foreground characters in the magnetic code bar image are located to generate an intermediate transition image; and processing the intermediate transition image to generate the first to-be-fused image.

[0010] According to an embodiment of the present disclosure, the fusing the image content of the first to-be-fused image into the background check image to obtain a first augmented image comprises: generating a second to-be-fused image containing the image content of the first to-be-fused image, wherein the size of the second to-be-fused image is consistent with the size of the background check image, and the background of the second to-be-fused image is transparent; and superimposing the second to-be-fused image on the background check image to obtain the first augmented image.

[0011] According to an embodiment of the present disclosure, the obtaining the first position annotation information of the magnetic code region in the first augmented image comprises: based on the position of the image content of the first to-be-fused image in the second to-be-fused image when the second to-be-fused image is generated, setting the first position annotation information.

[0012] According to an embodiment of the present disclosure, the method further comprises: aggregating the generated magnetic code strip images to generate a set of magnetic code strip images; aggregating the generated background check images to generate a set of background check images; and before converting the magnetic code strip image into the first to-be-fused image and fusing the image content of the first to-be-fused image into the background check image, randomly selecting the magnetic code strip image and the background check image from the set of magnetic code strip images and the set of background check images, respectively.

[0013] According to an embodiment of the present disclosure, the method further comprises: training a character detection model using an augmented sample set obtained after augmenting the original sample set, so that the character detection model can identify a magnetic code region in an arbitrary check image and output position annotation information of the magnetic code region in the arbitrary check image. Wherein, based on the position annotation information, the magnetic code region can be located and segmented from the arbitrary check image when the arbitrary check image is image segmented.

[0014] In a second aspect, an apparatus for augmenting magnetic code data in a bill is provided. The apparatus includes an obtaining module, a feature extraction module, a background separation module, and an image fusion module. The obtaining module is configured to obtain an original sample set including at least one original bill image and original position annotation information of a magnetic code region in the original bill image. The feature extraction module is configured to extract image content in the magnetic code region from the original bill image based on the original position annotation information, and generate a magnetic code bar image. The background separation module is configured to replace the image content in the magnetic code region in the original bill image with background pixels in the magnetic code region in the original bill image, and generate a background check image. The image fusion module is configured to convert the magnetic code bar image into a first to-be-fused image with a transparent background, fuse the image content of the first to-be-fused image into the background check image to obtain a first augmented image, and augment the original sample set with the first augmented image.

[0015] According to an embodiment of the present disclosure, the image fusion module is further configured to obtain first position annotation information of the magnetic code region in the first augmented image based on a fusion position of the image content of the first to-be-fused image in the first augmented image, and augment the original sample set with the first augmented image and the first position annotation information.

[0016] According to an embodiment of the present disclosure, the apparatus further includes a random generation module. The random generation module is configured to generate a magnetic code bar string based on characters randomly read from a character library and character layout settings, where the character library stores a set of characters used to generate a magnetic code, insert the magnetic code bar string into the background check image according to a randomly generated offset value that satisfies a predetermined condition to generate a second augmented image, where the offset value includes up, down, left, and right offset values when the magnetic code bar string is inserted into the background check image, the predetermined condition is a value range of the up, down, left, and right offset values, and augment the original sample set with the second augmented image.

[0017] According to an embodiment of the present disclosure, the random generation module is further configured to determine second position annotation information of the magnetic code region in the second augmented image based on the offset value, and augment the original sample set with each second augmented image and its corresponding second position annotation information.

[0018] In a third aspect, an electronic device is provided. The electronic device includes one or more processors and memory. The memory is configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method described above.

[0019] The fourth aspect of the embodiments of the present disclosure further provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method described above.

[0020] The fifth aspect of the embodiments of the present disclosure further provides a computer program product comprising a computer program which, when executed by a processor, implements the method described above.

[0021] The one or more embodiments described above have the following advantages or beneficial effects: the drawbacks of traces caused by pasting the fragment image to the background image are avoided, and the background around the magnetic code characters in the generated first augmented image is the background pixels in the background check image, seamless fusion of the magnetic code character pixels and the background pixels in the background check image is achieved, and the generated check image is closer to the original check image obtained by scanning the paper check. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0023] Figure 1 An application scenario diagram of an augmentation method, device, equipment, medium and program product for magnetic code data in a check according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 2 A flowchart of an augmentation method for magnetic code data in a check according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 3 A flowchart of an augmentation method for magnetic code data in a check according to another embodiment of the present disclosure is schematically shown;

[0026] Figure 4 A model generalization method flowchart of expanding a check sample by using an augmentation method for magnetic code data in a check according to an embodiment of the present disclosure is schematically shown;

[0027] Figure 5 A magnetic code strip image generated according to an embodiment of the present disclosure is schematically shown;

[0028] Figure 6 A background check image generated according to an embodiment of the present disclosure is schematically shown;

[0029] Figure 7 A diagram schematically showing generation of a second augmented image according to a random generation method in an embodiment of the present disclosure is schematically shown;

[0030] Figure 8 A flowchart schematically showing generation of a to-be-fused image in an embodiment of the present disclosure is schematically shown.

[0031] Figure 9 A schematic diagram illustrating a first augmented image generated according to an image fusion method in an embodiment of the present disclosure is shown;

[0032] Figure 10 A block diagram of an augmentation device for magnetic code data in a bill according to an embodiment of the present disclosure is shown schematically; and

[0033] Figure 11 A block diagram of an electronic device adapted to implement an augmentation method for magnetic code data in a bill according to an embodiment of the present disclosure is shown schematically. DETAILED DESCRIPTION

[0034] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring aspects of the present disclosure.

[0035] The terms used herein are merely used to describe specific embodiments and are not intended to limit the present disclosure. The terms "include" and "have" and the like used herein indicate the presence of the described features, steps, operations, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

[0036] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present description, and should not be interpreted in an idealized or excessively formal manner.

[0037] In the case of using expressions similar to "at least one of A, B, and C, etc.", it is generally to be interpreted as including one or more of the same as the meaning generally understood by one of ordinary skill in the art (for example, "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.). The designations "first", "second", etc. used herein are merely for the purpose of distinguishing, and have no limiting meaning, and the number of any elements in the drawings is for illustration only and not limiting.

[0038] Key terms used herein are defined as follows:

[0039] Bill: refers to a check with a magnetic code;

[0040] Magnetic code: refers to a magnetic number, which is printed on the front of a check or other ticket in a specific position by using magnetic ink or magnetic ribbon to print the numbers 0-9 and four symbols;

[0041] Character detection model: after inputting a ticket image into the model, the region where the magnetic code on the ticket image is located (referred to as a magnetic code region) can be identified, and the magnetic code region is labeled, and a labeling result is output;

[0042] Data augmentation: refers to increasing the training data set used when training a model, so that the data set is as diverse as possible, so that the trained model has stronger generalization ability;

[0043] Small sample learning: usually, a machine learning model relies on a large number of data samples for learning, and too few samples will result in poor learning effect, and small sample learning refers to good learning effect in the case of only a small number of samples;

[0044] Generalization ability: refers to the learning and adaptation ability of a machine learning model to new samples. For data other than the learning set with the same rule, the trained model can also give correct output.

[0045] In order to avoid the disadvantage that pasting the fragment image to the background image will have traces, the embodiment of the disclosure provides a method, device, equipment, medium and program product for augmenting magnetic code data in a ticket, which can separate the magnetic code region from the original ticket image obtained by scanning a paper ticket, then generate a magnetic code strip image containing only the image content in the magnetic code region, and a background check image filled with the background image of the magnetic code region in the magnetic code region of the original ticket image, then a new ticket image containing magnetic code data can be generated by randomly fusing the magnetic code strip image and the background check image and the like. The generated ticket image is closer to the real sample, and the training effect of the model is better.

[0046] Figure 1 An application scenario diagram of the method, device, equipment, medium and program product for augmenting magnetic code data in a ticket according to the embodiment of the disclosure is schematically shown.

[0047] As shown in Figure 1 The application scenario 100 according to the embodiment can include a terminal device 101, a network 102 and a server 103. The network 102 is used as a medium to provide a communication link between the terminal device 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0048] A user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. The server 103 can be deployed with a character detection model.

[0049] The user can upload the original bill image obtained by scanning the paper bill to the server 103 through the terminal device 101.

[0050] The server 103 can perform the augmentation method for magnetic code data in bills of the embodiments of the present disclosure to generate an augmented image based on the original bill image uploaded by the user, to expand the data set for training the character detection model. The augmentation apparatus, device, medium and program product for magnetic code data in bills provided by the embodiments of the present disclosure can be arranged in the server 103. It should be noted that, Figure 1 The system architecture shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0051] The following will be based on Figure 1 The scenario described, by Figures 2-9 The augmentation method for magnetic code data in bills of the embodiments of the present disclosure will be described in detail. It should be noted that the serial numbers of the various operations in the following method are only used to represent the operations for description, and should not be regarded as representing the execution sequence of the various operations. Unless explicitly stated, the method does not need to be executed in the order shown.

[0052] Figure 2 The flowchart of the augmentation method for magnetic code data in bills according to an embodiment of the present disclosure is schematically shown.

[0053] As Figure 2 shown, according to the embodiment, the augmentation method for magnetic code data in bills can include operations S201-S206.

[0054] First, in operation S201, an original sample set is obtained, which includes at least one original bill image and original position annotation information of a magnetic code region in the original bill image. In an embodiment, the shape of the magnetic code region can be a rectangular magnetic code box, for example, and the position annotation information of the magnetic code region can be the coordinates of the four corners of the magnetic code box. The magnetic code box can be horizontal or inclined at any angle, which is not limited by the present disclosure.

[0055] Then, in operation S202, based on the original position annotation information, the image content in the magnetic code region is extracted from the original bill image to generate a magnetic code strip image. As shown below Figure 5 the magnetic code strip image 501.

[0056] And in operation S203, the image content in the magnetic code region in the original bill image is replaced with background pixels to generate a background check image. As shown below Figure 6 the background check image 601.

[0057] In operation S203, the background pixels used to fill the magnetic code region are the pixels in the background of the magnetic code region of the original check image. For example, the coordinates of the four corners of the magnetic code frame of each original check image in the original sample set can be read, and then the image content on or around the position of the frame of the magnetic code frame can be sampled to obtain the background pixels.

[0058] Next, in operation S204, the magnetic code bar image is converted into a first to-be-fused image with a transparent background. The foreground of the first to-be-fused image can be black or other colors (for example, sienna) used when printing the magnetic code characters. As shown below, the magnetic code bar image 501 can be processed into a first to-be-fused image 803. The purpose of such processing is to remove the image background carried by the magnetic code bar image 501 itself and only keep the magnetic code character image in the foreground. Figure 8

[0059] In one embodiment, when the magnetic code bar image is converted into the first to-be-fused image, a clustering algorithm can be used to extract the pixels where the foreground characters in the magnetic code bar image are located, and then an intermediate transition image is generated based on the extracted pixels where the foreground characters are located, and the intermediate transition image is processed to generate the first to-be-fused image. By using the clustering algorithm to sample the pixels where the foreground characters are located, the pixels constituting the magnetic code characters can be accurately sampled, the background pixels can be reduced as much as possible, the sampling noise of the first to-be-fused image can be reduced, and the sampling accuracy of the magnetic code characters in the first to-be-fused image can be improved.

[0060] And in operation S205, the image content of the first to-be-fused image is fused into the background check image to obtain a first augmented image (as shown below, the first augmented image 901). Figure 9

[0061] In one embodiment, since the first to-be-fused image is transparent except for the pixels corresponding to the foreground magnetic code characters, the first to-be-fused image can be superimposed into the background check image according to the insertion position set by the user.

[0062] In another embodiment, a second to-be-fused image containing the image content of the first to-be-fused image can be generated first, where the size of the second to-be-fused image is consistent with the size of the background check image, and the background of the second to-be-fused image is transparent, as shown below. Figure 8 or Figure 9 ​​The second image to be fused 805 is then superimposed with the background check image and the second image to be fused to obtain the first augmented image. The position of the magnetic code area in the first augmented image is determined by the position setting of the first image to be fused (or the pixels corresponding to the magnetic code characters in the foreground) when generating the second image to be fused. By superimposing the second image to be fused with the background check image at the same size, the position setting of the magnetic code characters and the superposition with the background check image can be separated into two steps, making the operation more concise and clear, and more accurate in obtaining the position information of the magnetic code area in the first augmented image.

[0063] In this way, in the generated first augmented image (such as the first augmented image 901), the background around the magnetic code characters is the background pixels in the background check image, which can achieve seamless fusion of the magnetic code character pixels and the background pixels in the background check image, avoiding the problem of traces left when pasting fragmented images onto the background image. It also avoids the problem of fusion noise caused by the magnetic code characters carrying the background pixels in the magnetic code strip image in which they are located, resulting in the mixing of other pixels between the magnetic code characters and the background in the background check image. This makes the generated bill image closer to the original bill image obtained by scanning the paper bill.

[0064] Next, in operation S206, the first augmented image is added to the original sample set, thereby increasing the image data in the original sample set used for training the character detection model.

[0065] In one embodiment, the generated magnetic code stripe images can be aggregated to form a magnetic code stripe image set, and the generated background check images can be aggregated to form a background check image set. Then, before operations S204 and S205, magnetic code stripe images and background check images are randomly extracted from the magnetic code stripe image set and the background check image set, respectively. The magnetic code stripe image conversion in operation S204 and the image fusion in operation S205 are then performed. In this way, the magnetic code stripe image and background check image used to generate the first augmented image may be derived from different original check images, thereby generating new, previously non-existent check images as training data, increasing the diversity of the model training dataset.

[0066] Because the character detection model must also be trained to identify the magnetic code region in a bill image, the location information of the magnetic code region must be annotated. Therefore, in one embodiment, after obtaining the first augmented image, the fusion position of the image content of the first image to be fused in the first augmented image can be obtained based on the generation process of the first augmented image, thereby obtaining the first location annotation information of the magnetic code region in the first augmented image. The original sample set is then expanded using the first augmented image and the first location annotation information. In this way, the magnetic code region of the first augmented image can be automatically annotated, quickly obtaining model training data.

[0067] Figure 3 A flowchart of an augmentation method for magnetic code data in a bill according to another embodiment of the disclosure is schematically shown.

[0068] As shown, according to this embodiment, after operations S201-S203, the augmentation method for magnetic code data in a bill can generate an augmented image by two different methods, including generating a first augmented image by an image fusion method in operations S204-S205, and generating a second augmented image by a random generation method in operations S324-S325. Figure 3

[0069] Wherein, operations S201-S203 and operations S204-S205 can refer to the foregoing description, and will not be repeated here.

[0070] In operation S324, a magnetic code string is generated based on characters randomly read from a character library and character layout settings, wherein the character library stores a set of characters used to generate the magnetic code.

[0071] In operation S325, the magnetic code string is inserted into the background check image according to a randomly generated offset value that satisfies a predetermined condition, to generate a second augmented image (as shown in Figure 7 Wherein, the offset value includes up, down, left and right offset values when the magnetic code string is inserted into the background check image; and the predetermined condition is a value range of the up, down, left and right offset values. For example, the predetermined condition is set so that the magnetic code string is inserted in the lower half of the background check image, so that the second augmented image is closer to the real bill image.

[0072] The second augmented image (as shown in the second augmented image 701) is shown in Figure 7 Wherein, the magnetic code is inserted into the background check image in the form of characters, and the background of the magnetic code region in the background check image comes from the real scanned bill image (i.e., the original bill image), so that the second augmented image is closer to the original bill image scanned from the paper bill, and can effectively increase the data required for training the character detection model.

[0073] Further, after obtaining the first augmented image and the second augmented image, the annotation position information of the magnetic code region in the first augmented image and the second augmented image can also be automatically obtained according to the respective generation processes.

[0074] Specifically, in one embodiment, after generating the first augmented image in operation 205, first position annotation information of the magnetic code region in the first augmented image can also be obtained in operation S316 based on the fusion position of the image content of the first to-be-fused image in the first augmented image.

[0075] ​Correspondingly, after the second augmented image is generated in operation S325, the second position annotation information of the magnetic code region in the second augmented image can also be determined in operation S326 based on the offset value when the magnetic code string is inserted in the background check image when the second augmented image is generated.

[0076] Then the original sample set can be expanded with the first augmented image and the first position annotation information in operation S307, and the original sample set can be expanded with the second augmented image and the second position annotation information thereof.

[0077] The embodiments of the present disclosure can perform augmentation processing on the magnetic code data in the check image in two ways, the generated image is closer to the real sample, and the training effect of the model is improved. Moreover, the embodiments of the present disclosure can also automatically annotate the position information of the magnetic code region in the generated image, and improve the generation rate of the model training sample data.

[0078] Figure 4 The model generalization method flowchart for expanding the check sample by applying the augmentation method for the magnetic code data in the check according to an embodiment of the present disclosure is schematically shown.

[0079] As shown in Figure 4 , the model generalization method for expanding the check sample by applying the augmentation method for the magnetic code data in the check according to the embodiments of the present disclosure can include five steps S1-S5, which are: S1 background and feature separation, S2 data augmentation, S3 model training, S4 model testing and S5 model accuracy analysis and output.

[0080] S1 background and feature separation can be divided into two steps of feature extraction and background separation, and the two steps can be performed in parallel.

[0081] (1) Feature extraction step: input the original sample set, and cut and copy the image content in the magnetic code region of the original check image in the original sample set to make a magnetic code strip image, such as Figure 5 the magnetic code strip image 501 in

[0082] The original sample set includes the original check image and the original position annotation information of the magnetic code region in the original check image. The original check image is an electronic image scanned from a real check. The original position annotation information can be represented by the coordinates of the four corners of the magnetic code box, and can be formed by manual annotation on the original check image.

[0083] The coordinates of the four corners of the magnetic code box of each original sample image in the original sample set can be read, and the image content in the coordinates of the four corners of the magnetic code box can be cut and copied to generate a magnetic code strip image. All original check images in the original sample set can be automatically processed by the above feature extraction process in turn, and finally a magnetic code strip image set is generated.

[0084] (2) Background separation step: duplicate the original sample set, sample the image content in the background of the magnetic code area in the original bill image, obtain the background pixels, and then fill the magnetic code area in the original bill image with the background pixels to make a background check image without magnetic code, such as the background check image 601 shown in Figure 6

[0085] Specifically, for example, the coordinates of the four corners of the magnetic code frame of each original bill image in the original sample set can be read, and then the image content on or around the magnetic code frame position can be sampled, and then the sampled content (i.e. background pixels) is used to fill the magnetic code area in the original bill image to generate a background check image without magnetic code. All original bill images of the duplicated original sample set are sequentially and automatically subjected to the above background separation processing, and finally a background check image set is generated.

[0086] S2 data augmentation can be divided into two ways: data augmentation according to a random generation method and data augmentation according to an image fusion method, wherein the two ways can be performed in parallel.

[0087] The implementation process of data augmentation according to the random generation method is as follows.

[0088] Figure 7 An illustrative diagram of generating a second augmented image according to the random generation method in an embodiment of the present disclosure is shown.

[0089] As shown in Figure 7 , first input the background check image set, obtain a background check image (e.g. background check image 601) from the background check image set, then adjust various parameters such as the number of characters and fonts extracted from the magnetic code library to generate a magnetic code bar string, then insert the magnetic code bar string into the lower half of the background check image 601, and determine the position range of the magnetic code bar string inserted into the background check image 601 by setting the up-down and left-right offset range, to obtain a second augmented image 701. The position of the inserted magnetic code string in the second augmented image 701 can be labeled to obtain the second position labeling information of the magnetic code area in the second augmented image.

[0090] ​Wherein, before generating the second augmented image, the following information needs to be configured first. (a) The background of the picture: read the background check image set, set the random reading strategy, and select the background check image as the background of the second augmented image to be generated. (b) Configure the character library: according to the characteristics of the magnetic code, add the numbers 0~9, capital letters A~D and spaces to the character library. (c) Font: according to the characteristics of the magnetic code, select the font of the magnetic code. (d) Generation parameters: including the number of generated second augmented images, the length of the generated magnetic code string, font, size and other parameters. The number of generated second augmented images can be adjusted according to the number of images in the training set, validation set and test set in the data set, and other parameters can be adjusted according to the actual situation. (e) Background cutting range, refers to the position range in the background check image for filling the magnetic code string. The background cutting range can be adjusted to most of the background in the background check image, and the offset values of up, down, left and right can be randomly changed to make the insertion position of the magnetic code string random. In order to facilitate the generalization ability of machine learning, the change degree of up, down, left and right offset in the offset value needs to be limited within a certain range according to the position characteristics of the magnetic code in the check, to ensure that the magnetic code string can be randomly inserted into the lower half of the background check image.

[0091] After the parameters are configured, the characters are randomly extracted from the character library and the background check images are randomly extracted from the background check image set according to the set parameters, and the second augmented image (such as second augmented image 701) is generated according to the random generation method. According to the offset value when the magnetic code string is inserted into the second augmented image, the position of the magnetic code region in the second augmented image is labeled. Then according to the needs of the generated data set, the generated second augmented image is divided into training set, validation set and test set in turn.

[0092] The implementation process of data augmentation according to the image fusion method is as follows. The combination of Figure 8 and Figure 9 is explained as follows.

[0093] The magnetic code bar image set and the background check image set are input, the magnetic code bar image (for example, magnetic code bar image 501) is extracted from the magnetic code bar image set, the magnetic code characters in the foreground are extracted, and the transparent background is processed to obtain the first to-be-fused image (for example, first to-be-fused image 803). Then, based on the first to-be-fused image with transparent background and the background check image, the first augmented image is automatically generated by fusion merging. Then the position of the magnetic code region in the first augmented image can be labeled to obtain the first position labeling information, which is expanded to the original sample set.

[0094] Figure 8The flowchart of generating the image to be fused in an embodiment of the present disclosure is shown schematically. The generation of the image to be fused by the magnetic code strip image 501 is taken as an example to illustrate the process in the image fusion method.

[0095] As shown in Figure 8 , first, the k-means clustering algorithm is used to extract the pixels where the foreground characters are located in the magnetic code strip image 501, to obtain the intermediate transition image 802 of the black background transparent characters. Then the intermediate transition image 802 of the black background transparent characters can be converted into a gray-scale image, and the maximum gray-scale value in the converted gray-scale image is subtracted from the original image before conversion, to obtain the first image to be fused 803 with the foreground being black and the background being transparent after inversion. Then the size of the first image to be fused 803 is adjusted according to actual needs, to obtain the image 804, and then the image 804 is filled with a transparent background to the size of the background check image, to generate the second image to be fused 805.

[0096] Figure 9 The schematic diagram of generating the first augmented image according to the image fusion method in an embodiment of the present disclosure is shown schematically.

[0097] Next, as shown in Figure 9 , the background check image (for example, the background check image 601) and the second image to be fused 805 are fused to generate an image randomly fused by the real magnetic code strip and the real check background, that is, the first augmented image 901.

[0098] In Figure 8 , when the second image to be fused 805 is generated, the image 804 can be limited to the lower half of the image according to the characteristics of the magnetic code by using the up-down-left-right offset parameters. By randomly adjusting the up-down-left-right offset parameters, the generation position of the magnetic code can be made random. According to the up-down-left-right offset parameters, each four-corner coordinate of the position information for identifying the magnetic code region in the first augmented image can be deduced and recorded for labeling.

[0099] According to the number of images in the data set used by the training model, and / or the number of images in the training set, validation set and test set in the data set, the number of generated first augmented images is adjusted and the generated first augmented images are divided into data sets.

[0100] S3 model training: after the data set expansion is completed, all sample data are divided into a training set, a validation set and a test set.

[0101] First, the multiple machine learning algorithm models are trained using the training data set, so that multiple models can be trained. Then the validation set data is predicted using each trained model, the accuracy of each model is recorded, and finally the model with the best training effect is selected as the optimal model for testing.

[0102] The model is trained using the training set, which generates multiple models. After training, the models are verified using the validation set. After inputting the validation set, the model outputs the prediction results and accuracy. Based on the verification results, the model with the highest accuracy is selected as the optimal model with the best training effect for testing. The trained model can be adjusted according to actual needs. In one embodiment, three data sets were used, and the Faster R-CNN and Swin Transformer models were selected for training. The model with the best training effect of the two models was output.

[0103] S4 Model Testing: Use the test dataset to predict the model and obtain the model's prediction results. The prediction results include the location information of the magnetic code area in the sample image recognized by the model and the model's confidence in the recognition result.

[0104] The model makes a prediction for each image in the test set and outputs a label and confidence score.

[0105] In one embodiment, three datasets were used to test the trained Faster R-CNN and Swin Transformer models. Specifically, three datasets were designed based on practical needs, as shown in Table 1. Datasets 1 and 2 were generated using a random generation method, using black and white document backgrounds and check backgrounds, respectively. The training, validation, and test sets consisted of 6,000, 2,000, and 2,000 images, respectively. Dataset 3 was generated using an image fusion method, with the training, validation, and test sets consisting of 6,000, 2,000, and 2,000 images, respectively.

[0106] Table 1

[0107]

[0108] S5 Model Accuracy Analysis and Output: Obtain the model's test results, calculate the model's average accuracy, and compare it with the preset value for judgment. If the accuracy meets the preset requirement, the model is output; if the accuracy is less than the preset requirement, the model is replaced or the training parameters are adjusted and the model is retrained until the accuracy reaches the required level.

[0109] The confidence of the model on each test picture is automatically obtained, and the average value is taken as the detection accuracy. The accuracy is compared with the preset value to determine whether the model is output if the accuracy exceeds the preset value, or the model is reselected or the training parameters are reset, and the model is retrained until the model accuracy can reach the preset value. The preset value is adjusted according to actual needs. Table 2 is the test results of two models on three data sets in this example. From the comparison results, the accuracy of Faster R-CNN on data set 3 is higher, so the trained Faster R-CNN model is selected as the final model output. After post-processing (increasing the threshold value and specifying the detection frame position) on the model positioning result, the positioning accuracy of the test set can reach 100%, achieving good training effect.

[0110] Table 2

[0111]

[0112] The two ways of augmenting the magnetic code data in the bill image in the embodiments of the present disclosure, compared with the existing methods, the generation step is more simplified, the generated image is closer to the real sample, and the training effect of the detection model is better. Through model accuracy analysis, it can be ensured that the output model has higher accuracy.

[0113] Figure 10 A block diagram of an augmentation apparatus 1000 for magnetic code data in bills according to an embodiment of the present disclosure is schematically shown.

[0114] As Figure 10 shown, according to this embodiment, the augmentation apparatus 1000 for magnetic code data in bills can include an acquisition module 1010, a feature extraction module 1020, a background separation module 1030, and a data augmentation module 1040. The apparatus 1000 can perform the augmentation method for magnetic code data in bills described with reference to Figures 2-9 .

[0115] Specifically, the acquisition module 1010 is configured to acquire an original sample set, the original sample set including at least one original bill image and original position annotation information of a magnetic code region in the original bill image. In one embodiment, the acquisition module 1010 can perform the operation S201 introduced in the foregoing.

[0116] The feature extraction module 1020 is configured to extract image content in the magnetic code region from the original bill image based on the original position annotation information, and generate a magnetic code bar image. In one embodiment, the feature extraction module 1020 can perform the operation S202 introduced in the foregoing.

[0117] The background separation module 1030 is configured to replace image content in the magnetic code area in the original check image with background pixels to generate a background check image, where the background pixels are pixels in the background in the magnetic code area of the original check image. In an embodiment, the background separation module 1030 can perform the operation S203 described above.

[0118] The data augmentation module 1040 is configured to collect the generated magnetic code strip images to obtain a magnetic code strip image set, and collect the generated background check images to obtain a background check image set, and then can construct a new check image based on the magnetic code strip images in the magnetic code strip image set and / or the background check images in the background check image set.

[0119] Specifically, the data augmentation module 1040 can include an image fusion module 1041 and a random generation module 1042.

[0120] The image fusion module 1041 can be configured to convert the magnetic code strip image into a first to-be-fused image with a transparent background, fuse image content of the first to-be-fused image into the background check image to obtain a first augmented image, and augment the original sample set with the first augmented image. In an embodiment, the image fusion module 1041 can perform the operation S204, the operation S205, and the operation S206 described above.

[0121] In some embodiments, the image fusion module 1041 is further configured to obtain first position labeling information of the magnetic code area in the first augmented image based on a fusion position of the image content of the first to-be-fused image in the first augmented image, and then augment the original sample set with the first augmented image and the first position labeling information. In an embodiment, the image fusion module 1041 can perform the operation S316 described above.

[0122] The random generation module 1042 is configured to generate a magnetic code strip character string based on characters randomly read from a character library and character layout settings, where the character library stores a set of characters used to generate the magnetic code, insert the magnetic code strip character string into the background check image according to at least one offset value randomly generated and satisfying a predetermined condition to generate at least one second augmented image, where the offset value includes up, down, left, and right offset values when the magnetic code strip character string is inserted into the background check image, the predetermined condition is a value range of the up, down, left, and right offset values, and augment the original sample set with the at least one second augmented image. In an embodiment, the random generation module 1042 can perform the operation S324 and the operation S325 described above.

[0123] In some embodiments, the random generation module 1042 is further configured to: determine, based on the offset value when each second augmented image is generated, second position annotation information of the magnetic code region in each second augmented image; and expand the original sample set with each second augmented image and its corresponding second position annotation information. In some embodiments, the random generation module 1042 can perform the operation S326 as previously described.

[0124] According to embodiments of the present disclosure, any of the modules of the acquisition module 1010, the feature extraction module 1020, the background separation module 1030, the data augmentation module 1040, the image fusion module 1041 and the random generation module 1042 can be combined in one module, or any of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of other modules, and implemented in one module. According to embodiments of the present disclosure, at least one of the acquisition module 1010, the feature extraction module 1020, the background separation module 1030, the data augmentation module 1040, the image fusion module 1041 and the random generation module 1042 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or any one of software, hardware and firmware or any appropriate combination of several of them. Alternatively, at least one of the acquisition module 1010, the feature extraction module 1020, the background separation module 1030, the data augmentation module 1040, the image fusion module 1041 and the random generation module 1042 can be at least partially implemented as a computer program module which, when executed, can perform the corresponding function.

[0125] Figure 11 A block diagram of an electronic device suitable for implementing the method for augmenting magnetic code data in a bill according to embodiments of the present disclosure is schematically shown.

[0126] As Figure 11As shown, the electronic device 1100 according to embodiments of the present disclosure includes a processor 1101 that can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 1102 or loaded from a storage section 1108 into a random access memory (RAM) 1103. The processor 1101 can include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chip set, and / or a special purpose microprocessor (e.g., an application specific integrated circuit (ASIC)), and so on. The processor 1101 can also include on-board memory for cache purposes. The processor 1101 can include a single processing unit or multiple processing units to perform the various actions of the method processes according to embodiments of the present disclosure.

[0127] In the RAM 1103, various programs and data required for the operation of the electronic device 1100 are stored. The processor 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. The processor 1101 performs various operations of the method processes according to embodiments of the present disclosure by executing the programs in the ROM 1102 and / or the RAM 1103. Note that the programs can also be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 can also perform various operations of the method processes according to embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0128] According to embodiments of the present disclosure, the electronic device 1100 can also include an input / output (I / O) interface 1105 that is also connected to the bus 1104. The electronic device 1100 can also include one or more of the following components connected to the I / O interface 1105: an input section 1106 including a keyboard, a mouse, etc.; an output section 1107 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1108 including a hard disk, etc.; and a communication section 1109 including a network interface card such as a LAN card, a modem, etc. The communication section 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as necessary. A removable medium 1111 such as a magnetic disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1110 as necessary, so that a computer program read out from the removable medium 1111 is installed into the storage section 1108 as necessary.

[0129] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present disclosure.

[0130] According to an embodiment of the present disclosure, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more of the ROM 1102 and / or the RAM 1103 described above, and / or one or more memories other than the ROM 1102 and the RAM 1103.

[0131] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the methods provided by the embodiments of the present disclosure.

[0132] The above functions defined in the system / apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1101. According to an embodiment of the present disclosure, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0133] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal via a network medium, and be downloaded and installed through the communication part 1109 and / or installed from the detachable medium 1111. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0134] In such embodiments, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the detachable medium 1111. When the computer program is executed by the processor 1101, the above-described functions defined in the system of the embodiments of the present disclosure are executed. According to the embodiments of the present disclosure, the system, device, apparatus, module, unit, and the like described above can be implemented by computer program modules.

[0135] According to the embodiments of the present disclosure, the program code for executing the computer program provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming language, and / or assembly / machine language. The programming language includes, but is not limited to, such as Java, C++, python, "C" language or similar programming language. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case involving a remote computing device, the remote computing device can be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, connected to the Internet through an Internet service provider).

[0136] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a segment, or a portion of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the figures. For example, two blocks noted in succession can actually be executed substantially concurrently, or they can sometimes be executed in reverse order, depending on the functionality involved. It should also be noted that each block in the flowcharts or block diagrams, and combinations of blocks in the flowcharts or block diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0137] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations and / or integrations, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations and / or integrations without departing from the spirit and teachings of the present disclosure. All such combinations and / or integrations fall within the scope of the present disclosure.

[0138] The above described embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be advantageously used in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and all such substitutions and modifications shall fall within the scope of the present disclosure.

Claims

1. A method for augmenting magnetic code data in a bill, comprising: obtaining an original sample set comprising at least one original bill image and original position annotation information of a magnetic code region in the original bill image; extracting image content in the magnetic code region from the original bill image based on the original position annotation information to generate a magnetic code bar image; replacing the image content in the magnetic code region in the original bill image with background pixels to generate a background check image, wherein the background pixels are pixels in the background of the magnetic code region in the original bill image; converting the magnetic code bar image into a first to-be-fused image with a transparent background; fusing the image content of the first to-be-fused image into the background check image to obtain a first augmented image; obtaining first position annotation information of the magnetic code region in the first augmented image based on the fusion position of the image content of the first to-be-fused image in the first augmented image; and augmenting the first augmented image and the first position annotation information to the original sample set.

2. The method of claim 1, wherein, The method further comprises: generating a magnetic code bar string based on characters randomly read from a character library and character layout settings, wherein the character library stores a set of characters used to generate a magnetic code; inserting the magnetic code bar string into the background check image according to a randomly generated offset value satisfying a predetermined condition to generate a second augmented image, wherein the offset value comprises up, down, left and right offset values when the magnetic code bar string is inserted into the background check image; the predetermined condition is a value range of the up, down, left and right offset values; and augmenting the second augmented image to the original sample set. 3.The method of claim 2, wherein the method further comprises: determining second position annotation information of the magnetic code region in the second augmented image based on the offset value; and the augmenting the second augmented image to the original sample set comprises: augmenting the original sample set with the second augmented image and the second position annotation information.

4. The method of claim 1, wherein, The converting the magnetic code bar image into a first to-be-fused image with a transparent background comprises: extracting pixels where foreground characters in the magnetic code bar image are located using a clustering algorithm to generate an intermediate transition image; and processing the intermediate transition image to generate the first to-be-fused image.

5. The method of claim 1, wherein, The fusing the image content of the first to-be-fused image into the background check image to obtain a first augmented image comprises: generating a second to-be-fused image containing the image content of the first to-be-fused image, wherein the size of the second to-be-fused image is consistent with the size of the background check image, and the background of the second to-be-fused image is transparent; and superimposing the second to-be-fused image on the background check image to obtain the first augmented image.

6. The method of claim 5, wherein, The obtaining first position annotation information of the magnetic code region in the first augmented image comprises: obtaining the first position annotation information based on the position setting of the image content of the first to-be-fused image in the second to-be-fused image when the second to-be-fused image is generated.

7. The method according to any one of claims 1 to 6, wherein, The method further comprises: generating a magnetic code bar image set by aggregating the generated magnetic code bar images; aggregate the generated background check image to generate a background check image set; and before converting the magnetic code strip image into the first to-be-fused image and fusing image content of the first to-be-fused image into the background check image, randomly selecting the magnetic code strip image and the background check image from the magnetic code strip image set and the background check image set respectively.

8. The method of claim 7, wherein, The method further includes: training a character detection model by using the augmented sample set obtained after augmenting the original sample set, so that the character detection model can recognize a magnetic code region in an arbitrary check image and output position labeling information of the magnetic code region in the arbitrary check image; wherein based on the position labeling information, the magnetic code region can be located and segmented from the arbitrary check image when the arbitrary check image is image segmented. 9.An augmentation device for magnetic code data in a check, comprising: an acquisition module configured to acquire an original sample set, the original sample set including at least one original check image and original position labeling information of a magnetic code region in the original check image; a feature extraction module configured to extract image content in the magnetic code region from the original check image based on the original position labeling information, and generate a magnetic code strip image; a background separation module configured to replace the image content in the magnetic code region in the original check image with background pixels to generate a background check image; wherein the background pixels are pixels in a background of the magnetic code region in the original check image; an image fusion module configured to: convert the magnetic code strip image into a first to-be-fused image with a transparent background; fuse image content of the first to-be-fused image into the background check image to obtain a first augmented image; obtain first position labeling information of the magnetic code region in the first augmented image based on a fusion position of the image content of the first to-be-fused image in the first augmented image; and augment the first augmented image and the first position labeling information to the original sample set. 10.An electronic device, comprising: one or more processors; memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method of any one of claims 1-8. 11.A computer-readable storage medium having computer program instructions stored thereon, the computer program instructions being executed by a processor to implement the method of any one of claims 1-8. 12.A computer program product comprising computer program instructions, the computer program instructions being executed by a processor to implement the method of any one of claims 1-8.

Images