Training sample generation method, electronic device and computer storage medium

By generating image masks and matching them with real scanned images, the method addresses high costs and deviation issues in scanning software training, improving model performance and efficiency.

CN117422945BActive Publication Date: 2024-12-06UCWEB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311355152.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-18
Publication Date
2024-12-06
Estimated Expiration
2043-10-18

AI Technical Summary

Technical Problem

Existing scanning software requires high implementation costs and time due to the complexity of professional rendering software, and the simulated training samples deviate significantly from real data distributions, leading to poor performance on actual documents.

Method used

Generate an image mask with multiple flag bits for a scanned document image, perform image matching with a real second scanned document image obtained by a physical device, and combine them into training samples to train a machine learning model for software scanning.

Benefits of technology

Reduces implementation costs and time by using real images, improves the training effect of the machine learning model to better handle real data distributions, enhancing its performance on actual documents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117422945B_ABST
    Figure CN117422945B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a training sample generation method, an electronic device, and a computer storage medium. The training sample generation method includes: obtaining a first scanned document image after filtering; generating an image mask with multiple flags for the first scanned document image, wherein different flags carry different coding information; obtaining a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device; performing image matching on the image mask and the second scanned document image based on the multiple flags, and generating a training sample pair based on the matched image mask and the second scanned document image. The training sample pairs generated by the embodiments of the present application can be closer to the real sample distribution in terms of data distribution, thereby improving the training effect of the machine learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a training sample generation method, an electronic device, and a computer storage medium. Background Art

[0002] With the development of computer technology, people often need to electronically scan documents in their daily lives and work, such as electronic books, invoice reimbursement, scanning and printing of work documents, scanning of certificate materials, and other application scenarios.

[0003] Since professional scanners are expensive, scanning software has emerged. Through scanning software, users can scan documents conveniently and at low cost anytime and anywhere. With the widespread use of scanning software, scanning scenarios are becoming more and more diverse, which requires scanning software to be able to continuously iterate and update to adapt to new scenario requirements. However, the iterative update of scanning software requires a large number of training samples in various scanning scenarios as a prerequisite. To this end, in an existing method, professional rendering software (such as unity3D, etc.) is used to build a virtual lighting environment, which can include different light sources, different occluders, different texture backgrounds, etc., and the scene of the user scanning the document is simulated by changing the position, strength, color temperature, etc. of the light source and then matching different occluders and paper textures, so that the generated scanned image is more realistic. However, in this method, first, the threshold for using such professional rendering software is high, and it takes a long time to debug and learn to achieve a certain simulation effect, and the time cost is also high, resulting in a high overall implementation cost; second, the scanned images simulated and generated by such professional rendering software have a relatively large deviation in data distribution from the real sample distribution, resulting in that although the machine learning model for implementing software scanning works well on the simulated training samples, it works poorly on the real data. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a training sample generation solution to at least partially solve the above-mentioned problem.

[0005] According to a first aspect of an embodiment of the present application, a training sample generation method is provided, comprising: obtaining a first scanned document image after filtering; generating an image mask having multiple flag bits for the first scanned document image, wherein different flag bits carry different encoding information; obtaining a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device; performing image matching on the image mask and the second scanned document image based on the multiple flag bits, and generating a training sample pair based on the matched image mask and the second scanned document image.

[0006] According to the second aspect of an embodiment of the present application, there is provided an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect.

[0007] According to a third aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in the first aspect is implemented.

[0008] According to the scheme provided by the embodiment of the present application, an image mask is generated based on the first scanned document image, and then the image mask is imaged by a physical image acquisition device to obtain a real second scanned document image corresponding to the image mask, and then the image mask and the second scanned document image are matched based on the flag, and the two are combined into a training sample pair, so as to train the machine learning model for implementing software scanning in the future. On the one hand, because the second scanned document image is an image collected by a physical image acquisition device, it does not need to be generated by adjusting parameters, and it is a real image itself. Therefore, it does not require long-term debugging and learning by the corresponding personnel, the time cost is low, and the overall implementation cost of training sample generation is also low. On the other hand, the first scanned document image after the filter processing is an image with a relatively clean background, and the second scanned document image is a real image taken based on the image mask generated by the first scanned document image, and its background is a real background, which may contain more interference and noise data. The image mask and the second scanned document image are combined as a training sample pair to train the machine learning model for implementing software scanning, which can be closer to the real sample distribution in data distribution, improve the training effect of the machine learning model, and the subsequent performance on real data. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0010] Figure 1 A schematic diagram of an exemplary system applicable to the embodiment scheme of the present application.

[0011] Figure 2 The present invention is a flowchart of a method for generating training samples according to an embodiment of the present application.

[0012] Figure 3This is an optional sub-step flowchart of “filter processing for removing background from the first scanned document image” of the present application.

[0013] Figure 4 This is an optional sub-step flowchart of step S104 of the present application.

[0014] Figure 5 This is an optional sub-step flowchart of step S108 of the present application.

[0015] Figure 6 This is an optional sub-step flowchart of sub-step S1081 of the present application.

[0016] Figure 7 This is an optional sub-step flowchart of sub-step S1081C of the present application.

[0017] Figure 8 A schematic diagram of an example scenario in an embodiment of the present application.

[0018] Fig. 9 It is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0020] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0021] Figure 1 An exemplary system applicable to the embodiment of the present application is shown. Figure 1 As shown, the system 100 may include a cloud service end 102, a communication network 104 and / or one or more user devices 106. Figure 1 It should be noted that the solution of the embodiment of the present application can be independently completed by the cloud service end 102, or can also be independently completed by the user device 106 with higher software and hardware performance.

[0022] The cloud service end 102 may be any suitable device for storing information, data, programs and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud service end clusters, etc. In some embodiments, the cloud service end 102 may perform any suitable function. For example, when the cloud service end 102 independently completes the solution of the embodiment of the present application, in some embodiments, the cloud service end 102 may be used to generate a training sample pair for training a machine learning model for implementing software scanning. As an optional example, in some embodiments, the cloud service end 102 may first obtain a first scanned document image after filter processing; then, an image mask with multiple flags may be generated for the first scanned document image, wherein different flags carry different encoding information; then a second scanned document image obtained by image acquisition of the image mask by a physical image acquisition device; then, based on the multiple flags, image matching is performed on the image mask and the second scanned document image, and a training sample pair is generated based on the matched image mask and the second scanned document image. In some embodiments, after generating the training sample pair, the cloud service end 102 may use the training sample pair to train the machine learning model for implementing software scanning. After the training is completed, the cloud service end 102 can receive the scanned image with unclean background sent by the user device 106, and process it through the machine learning model to generate a scanned image with clean background, and then return it to the user device 106.

[0023] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN) and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0024] The user device 106 may include any one or more user devices suitable for presenting images, interacting with users, etc. When the solution of the embodiment of the present application is independently completed by the user device 106, the user device 106 can be used to generate a training sample pair for training a machine learning model for implementing software scanning. As an optional example, in some embodiments, the user device 106 may first obtain a first scanned document image after filter processing; then, an image mask with multiple flags may be generated for the first scanned document image, wherein different flags carry different encoding information; then a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device; then, based on the multiple flags, the image mask and the second scanned document image are image matched, and a training sample pair is generated based on the matched image mask and the second scanned document image. In some optional embodiments, the user device 106 may send the generated training sample pair to the cloud service end 102 to train the machine learning model for generating and implementing software scanning in the cloud service end 102. In some embodiments, the user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, and / or any other suitable type of user device.

[0025] Based on the above system, an embodiment of the present application provides a training sample generation solution, which is described below through multiple embodiments.

[0026] Figure 2 A flow chart of the steps of a training sample generation method according to an embodiment of the present application. According to a first aspect of the present application, a training sample generation method is provided, referring to Figure 2 As shown, the method includes steps S102, S104, S106 and S108, specifically:

[0027] S102: Acquire a first scanned document image after filter processing.

[0028] In the present application, the first scanned document image needs to be subjected to filter processing first to obtain an image with a relatively clean background.

[0029] Among them, the first scanned document image can be a document image obtained by any appropriate method (such as manual shooting, image generation or synthesis software or scanning software after initial training, etc.) (the document can include but is not limited to books, invoices, contracts, business licenses, work documents, certificate materials, identity cards, driver's licenses, etc.). For example, taking the document as an example of an invoice, the first scanned document image can be obtained in the following manner: the user places the invoice to be scanned on a support (such as a desktop, etc.) or holds it in the hand, and then uses the camera on the mobile terminal to shoot the invoice from the direction of the invoice, thereby obtaining the corresponding first scanned document image. Of course, this is just an example and does not serve as any limitation to this application. It should be understood that in this case, the first scanned document image can be obtained by shooting when necessary and then directly obtained, or it can be obtained by shooting in advance, and then the first scanned document image is stored in a predetermined storage space (including but not limited to storage media such as disks, hard disks, memories, or databases, etc.), and directly obtained from the storage space when needed, and there is no limitation in this application.

[0030] Alternatively, the first scanned document image may be obtained through a pre-generated electronic file, for example, it may be obtained by printing or scanning software. The electronic file may be in PDF format, or it may be in another file format. The electronic file may include one or more pages, or the first scanned document image may be generated based on these pages. Similarly, the first scanned document image generated in this way may also be first stored in a predetermined storage space (including but not limited to storage media such as disks, hard disks, memories, or databases, etc.), and directly obtained from the storage space when needed, and there is no restriction on this in this application.

[0031] After obtaining the first scanned image, the first scanned image may be subjected to filter processing. The present application does not limit the specific implementation method of the filter processing on the first scanned image, and any filter processing method may be performed, including but not limited to: machine learning model method, image enhancement method, etc.

[0032] In some optional embodiments, referring to Figure 3 In the flowchart shown, step S102 may include: performing a filter process on the first scanned document image to remove the background, so as to obtain a first scanned document image with a solid background color. In this way, it is convenient to perform data processing using the first scanned document image with a solid background color in the subsequent process, thereby reducing the amount of calculation.

[0033] Optionally, the solid color background may be a white background, which is closer to the actual background of the scanned document, more convenient for subsequent processing, and reduces the amount of calculation.

[0034] In some optional embodiments, “performing a filter process to remove the background of the first scanned document image” includes the following sub-steps S1021 and S1022, specifically:

[0035] S1021: Perform target content detection on the first scanned document image, and determine a corresponding target content area according to the detection result.

[0036] The first scanned document image generally includes one or more types of target content, and the target content may generally be information content with practical significance. For example, the target content may include but is not limited to at least one of text type content, chart type content, and color block pattern type content. In addition, the text type content, chart type content, and color block pattern type content are not specifically limited in this application. For example, the following description may be referred to for schematic understanding. The text may include but is not limited to Chinese characters, Arabic numerals, English text, etc.; the chart may include but is not limited to bar charts, pie charts, line charts, tables, etc.; the color block pattern may include any regular color block pattern or irregular color block pattern.

[0037] In the present application, any suitable algorithm may be used to perform target content detection on the first scanned document image, and after obtaining the detection result, a target detection area corresponding to the detected target content is determined from the first scanned document image based on the target detection result.

[0038] For example, the target content (including but not limited to at least one of text type content, chart type content and color block pattern type content) can be first detected by a target detection algorithm, and then further processed: for example, if text type content is detected, a text recognition algorithm (the text recognition algorithm includes but is not limited to OCR (Optical Character Recognition) algorithm, etc.) can be used to identify the text type content, so as to determine the target content area corresponding to each character according to the text recognition result; if chart type content is detected, an image segmentation algorithm can be used to perform image segmentation on the chart type content to obtain the target content area corresponding to the chart type content; if color block pattern type content is detected, an image segmentation algorithm can be used to perform image segmentation on the color block pattern type content to obtain the target content area corresponding to the color block pattern type content. For example, the image segmentation algorithm includes but is not limited to a semantic segmentation algorithm, an instance segmentation algorithm, etc.

[0039] S1022: Setting pixel values ​​of other areas in the first scanned document image except the target content area to preset pure color pixel values ​​to obtain a first scanned document image with a pure color background color.

[0040] After the target content area is determined, the pixel values ​​of other areas except the target content area can be set to preset solid color pixel values, so that the original first scanned document image is transformed into an image with a solid color background that only includes the target content (including but not limited to at least one of text type content, chart type content and color block pattern type content).

[0041] The preset pure color pixel value can be set as needed. For example, taking the pure color background as a white background, the preset pure color pixel can be set to 255. After setting the pixel values ​​of other areas except the target content area in the first scanned document image to 255, a first scanned document image with a white background can be obtained.

[0042] Based on this, in the present application, through the optional implementation of the above-mentioned sub-steps S1021 to S1022, the first scanned document image can be accurately and effectively subjected to filter processing to remove the background, so as to obtain a first scanned document image with a solid background color, so as to facilitate the subsequent use of the first scanned document image with a solid background color for data processing and reduce the amount of calculation.

[0043] However, the above method is not limited thereto, and other background removal filter methods are also applicable to the solution of the present application.

[0044] In some optional embodiments, after determining the corresponding target content area according to the detection results in sub-step S1021, the training sample generation method also includes: performing mask processing on the target content area to obtain the corresponding target content area mask; step S1022 includes: setting the pixel values ​​of other areas outside the target content mask in the first scanned document image to preset solid color pixel values.

[0045] Based on this, in the present application, by performing mask processing on the target content area to obtain the corresponding target content area mask, and then setting the pixel values ​​of other areas outside the target content mask in the first scanned document image to preset solid color pixel values, the first scanned document image can be more accurately and effectively subjected to filter processing to remove the background, so as to obtain a first scanned document image with a solid background color, so as to facilitate the subsequent use of the first scanned document image with a solid background color for data processing and reduce the amount of calculation.

[0046] The specific method of obtaining the mask (i.e., the target content area mask) can be implemented by those skilled in the art in any appropriate manner, including but not limited to the method of obtaining it through a machine learning model for masking the target object in the image. In addition, optionally, when the machine learning model outputs the mask corresponding to the target content area, it can also output the confidence corresponding to each mask at the same time. Optionally, it is also possible to determine whether to retain the corresponding target content mask based on the confidence level of each target content area mask, thereby achieving the purpose of filtering multiple target content area masks. For example, it can be achieved by determining whether the confidence of the target content area mask is greater than a preset confidence threshold. If so, it is retained, and if not, it is not retained. Based on this, the target content mask with a lower confidence level can be filtered out to ensure the accuracy of the target content mask.

[0047] In some optional embodiments, in the above optional embodiments, after obtaining the corresponding target content area mask, the training sample generation method also includes: performing morphological processing on the target content area mask; step S1022 includes: setting the pixel values ​​of other areas in the first scanned document image, except for the target content mask after morphological processing, to preset pure color pixel values.

[0048] Based on this, in the present application, by performing morphological processing on the target content area mask, and then setting the pixel values ​​of other areas in the first scanned document image, except for the target content mask after morphological processing, to preset pure color pixel values, the first scanned document image can be more accurately and effectively subjected to filter processing to remove the background, so as to obtain a first scanned document image with a pure background color having a better effect, so as to facilitate the subsequent use of the first scanned document image with a pure background color having a better effect for data processing and reduce the amount of calculation.

[0049] Optionally, the target content region mask is subjected to morphological processing, which may include at least one of an erosion process and a dilation process. Different morphological parameters may be used to perform different types of morphological processing according to the needs of different target content region masks, which is not particularly limited in the present application.

[0050] In some optional embodiments, the training sample generation method further includes: determining whether the first scanned document image after the background removal filter processing satisfies a predetermined condition, and using the first scanned document image that satisfies the predetermined condition to perform the subsequent step S104. The predetermined condition is used to determine whether the background of the first scanned document image after the background removal filter processing satisfies the requirement and becomes a basic solid color background, so as to prevent the background of the first scanned document image in the subsequent step S104 from reducing the subsequent processing effect.

[0051] Among them, the preset conditions can be determined as needed and are not limited here. Optionally, the "determining whether the first scanned document image after the filter processing for removing the background meets the predetermined conditions" includes: determining the mean of the first scanned document image after the filter processing for removing the background, and judging whether the mean is greater than a preset mean threshold. If so, the predetermined conditions are met, and if not, the predetermined conditions are not met. For example, taking the solid color background as a white background as an example, the preset mean threshold can be set to 250 (of course, it can also be set to other values ​​close to 255, which is not limited here). If the calculated mean of the first scanned document image after the filter processing for removing the background is greater than 250, it can be considered that the predetermined conditions are met. If it is less than 250, it can be considered that the predetermined conditions are not met. Of course, this is just an example and does not serve as any limitation in this application.

[0052] S104: Generate an image mask having a plurality of flag bits for the first scanned document image, wherein different flag bits carry different encoding information.

[0053] In the present application, the flag bit can be in any form, for example, it can include but not limited to a flag code, a flag point, or a block-shaped flag block. The setting position of the flag bit on the first scanned document image can be set as needed. In some exemplary embodiments, the flag bit can be implemented by selecting a QR (Quick Response) code according to the needs, or by using an AprilTags code, or can also be implemented in other ways.

[0054] Different flag bits carry different coding information, which can facilitate subsequent steps to use different coding information to perform image matching through different flag bits.

[0055] The specific implementation of step S104 is not limited in the present application, and an image mask with multiple flags can be generated for the first scanned document in any manner. In some optional embodiments, refer to Figure 4 In the flowchart shown, step S104 includes sub-steps S1041 and S1042, specifically:

[0056] S1041: Generate multiple markers for the first scanned document image according to the image content shape and image content boundary information of the first scanned document image, wherein the image convex hull formed by connecting the multiple markers can completely cover the image content area of ​​the first scanned document image.

[0057] In the present application, the outermost points of multiple markers are connected to form a convex polygon, and the convex polygon can be used as the convex hull of the image.

[0058] The image content shape and image content boundary information of the first scanned document refer to the shape and boundary information of the image content with substantial content in the first scanned document image, respectively. For example, the image content may be the whole of each target content in the first scanned document image. Based on the image content shape and image content boundary information of the first scanned document image, a plurality of markers are generated for the first scanned document image, so as to ensure that the image convex hull formed by connecting the plurality of markers can completely cover the image content area of ​​the first scanned document image, thereby avoiding omission of image content.

[0059] For example, if the image content of the first scanned document image is approximately rectangular in shape, and the image content boundary information is approximately rectangular in shape, then multiple flags can be set above or outside the rectangular boundary of the image content in the first scanned document image, respectively, and a suitable number of flags can be set as needed to ensure that the image convex hull formed by the connected flags can completely cover the image content area of ​​the first scanned document image. Of course, this example using a rectangle is only for ease of understanding and is not intended to be any limitation to the present application, and other regular or irregular shapes may also be used.

[0060] It should be understood that the number of flags is not particularly limited in the present application. Generally, the more flags there are, the higher the accuracy of image matching in the subsequent step S106. For example, in some optional embodiments, the multiple flags are at least 4 flags. By using at least 4 flags, the time for processing and setting more flags can be saved on the basis of being able to effectively meet the needs, and the amount of calculation for image matching in the subsequent step S106 can be reduced to the greatest extent. For example, it can also be combined with Figure 8 It is understood that it schematically shows an image mask of 4 flag bits.

[0061] For example, taking the setting of four flags as an example, the four flags can be respectively set near or outside the four corners of the minimum circumscribed rectangle of the image content area of ​​the first scanned document image, so as to ensure that the image convex hull formed by the four flags connected can completely cover the image content area of ​​the first scanned document image. Of course, this is just an example, and the setting positions of the four flags can also be selected as needed.

[0062] S1042: Generate an image mask based on the first scanned document image and multiple flags.

[0063] In the present application, the first scanned document image with multiple markers added thereto is used to obtain an image mask, that is, the image mask contains both the image content in the first scanned document image and the markers.

[0064] Based on this, through the optional implementation of sub-steps S1041 to S1042 in the present application, an image mask with multiple flag bits can be accurately and effectively generated for the first scanned document image to facilitate data processing in subsequent steps.

[0065] S106: Acquire a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device.

[0066] In the present application, the physical image acquisition device may include any electronic device with a camera, for example, including but not limited to at least one of a camera, a mobile phone, a computer, a tablet computer, etc. The physical image acquisition device may be used to capture the image mask through its camera to achieve image acquisition, so as to obtain the second scanned document image.

[0067] Through step S106, it is convenient to obtain the second scanned document image by utilizing the most realistic shooting environment, which can better solve the sample distribution problem. At the same time, when the machine learning model is trained with the subsequently generated training samples, the machine learning model can also have better generalization.

[0068] In some optional embodiments, step S106 includes: acquiring a second scanned document image obtained by a physical image acquisition device performing image acquisition on an image mask output onto a physical medium.

[0069] Based on this, in this application, the real second scanned document image can be effectively collected and obtained in this way without the need to adjust parameters. Therefore, it does not require long-term debugging and learning by relevant personnel, the time cost is low, and the overall implementation cost of training sample generation is also low.

[0070] In the present application, the physical medium includes but is not limited to a screen display, real paper, curtain, etc. For example, taking the physical image acquisition device as a mobile phone as an example, the image mask can be output on the screen display (including but not limited to a computer screen, a TV screen, a mobile phone screen, etc.) by means of screen display, and then the mobile phone is used to shoot to collect the complete image mask including the mark position on the screen display, thereby obtaining the second scanned document image, and in addition, the position and shooting angle of the mobile phone can be changed while shooting, and the real second scanned document image with different moiré effects can be obtained. Alternatively, the image mask can be output on real paper by printing, and then the mobile phone is used to shoot to collect the complete image mask including the mark position on the paper, thereby obtaining the second scanned document image, and in addition, the position and shooting angle of the mobile phone can be changed while shooting, and the real second scanned document image with different effects can be obtained. Alternatively, the image mask can be output on the curtain by projection, and then the mobile phone is used to shoot to collect the complete image mask including the mark position on the curtain, thereby obtaining the second scanned document image, and in addition, the position and shooting angle of the mobile phone can be changed while shooting, and the real second scanned document image with different effects can be obtained. It should be understood that the above description is only for example and is not intended to limit the present application in any way.

[0071] Alternatively, other feasible methods may be used to obtain the second scanned document image, so as to facilitate subsequent processing to obtain the required training sample pairs, which is not limited in this application.

[0072] Of course, the specific physical media to be used can be determined according to needs, and this application does not limit this. For example, in one implementation, different physical media can be selected according to different types of filter processing performed above to generate different training sample pairs.

[0073] By collecting real second scanned document images (for example, second scanned document images with moiré effects) and obtaining training samples therefrom to train the machine learning model, it is also helpful to expand the capabilities of the machine learning model.

[0074] S108: performing image matching on the image mask and the second scanned document image based on the multiple flags, and generating a training sample pair based on the matched image mask and the second scanned document image.

[0075] Since the second scanned document image is acquired based on the image mask, the multiple markers on the image mask also exist on the second scanned document image. The multiple markers on the second scanned document image correspond one-to-one to the multiple markers on the image mask. Therefore, image matching of the image mask and the second scanned document image can be achieved based on the multiple markers, and then the required training sample pairs can be generated based on the matched image mask and the second scanned document image, so as to facilitate the training of the machine learning model used to implement software scanning and improve the training effect of the machine learning model.

[0076] Based on the optional implementation of the above steps S102 to S108, an image mask can be generated based on the first scanned document image, and then the image mask is used for image acquisition through a physical image acquisition device to obtain a real second scanned document image corresponding to the image mask, and then the image mask and the second scanned document image are matched based on the flag, and the two are combined into a training sample pair for subsequent training of the machine learning model for software scanning. On the one hand, because the second scanned document image is an image acquired by a physical image acquisition device, it does not need to be generated by adjusting parameters, and it is a real image itself. Therefore, it does not require long-term debugging and learning by the corresponding personnel, and the time cost is low. The overall implementation cost of training sample generation is also low. On the other hand, the first scanned document image after filtering is an image with a relatively clean background, and the second scanned document image is a real image taken of the image mask generated based on the first scanned document image. Its background is the real background and may contain more interference and noise data. Combining the image mask and the second scanned document image as a training sample pair to train the machine learning model that implements software scanning can make the data distribution closer to the real sample distribution, thereby improving the training effect of the machine learning model and its subsequent performance on real data.

[0077] The specific implementation of step S108 is not limited in this application. In some optional embodiments, refer to Figure 5 In the flowchart shown, step S108 includes sub-steps S1081, S1082 and S1083, specifically:

[0078] S1081: Based on multiple flags, image matching is performed on the image mask and the second scanned document image.

[0079] Optionally, a plurality of markers in the image mask and a plurality of markers in the second scanned document image may be matched first, and then image matching between the image mask and the second scanned document image may be achieved based on the matching result.

[0080] In some optional embodiments, referring to Figure 6In the flowchart shown, sub-step S1081 includes the following sub-steps S1081A, S1081B and S1081C, specifically:

[0081] S1081A: Extract multiple first markers in the image mask and multiple second markers in the second scanned document image respectively.

[0082] For ease of explanation, in this embodiment, the first flag is used to indicate the flag in the image mask, and the second flag is used to indicate the flag in the second scanned document image. Then, optionally, the first coordinates of multiple first flags in the image mask and the second coordinates of multiple second flags in the second scanned text image can be extracted respectively, so as to extract multiple first flags from the image mask and multiple second flags from the second scanned document image respectively. The extraction of flags can be implemented in any appropriate manner, including but not limited to flag detection or recognition algorithms, machine learning models for extracting target objects (flags in this example), etc.

[0083] S1081B: Match multiple first flag bits and multiple second flag bits according to the encoding information of the flag bit to obtain multiple pairs of matching flag bit pairs.

[0084] Optionally, the indication of the coding information may be determined, and the first marker and the second marker at the same position in the two images may be determined and matched respectively, thereby obtaining a plurality of matching marker pairs.

[0085] For example, if the image mask includes 4 first markers, and the second scanned document image also includes 4 second markers, the 4 first markers correspond to the 4 second markers one by one, and 4 matching marker pairs are obtained by matching. The rest of the cases can be deduced in the same way.

[0086] S1081C: Based on multiple pairs of matching flag bits, image matching is performed on the image mask and the second scanned document image.

[0087] On the basis of obtaining multiple pairs of matching flag bits, multiple pairs of matching flag bits can be used for calculation, and any appropriate image matching algorithm can be used to perform image matching between the image mask and the second scanned document image to achieve the target effect.

[0088] Based on this, in the present application, through the optional implementation of the above sub-steps S1081A to S1081C, it is possible to more accurately and effectively achieve image matching of the image mask and the second scanned document image based on multiple markers.

[0089] Of course, the present application does not limit the specific implementation of sub-step S1081C. In some optional embodiments, refer to Figure 7In the flowchart, sub-step S1081C includes the following sub-steps S1081C1 and S1081C2, specifically:

[0090] S1081C1: Based on multiple pairs of matching flag bits, solve the homography matrix between the image mask and the second scanned document image.

[0091] Optionally, a homography transformation can be performed based on multiple pairs of matching flags, and a homography matrix can be obtained by solving the result of the homography transformation. Homography transformation is a commonly used transformation method in the field of computer vision. It is a method of mapping points on a plane to another plane. Its purpose is to achieve geometric transformation of images, mapping all points on an image to another image. It can be simply understood as describing the position mapping relationship between the world coordinate system and the pixel coordinate system of an object. The transformation matrix corresponding to the homography transformation is the homography matrix.

[0092] For example, Figure 8 The image mask and the second scanned document image in the example are understood as follows: the image mask includes 4 first markers, and the second scanned document image includes 4 second markers. The 4 first markers correspond to the 4 second markers one by one, and there are 4 matching marker pairs in total. Then, through the same homography transformation, that is, using the same homography matrix, each of the 4 first markers can be transformed to the second marker that forms a matching marker pair with the first marker, and therefore, the homography transformation algorithm can be used for calculation to realize the solution of the homography matrix between the image mask and the second scanned document image based on the 4 matching marker pairs.

[0093] S1081C2: According to the solution result, the second scanned document image is projected onto the image mask to perform image matching between the image mask and the second scanned document image.

[0094] After obtaining the solution result of the homography matrix, the second scanned document image can be projected onto the image mask using the homography matrix as a transformation parameter to achieve image matching between the image mask and the second scanned document image.

[0095] Based on this, in this application, through the above-mentioned sub-steps S1081C1~S1081C2, the homography matrix between the image mask and the second scanned document image is first solved based on the matching flag pair, and then the second scanned document image is projected onto the image mask according to the solution result, so that the image matching of the image mask and the second scanned document image can be achieved more accurately and effectively.

[0096] S1082: Based on the matching result, the same image area is cropped for the image mask and the second scanned document image.

[0097] After the image matching is completed, the image mask and the second scanned document image can be cropped in the same image area to obtain two cropped images. The background of the cropped image cropped from the image mask is relatively clean, while the cropped image cropped from the second scanned document image is the real background and may contain more interference and noise data, thereby facilitating the subsequent generation of training sample pairs based on the two cropped images.

[0098] Optionally, when cropping the same area, the main area of ​​the cropped image can be the main area, that is, the image content of the layer mask and the second scanned document image (the image content may refer to the entirety of each target content in the corresponding image) can be cropped to retain the image content details in the layer mask and the second scanned document image as much as possible.

[0099] In some optional embodiments, step S1082 includes: determining the same image area in the image mask and the second scanned document image based on the position correspondence between multiple pairs of matching marker pairs; and cropping the same image area in the image mask and the second scanned document image respectively.

[0100] In this way, the present application can accurately and effectively implement cropping of the same image area of ​​the image mask and the second scanned document image, so as to facilitate the subsequent generation of training sample pairs.

[0101] S1083: Generate a training sample pair based on the cropped image mask and the second scanned document image.

[0102] After the image mask and the second scanned document image are cropped, a training sample pair can be generated based on the two cropped images of the same image area obtained by cropping. For example, the two cropped images of the same image area obtained by cropping can be directly used as training samples, or the two cropped images of the same image area obtained by cropping can be further processed and then used as training samples.

[0103] Based on this, in this application, through the above-mentioned sub-steps S1081 to S1083, the image matching of the image mask and the second scanned document image can be achieved more accurately and effectively, and then based on the matched image mask and the second scanned document image, a better training sample pair can be generated to facilitate the training of the machine learning model used to implement software scanning and improve the training effect of the machine learning model.

[0104] In some optional embodiments, the training sample generating method further includes: training a machine learning model for implementing software scanning based on the training sample pairs.

[0105] The training samples obtained by the technical solution of this application can be used to train the machine learning model used to implement software scanning, so that the data distribution is closer to the real sample distribution, thereby improving the training effect of the machine learning model and its subsequent performance on real data. In addition, since the technical solution in this application can efficiently generate different training sample pairs, avoiding the cumbersome sample labeling and manual operation process, it is also convenient to improve the generalization of the machine learning model and quickly and specifically help the machine learning model to iterate.

[0106] Figure 8 A schematic diagram of an example scenario in an embodiment of the present application is shown. Figure 8 The training sample generation method in the embodiment of the present application is generally understood as shown in FIG. Figure 8 As shown, in step S102 of the present application: "obtaining a first scanned document image after filtering", in the first scanned document image after filtering, except for text type content, chart type content and color block pattern type content, it is processed into a pure color background (for example, a pure white background); then, through step S104, an image mask with multiple flag bits is generated for the first scanned document image after filtering, wherein different flag bits carry different encoding information, Figure 8 In the example, four flags are added, and the image convex hull formed by the four flags connected together can completely cover the image content area of ​​the first scanned document image; step S106 in the present application: "obtaining a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device", the second scanned document image is obtained based on the image mask, and the second scanned document image is an image acquired by a physical image acquisition device, and is generated without parameter adjustment. It is a real image itself, and therefore, does not require long-term debugging and learning by corresponding personnel, has low time cost, and the overall implementation cost of training sample generation is also low; then, the image mask obtained in step S104 and the second scanned document image obtained in step S106 are used, and after step S108, image matching is performed on the image mask and the second scanned document image based on multiple flags, and a training sample pair is generated based on the matched image mask and the second scanned document image. Figure 8 In the example, the left image of the training sample pair is obtained by cropping the image mask, and the right image is obtained by cropping the second scanned document image. The image mask and the second scanned document image crop the same image area. It should be understood that Figure 8 The example scenarios shown are only used to facilitate understanding of the embodiments of the present application and are not intended to be any limitation to the present application.

[0107] In summary, the scheme provided in the embodiment of the present application can generate an image mask based on the first scanned document image, and then perform image acquisition on the image mask through a physical image acquisition device to obtain a real second scanned document image corresponding to the image mask, and then match the image mask and the second scanned document image based on the flag, and combine the two into a training sample pair, so as to train the machine learning model for implementing software scanning in the future. On the one hand, because the second scanned document image is an image acquired by a physical image acquisition device, it does not need to be generated by adjusting parameters, and it is a real image itself. Therefore, it does not require long-term debugging and learning by the corresponding personnel, the time cost is low, and the overall implementation cost of training sample generation is also low. On the other hand, the first scanned document image after the filter processing is an image with a relatively clean background, and the second scanned document image is a real image taken based on the image mask generated by the first scanned document image, and its background is a real background, which may contain more interference and noise data. The image mask and the second scanned document image are combined as a training sample pair to train the machine learning model for implementing software scanning, which can be closer to the real sample distribution in data distribution, improve the training effect of the machine learning model, and the subsequent performance on real data.

[0108] It is understandable that the above description of the training sample generation method is only an exemplary description of the present application and does not constitute any limitation to the present application.

[0109] According to a second aspect of the embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction enables the processor to perform an operation corresponding to the method described in the first aspect. Fig. 9 , shows a schematic diagram of the structure of an electronic device according to an embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0110] like Fig. 9 As shown, the electronic device 900 may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

[0111] in:

[0112] The processor 902 , the communication interface 904 , and the memory 906 communicate with each other via a communication bus 908 .

[0113] The communication interface 904 is used to communicate with other electronic devices or servers.

[0114] The processor 902 is used to execute the program 910, and specifically can execute the relevant steps in the above-mentioned training sample generation method embodiment.

[0115] Specifically, the program 910 may include program codes, which include computer operation instructions.

[0116] The processor 902 may be a CPU, a GPU (Graphic Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0117] The memory 906 is used to store the program 910. The memory 906 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0118] The program 910 may include multiple computer instructions. Specifically, the program 910 may enable the processor 902 to execute operations corresponding to the training sample generation method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0119] The specific implementation of each step in program 910 can refer to the corresponding description of the corresponding steps and units in the above method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the above method embodiment, which will not be repeated here.

[0120] According to the third aspect of the embodiments of the present application, the embodiments of the present application further provide a computer storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in any of the foregoing multiple method embodiments is implemented. The computer storage medium includes, but is not limited to: a compact disc read-only memory (CD-ROM), a random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk, etc.

[0121] An embodiment of the present application further provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any training sample generation method in the above-mentioned multiple method embodiments.

[0122] The electronic device 900 / computer storage medium / computer program product embodiment in the embodiment of the present application has been described in detail in the aforementioned training sample generation method embodiment, so its related content and beneficial effects can be understood with reference to the above-mentioned method embodiment and will not be repeated here.

[0123] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0124] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0125] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or implemented as a computer code originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded through a network and stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., a random access memory (RAM), a read-only memory (ROM), a flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0126] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for specific applications, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0127] The above implementation methods are only used to illustrate the embodiments of the present application, and are not limitations on the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also belong to the scope of the embodiments of the present application. The scope of patent protection of the embodiments of the present application should be limited by the claims.

Claims

1. A training sample generation method, comprising: Acquire a first scanned document image after filter processing; Generating an image mask having a plurality of marker bits for the first scanned document image, wherein different marker bits carry different encoding information, and an image convex hull formed by connecting the plurality of marker bits can completely cover an image content area of ​​the first scanned document image; Acquire a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device; Based on the multiple flags, performing image matching on the image mask and the second scanned document image, and generating a training sample pair based on the matched image mask and the second scanned document image; The acquiring of the first scanned document image after filter processing includes: performing filter processing on the first scanned document image to remove the background, so as to obtain the first scanned document image with a pure background color.

2. The method according to claim 1, wherein: The performing image matching on the image mask and the second scanned document image based on the multiple flags, and generating a training sample pair based on the matched image mask and the second scanned document image, comprises: Based on the multiple flags, performing image matching on the image mask and the second scanned document image; According to the matching result, cropping the same image area of ​​the image mask and the second scanned document image; Based on the cropped image mask and the second scanned document image, a training sample pair is generated.

3. The method according to claim 2, wherein: The performing image matching on the image mask and the second scanned document image based on the multiple flag bits includes: Respectively extracting a plurality of first markers in the image mask and a plurality of second markers in the second scanned document image; According to the encoding information of the flag bit, the plurality of first flag bits and the plurality of second flag bits are matched to obtain a plurality of matching flag bit pairs; Based on the multiple pairs of matching flag bits, image matching is performed on the image mask and the second scanned document image.

4. The method according to claim 3, wherein: The performing image matching on the image mask and the second scanned document image based on the multiple pairs of matching flag bits includes: Based on the multiple pairs of matching flag pairs, solving the homography matrix between the image mask and the second scanned document image; According to the solution result, the second scanned document image is projected onto the image mask to perform image matching between the image mask and the second scanned document image.

5. The method according to claim 3, wherein: The step of cutting out the same image area of ​​the image mask and the second scanned document image according to the matching result includes: Based on the position correspondence between the multiple pairs of matching flag pairs, the same image area in the image mask and the second scanned document image is determined; and the same image area in the image mask and the second scanned document image is cropped respectively.

6. The method according to any one of claims 1 to 5, wherein: The step of generating an image mask having a plurality of flag bits for the first scanned document image comprises: generating a plurality of flag bits for the first scanned document image according to the image content shape and image content boundary information of the first scanned document image; An image mask is generated based on the first scanned document image and the multiple markers.

7. The method according to claim 6, wherein: The multiple flag bits are at least 4 flag bits.

8. The method according to claim 1, wherein: The performing filter processing on the first scanned document image to remove the background to obtain a first scanned document image with a pure background color includes: Performing target content detection on the first scanned document image, and determining a corresponding target content area according to the detection result; The pixel values ​​of other areas in the first scanned document image except the target content area are set to preset pure color pixel values ​​to obtain a first scanned document image with a pure color background color.

9. The method according to claim 8, wherein: After determining the corresponding target content area according to the detection result, the method further includes: performing mask processing on the target content area to obtain a corresponding target content area mask; The step of setting the pixel values ​​of other areas in the first scanned document image except the target content area to preset pure color pixel values ​​includes: setting the pixel values ​​of other areas in the first scanned document image except the target content mask to preset pure color pixel values.

10. The method according to claim 9, wherein: After obtaining the corresponding target content area mask, the method further includes: performing morphological processing on the target content area mask; The step of setting the pixel values ​​of other areas outside the target content mask in the first scanned document image to preset solid color pixel values ​​includes: setting the pixel values ​​of other areas outside the target content mask after morphological processing in the first scanned document image to preset solid color pixel values.

11. The method according to any one of claims 1 to 5, wherein: The step of acquiring a second scanned document image obtained by performing image acquisition on the image mask by a physical image acquisition device includes: A second scanned document image is obtained by acquiring an image of the image mask outputted onto a physical medium by a physical image acquisition device.

12. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1-11.

13. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.