Image processing apparatus, image processing system, detection apparatus, image processing method, and image processing program

Abstract

To provide an image processing apparatus, an image processing system, a detection apparatus, an image processing method, and an image processing program capable of efficiently generating various character images as learning data.SOLUTION: An image processing apparatus includes: a selection processing unit which selects a combination of character drawing information of a predetermined character string and background drawing information of the character string; an image generation processing unit which generates a character image based on the combination of the character drawing information and the background drawing information selected by the selection processing unit; and a learning data generation processing unit which generates learning data including the character image generated by the image generation processing unit.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present disclosure relates to a technique for performing image processing such as character recognition on an image. [Background technology]

[0002] Conventionally, there are known techniques for extracting character strings from images of documents such as documents and slips. For example, a character recognition device is known that judges whether a rectangle is valid or invalid based on a set reference value, sets the line direction based on the result of an area discrimination process, divides line rectangles extracted from valid rectangles into one-character units, and performs character recognition by comparing image features of the extracted characters with character features in a pattern dictionary (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-256772 A Summary of the Invention [Problem to be solved by the invention]

[0004] When generating learning data for text characters, it is desirable to generate text characters with a variety of foreground colors, background colors, and background patterns. However, if the color and background pattern of the text characters are randomly determined, it becomes difficult to efficiently generate a character string with the characteristics to be learned.

[0005] An object of the present disclosure is to provide an image processing device, an image processing system, a detection device, an image processing method, and an image processing program capable of efficiently generating a variety of character images as learning data. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present disclosure includes a selection processing unit, an image generation processing unit, and a learning data generation processing unit. The selection processing unit selects a combination of character drawing information of a predetermined character string and background drawing information of the character string. The image generation processing unit generates a character image based on the combination of the character drawing information and the background drawing information selected by the selection processing unit. The learning data generation processing unit generates learning data including the character image generated by the image generation processing unit.

[0007] An image processing system according to another aspect of the present disclosure includes the image processing device and a learning device that generates a trained model by performing machine learning using the learning data generated by the image processing device.

[0008] A detection device according to another aspect of the present disclosure detects character regions of a string of characters from an input image using the trained model generated by the learning device.

[0009] An image processing method according to another aspect of the present disclosure is a method in which one or more processors execute the following steps: selecting a combination of character drawing information for a specified character string and background drawing information for the character string; generating a character image based on the combination of the character drawing information and the background drawing information; and generating learning data including the character image.

[0010] An image processing program relating to another aspect of the present disclosure is a program for causing one or more processors to execute the following steps: selecting a combination of character drawing information of a specified character string and background drawing information of the character string; generating a character image based on the combination of the character drawing information and the background drawing information; and generating learning data including the character image. Effect of the Invention

[0011] According to the present disclosure, it is possible to provide an image processing device, an image processing system, a detection device, an image processing method, and an image processing program capable of efficiently generating a variety of character images as learning data. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a functional block diagram showing the configuration of an image processing device according to a first embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram illustrating an example of a document (quote) according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a diagram illustrating an example of a document image according to an embodiment of the present disclosure. [Figure 4] FIG. 4 is a flowchart showing an example of a procedure of image processing executed in the image processing device according to the first embodiment of the present disclosure. [Figure 5A] FIG. 5A is a diagram showing an example of a first character image according to the first embodiment of the present disclosure. [Figure 5B] FIG. 5B is a diagram showing an example of a first character image according to the first embodiment of the present disclosure. [Figure 5C] FIG. 5C is a diagram showing an example of a first character image according to the first embodiment of the present disclosure. [Figure 6] FIG. 6 is a diagram illustrating an example of an entire image according to an embodiment of the present disclosure. [Figure 7A] FIG. 7A is a diagram showing an example of a second character stroke according to the first embodiment of the present disclosure. [Figure 7B] FIG. 7B is a diagram showing an example of a second character stroke according to the first embodiment of the present disclosure. [Figure 7C] FIG. 7C is a diagram showing an example of a second character stroke according to the first embodiment of the present disclosure. [Figure 8] FIG. 8 is a diagram showing a region of a character string according to the first embodiment of the present disclosure. [Figure 9] FIG. 9 is a diagram showing layout coordinates of the text color region according to the first embodiment of the present disclosure. [Figure 10]FIG. 10 is a diagram showing an example of an entire image according to the first embodiment of the present disclosure. [Figure 11] FIG. 11 is a functional block diagram showing the configuration of an image processing device according to the second embodiment of the present disclosure. [Figure 12] FIG. 12 is a flowchart showing an example of a procedure of image processing executed in the image processing device according to the second embodiment of the present disclosure. [Figure 13A] FIG. 13A is a diagram showing an example of text according to the second embodiment of the present disclosure. [Figure 13B] FIG. 13B is a diagram showing an example of text according to the second embodiment of the present disclosure. [Figure 13C] FIG. 13C is a diagram showing an example of text according to the second embodiment of the present disclosure. [Figure 14A] FIG. 14A is a diagram showing an example of image processing according to the second embodiment of the present disclosure. [Figure 14B] FIG. 14B is a diagram showing an example of image processing according to the second embodiment of the present disclosure. [Figure 15A] FIG. 15A is a diagram showing an example of image processing according to the second embodiment of the present disclosure. [Figure 15B] FIG. 15B is a diagram showing an example of image processing according to the second embodiment of the present disclosure. [Figure 15C] FIG. 15C is a diagram showing an example of image processing according to the second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments are examples of the present disclosure and are not intended to limit the technical scope of the present disclosure.

[0014] "First embodiment" [Image processing device 1] 1 is a block diagram showing a configuration of an image processing device 1 according to a first embodiment of the present disclosure. The image processing device 1 executes a process of generating data of a character image and data of the arrangement coordinates of the character image as learning data (teacher data) to be used for machine learning. The image processing device 1 also performs machine learning using the learning data to generate a trained model that detects a character region of a character string from an input image.

[0015] 1, the image processing device 1 includes a control unit 11, a storage unit 12, an operation display unit 13, and a communication unit 14. The image processing device 1 may be one or more cloud servers, or one or more physical servers.

[0016] The communication unit 14 is a communication interface for connecting the image processing device 1 to a network in a wired or wireless manner and for executing data communication with an external device via the network in accordance with a predetermined communication protocol. The network may be, for example, the Internet or a LAN.

[0017] The operation display unit 13 is a user interface that includes a display unit, such as a liquid crystal display or an organic EL display, that displays various information, and an operation unit, such as a mouse, a keyboard, or a touch panel, that accepts operations.

[0018] The storage unit 12 is a non-volatile storage unit such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory that stores various information. The storage unit 12 stores a control program such as an image processing program (one example of an image processing program of the present disclosure) for causing the control unit 11 to execute image processing (see FIG. 4) described below. For example, the image processing program is non-temporarily recorded on a computer-readable recording medium such as a CD or a DVD, and is read by a reading device (not shown) such as a CD drive or a DVD drive provided in the image processing device 1 and stored in the storage unit 12. The image processing program may be distributed from a cloud server and stored in the storage unit 12.

[0019] The storage unit 12 also stores image data (scanned data, etc.) of documents and the like acquired from external devices.

[0020] FIG. 2 shows an estimate as an example of a document. As shown in FIG. 2, the estimate includes multiple character strings such as the document category ("estimate"), issue date, destination, contact information of the estimator, estimated amount, product name, quantity, standard price, discount amount, subtotal, consumption tax, and total amount. The user scans the estimate using an external device and uploads image data (input image) to the image processing device 1. When the control unit 11 acquires the image data of the estimate, it stores it in the storage unit 12. As another embodiment, the control unit 11 may acquire a document file of the estimate created in an external device and store the document file in the storage unit 12.

[0021] The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic operations. The ROM stores in advance control programs such as a BIOS and an OS for causing the CPU to execute various processes. The RAM stores various information and is used as a temporary storage memory (work area) for the various processes executed by the CPU. The control unit 11 controls the image processing device 1 by executing, by the CPU, various control programs that are stored in advance in the ROM or the storage unit 12.

[0022] When the control unit 11 acquires image data of a document, it performs preprocessing such as resolution conversion processing, skew correction processing, and background removal processing on the image data, and then performs character string (document part) recognition processing (extraction processing). For example, the control unit 11 recognizes each part such as character strings, tables, illustrations, and seals as rectangular areas of objects in the image data on which the preprocessing has been performed. Furthermore, after performing document part recognition processing, the control unit 11 performs character recognition processing (OCR processing) on ​​the entire image data (entire image). Furthermore, the control unit 11 uses the result of the OCR processing to integrate the character data into character strings (wordization).

[0023] For example, in image data D1 (input image) of an estimate shown in Fig. 3, the control unit 11 recognizes character strings using an object recognition processing AI. The control unit 11 sets character string rectangles (rectangular areas) indicating the range of each recognized character string in the image data D1. The rectangular frame shown in Fig. 3 is an example of a character string rectangle (rectangular area). When a character string is properly recognized, multiple characters from the start character to the end character of each character string will fit within the rectangular area, as shown in Fig. 3.

[0024] Here, a character string may be composed of characters of various fonts, sizes, decorations, etc. In conventional techniques, it is not possible to set an optimal character string rectangular area (rectangular area) when recognizing and drawing a character string in an input image, making it difficult to detect a correct character area of ​​a character string from an input image. In response to this, the image processing device 1 according to the present disclosure has a configuration capable of appropriately detecting a character area of ​​a character string in an input image, as will be described below.

[0025] Specifically, as shown in Fig. 1, the control unit 11 includes various processing units such as a first image generation processing unit 111, a second image generation processing unit 112, a calculation processing unit 113, a decision processing unit 114, a learning data generation processing unit 115, and a learning processing unit 116. The control unit 11 functions as the various processing units by executing various processes according to the image processing program. Some or all of the processing units included in the control unit 11 may be configured with electronic circuits. The image processing program may be a program for causing a plurality of processors to function as the various processing units.

[0026] The first image generation processing unit 111 generates a first character image Ps1 of a predetermined character string based on at least one of a character font, a foreground color, a background color, and a background pattern. Specifically, the first image generation processing unit 111 randomly determines at least one of a character font, a foreground color, a background color, and a background pattern to generate the first character image Ps1. The first image generation processing unit 111 also determines text characters selected from well-known dictionary data as the character string, and determines at least one of a character font, a foreground color, a background color, and a background pattern for the determined character string to generate the first character image Ps1. FIGS. 5A to 5C are examples of the first character image Ps1.

[0027] The second image generation processing unit 112 generates a second character image Ps2 for detecting the coordinates of the character string based on the predetermined character string. Specifically, the second image generation processing unit 112 fixes the foreground color and background color of the character string determined by the first image generation processing unit 111 to predetermined colors and generates an image excluding the background pattern as the second character image Ps2. That is, the second image generation processing unit 112 extracts only the text portion (character shape) of the first character image Ps1 and determines it as the second character image Ps2. FIGS. 7A to 7C are an example of the first character image Ps1.

[0028] The calculation processing unit 113 calculates the arrangement coordinates of the character region of the character string of the first character image Ps1 with respect to the whole image A1 corresponding to the learning data, based on the second character image Ps2 generated by the second image generation processing unit 112. Specifically, the calculation processing unit 113 calculates the position coordinates of the character string with respect to a predetermined rectangular area set for the character string of the first character image Ps1, and calculates the arrangement coordinates of the character region with respect to the whole image A1 based on the calculated position coordinates. The rectangular area is, for example, a roughly circumscribing rectangular area surrounding the character string. For example, the control unit 11 sets the rectangular area to an area obtained by adding a predetermined gap (margin area) from the start and end of the character string. The control unit 11 may set the margin area according to the type of the character string (hiragana, katakana, alphanumeric characters, symbols, full-width, half-width, etc.), the length and height of the character string, etc.

[0029] The determination processing unit 114 determines the arrangement coordinates of the rectangular area relative to the entire image A1. The calculation processing unit 113 calculates the arrangement coordinates of the character area relative to the entire image A1 based on the arrangement coordinates of the rectangular area determined by the determination processing unit 114 and the position coordinates of the character string relative to the rectangular area (the position of the character string within the rectangular area).

[0030] The learning data generation processing unit 115 generates learning data used for machine learning. Specifically, the learning data generation processing unit 115 generates learning data including a first character image Ps1 generated by the first image generation processing unit 111 and coordinate information of the arrangement coordinates of the character area calculated by the calculation processing unit 113. Specifically, the learning data generation processing unit 115 generates learning data including an entire image A1 (see FIG. 10) in which each of the multiple first character images Ps1 is arranged based on the arrangement coordinates, and the coordinate information of each of the multiple first character images Ps1.

[0031] The learning processing unit 116 generates a trained model by performing machine learning using the learning data generated by the learning data generation processing unit 115. The learning processing unit 116 may be configured as a device (learning device) separate from the image processing device 1. In this case, an image processing system including the image processing device 1 and a learning device that generates a trained model by performing machine learning using the learning data generated by the image processing device 1 may be constructed.

[0032] In addition, the trained model can be applied to various output devices (such as character recognition devices and general-purpose computers). For example, when an image to be character-recognized is input to the output device, the control unit of the output device uses the trained model to execute character recognition processing on the input image and output the character recognition result. For example, the control unit recognizes a character string in a document image by OCR processing or the like, and outputs the recognized character string as a character recognition result or registers it in a database. This makes it possible to appropriately extract characters of a character string from a document image. The output device is an example of a detection device of the present disclosure.

[0033] Machine learning includes algorithms such as supervised learning using supervised data, unsupervised learning using unsupervised data, and reinforcement learning. Furthermore, to realize these methods, a method called "deep learning" is used, which learns to extract features themselves. In this embodiment, the control unit 11 has a learning model based on the various algorithms described above. In this embodiment, data in which coordinate information is added to data of the entire image A1 corresponds to supervised data, and data of the entire image A1 corresponds to unsupervised data. The control unit 11 can perform machine learning using the supervised data and unsupervised data as input data (learning data) to generate a trained model that executes character recognition processing. That is, the image processing device 1 functions as a learning device that generates a trained model.

[0034] The trained model may be downloaded to an operation terminal for use, or may be stored in a server (cloud server) and accessed from an operation terminal via the Internet or the like for use. When an arbitrary input image is input to the operation terminal, the trained model outputs an optimal character recognition result. That is, the operation terminal (an example of a detection device of the present disclosure) detects a character region of a character string from an input image using the trained model generated by a learning device.

[0035] In addition, the control unit 11 of the image processing device 1 may display various information on the operation display unit 13 in the process of generating learning data. For example, the control unit 11 displays the first character image Ps1 generated by the first image generation processing unit 111, the second character image Ps2 generated by the second image generation processing unit 112, and the entire image A1 on the operation screen. This allows a user who generates learning data to generate the learning data while checking the generation process of the learning data.

[0036] The specific configuration of each processing unit will be described below with a specific example.

[0037] [Image processing] FIG. 4 is a flowchart showing an example of a procedure of image processing executed in the image processing device 1 according to the first embodiment.

[0038] The present disclosure can be understood as an image processing method (image processing method of the present disclosure) that executes one or more steps included in the image processing. One or more steps included in the image processing described here may be omitted as appropriate. The steps in the image processing may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the control unit 11 of the image processing device 1 executes each step in the image processing, in other embodiments, one or more processors may execute each step in the image processing in a distributed manner.

[0039] In step S1, the control unit 11 determines the text (character string) to be drawn. A method of determining the text may be to load word segments or sentences into an array, select them randomly, and read all or part of them. For example, the control unit 11 may select fixw=[“(a)”, “xyz”, “abcd”, “text”, “GHI”, “play”, “Yes”, “01”, “[2]”, “{3}”, “ <4> ", ",", "@pqr"], and each element from 0 to 12 is selected randomly. The control unit 11 may also determine the text using a general Japanese dictionary or English dictionary, or may determine the text by partially extracting it from an existing text document (input image). When extracting from a text document, the number of characters (Tn) in the selected document in the array may be found, and the start position (Ts) may be determined by randomly selecting "0 to Tn / 2", and the end position (Te) may be determined by randomly selecting "Ts+1 to Te".

[0040] Next, in step S2, the control unit 11 determines the character font of the determined text. For example, the control unit 11 lists a plurality of types of character fonts and style, and determines the index number randomly to determine the character font. The character font may also include italics, etc. The control unit 11 may also determine the font size of the determined text randomly from a numerical value in a predetermined range (points: 30 to 60).

[0041] Next, in step S3, the control unit 11 determines the foreground color and background color of the determined text. For example, the control unit 11 determines the foreground color as FGC=(Fr, Fg, Fb) and the background color as BGC=(Br, Bg, Bb), and determines Fr, Fg, Fb, Br, Bg, and Bb randomly.

[0042] Next, in step S4, the control unit 11 determines pattern images for the foreground and background of the determined text. For example, the control unit 11 randomly determines pattern images for the foreground and background, such as dots, lines, wavy lines, etc., including no pattern. The control unit 11 may determine the range of the background pattern image to be a range in which 10 pixels are added to the top, bottom, left, and right of the range of the text, or a range in which 5% is added to the top, bottom, left, and right of the range of the text.

[0043] Next, in step S5, the control unit 11 generates an image. Specifically, the control unit 11 generates a first character image Ps1 (an example of a first character image of the present disclosure) which is an image of text based on the information determined by the processes of steps S1 to S4. An example of the first character image Ps1 is shown in Fig. 5A to Fig. 5C. Fig. 5A shows the first character image Ps1 generated based on the text "ABCDE" and information on the character font, foreground color, background color, and pattern image, Fig. 5B shows the first character image Ps1 generated based on the text "XYZ" and information on the character font, foreground color, background color, and pattern image, and Fig. 5C shows the first character image Ps1 generated based on the text "PQRS" and information on the character font, foreground color, background color, and pattern image.

[0044] Next, in step S6, the control unit 11 determines the placement coordinates of the rectangular area relative to the entire image A1. Specifically, the control unit 11 determines the placement coordinates of the rectangular area, which is a roughly circumscribing rectangular area surrounding the text (first character image Ps1), relative to the entire image A1.

[0045] FIG. 6 shows a specific example of the entire image A1. For example, the entire image A1 has the same size as the document (input image) to be subjected to character recognition. The control unit 11 determines the position where the generated first character image Ps1 is arranged (drawn) in the entire image A1. In addition, the dotted frame in FIG. 6 corresponds to the rectangular area. For example, in the entire image A1, it is desirable that the rectangular area to be arranged does not overlap with the rectangular area already arranged immediately before (to the left) or above. Assuming that the start point of the rectangular area arranged immediately before is (Xs, Ys), the width is W, the height is H, and the space between the areas is S, the start point of the rectangular area to be arranged when arranged to the right at the same height is (Xs+W+S, Ys). In addition, the start point of the rectangular area to be arranged when arranged in another row (for example, below) is (Xs, Ys+H+S).

[0046] In addition, the start point P1 of the top left rectangular area that is placed first in the entire image A1 may be set to the margins (Sw,Sh) of the entire image A1, as in an actual document. In the example shown in Fig. 6, the start point P1 (Xs11,Ys11) of the first rectangular area is represented by (Sw,Sh). Moreover, the start point P2 (Xs12,Ys12) of the second (adjacent to the right) rectangular area is represented by (Xs11+W11+S,Sh).

[0047] Furthermore, the control unit 11 may determine the placement coordinates of the rectangular area on the second row by setting the maximum value H (H=max(H11, H12, . . . , H1N)) of the multiple (N) rectangular areas on the first row as the height (Y coordinate). For example, the start point (Xs21, Ys21) of the rectangular area on the second row is expressed as (Xs11, Ys11+H+S).

[0048] The control unit 11 determines the placement coordinates of each of the generated first character images Ps1 relative to the entire image A1.

[0049] Next, in step S7, the control unit 11 composites the first character image Ps1, which is an image, into the entire image A1. Specifically, the control unit 11 composites the image (first character image Ps1) generated in step S5 into the current entire image A1 at the position of the arrangement coordinates of the rectangular area determined in step S6. The control unit 11 may composite by replacing the original pixel values ​​of the entire image A1 with the pixel values ​​of the image image, or may composite by subtracting the difference between the maximum pixel value and the image data value of the image image from the entire image A1. The maximum pixel value is, for example, 255 in the case of 8-bit data. By compositing in this manner, a composite image close to reality can be generated without impairing the color of the original image of the entire image A1.

[0050] After the process of step S2, the control unit 11 executes the following processes of steps S21 to S23 in parallel with the processes of steps S3 to S7.

[0051] In step S21, the control unit 11 generates a second character image Ps2 (image) for detecting the coordinates of a character string. Specifically, the control unit 11 generates a second character image Ps2 consisting only of the character shapes of the text determined in steps S1 and S2. The control unit 11 generates a second character image Ps2 in which, for example, the foreground is fixed to black and the background is fixed to white. FIGS. 7A to 7C show an example of the second character image Ps2. FIG. 7A shows the second character image Ps2 generated based on the text "ABCDE" and information on the character font, foreground color ("black"), and background color ("white"); FIG. 7B shows the second character image Ps2 generated based on the text "XYZ" and information on the character font, foreground color ("black"), and background color ("white"); and FIG. 7C shows the second character image Ps2 generated based on the text "PQRS" and information on the character font, foreground color ("black"), and background color ("white").

[0052] Next, in step S22, the control unit 11 calculates the local coordinates (position coordinates) of the character string in the second character image Ps2. Specifically, the control unit 11 detects an area in which the character string exists within a rectangular area based on the second character image Ps2, as shown in Fig. 8. For example, the control unit 11 detects the area in which the character string exists by accumulating density values ​​(white "0" to black "100") in the horizontal and vertical directions and determining the rising and falling edges of the values.

[0053] For example, in the case of vertical accumulation, the first point from the left and right where the value is greater than or equal to 0 is found. The detected position on the left becomes the start point of the local coordinates of the string (lx), the detected position on the top becomes the start point of the local coordinates (ly), the difference between the left and right becomes the width in local coordinates (lw), and the difference between the top and bottom becomes the height in local coordinates (lh).

[0054] Next, in step S23, the control unit 11 calculates the layout coordinates of the character region of the character string of the second character image Ps2 in the entire image A1. Specifically, the control unit 11 converts the local coordinates calculated in step S22 into the coordinate system of the entire image A1 to calculate the layout coordinates of the character region of the character string of the second character image Ps2. For example, as shown in Fig. 9, the character region of the second character image Ps2 in a rectangular region has a start point (lx, ly) represented by (Xs+lx, Ys+ly), a width of lw, and a height of lh.

[0055] After the processes of steps S7 and S23, in step S8, the control unit 11 generates and outputs learning data. Specifically, the control unit 11 generates learning data including information on the overall image A1 synthesized with the first character image Ps1 and the layout coordinates of the character area of ​​the second character image Ps2. The layout coordinates of the character area of ​​the character string in the overall image A1 are calculated based on the layout coordinates (Xs, Ys) of the rectangular area of ​​the first character image Ps1 in the overall image A1 and the layout coordinates (lx, ly) of the character string (second character image Ps2) in the first character image Ps1.

[0056] The control unit 11 repeatedly executes the above-mentioned process for each of the multiple texts to generate learning data of an entire image A1 in which multiple first character images Ps1 are combined, as shown in FIG.

[0057] The control unit 11 performs machine learning using multiple learning data (multiple whole images A1 and multiple coordinate information) to generate a trained model that detects a character region of a character string from an input document image when a document image of an actual document (see FIG. 3) is input. The trained model may be a model that detects a character region from a document image and recognizes a character string in the detected character region from the document image.

[0058] Furthermore, when the trained model is installed in a character recognition device (such as a user terminal), when an image to be character recognized (such as a document image) is input, the character recognition device uses the trained model to execute a character region detection process and a character recognition process on the input image and outputs a character recognition result. For example, the character recognition device recognizes a character string in the detected character region by OCR processing or the like, and outputs the recognized character string as a character recognition result or registers it in a database. The character recognition device is an example of a detection device of the present disclosure.

[0059] As described above, the image processing device 1 according to the first embodiment generates a first character image Ps1 of a predetermined character string based on at least one of a character font, a foreground color, a background color, and a background pattern, and generates a second character image Ps2 for detecting the coordinates of the character string based on the predetermined character string. The image processing device 1 also calculates, based on the second character image Ps2, the arrangement coordinates of the character region of the character string relative to the entire image A1 corresponding to the learning data, and generates learning data including the first character image Ps1 and coordinate information of the arrangement coordinates.

[0060] That is, the image processing device 1 draws (generates) a character pattern (first character image Ps1) of a preloaded word by combining various character fonts, foreground colors, background colors, and background patterns, and generates a partial character pattern (second character image Ps2) for coordinate detection based on the character pattern. The image processing device 1 also detects the coordinates (range) of a character string using the second character image Ps2, and calculates the placement coordinates of the character string on the original drawn image (whole image A1).

[0061] According to the above configuration, it is possible to generate learning data including information on the layout coordinates of text (character strings) of any pattern relative to the entire image A1. Therefore, by performing machine learning using the learning data, it is possible to build a system (character recognition device) capable of appropriately detecting character regions of character strings in an input image.

[0062] [Second embodiment] Incidentally, when generating learning data for text characters (first character image Ps1), it is desirable to generate text characters with various foreground colors, background colors, and background patterns. However, a method of randomly determining the color and background pattern of text characters makes it difficult to efficiently generate a character string with the characteristics to be learned. In contrast, the image processing device 1 according to the second embodiment has a configuration capable of efficiently generating various character images as learning data.

[0063] In the following, configurations different from the first embodiment will be described, and configurations common to the first embodiment will be given the same reference numerals and descriptions thereof will be omitted.

[0064] 11 is a block diagram showing the configuration of an image processing device 1 according to the second embodiment. The control unit 11 further includes a selection processing unit 117 and an adjustment processing unit 118 in addition to the configuration according to the first embodiment (see FIG. 1).

[0065] The selection processing unit 117 selects a combination of character drawing information of a predetermined character string (text) and background drawing information of the character string. Specifically, the selection processing unit 117 selects a combination of a foreground color and a background color of the text (first character image Ps1). For example, when the first image generation processing unit 111 determines an arbitrary text and a character font, the selection processing unit 117 selects a combination of a foreground color and a background color of the text. In addition, the selection processing unit 117 selects at least two pieces of color information from among a dark color, a light color, and a high saturation color, and determines them as the foreground color and the background color.

[0066] For example, selection processing unit 117 sets the dark color to a luminance value equal to or less than a predetermined dark threshold, sets the light color to a luminance value equal to or more than a predetermined light threshold, sets at least one of the red component, green component, and blue component of the high saturation color to the dark threshold or less, and sets at least one of the red component, green component, and blue component of the foreground color to the light threshold or more. The foreground color is an example of character drawing information of the present disclosure, and the background color and background pattern image are examples of background drawing information of the present disclosure.

[0067] The first image generation processing unit 111 generates a character image (first character image Ps1) based on a combination of the character drawing information and the background drawing information selected by the selection processing unit 117. The learning data generation processing unit 115 generates learning data including the first character image Ps1 generated by the first image generation processing unit 111.

[0068] The adjustment processing unit 118 adjusts the foreground color and background color selected by the selection processing unit 117. Specifically, the visibility of the text (first character image Ps1) may be deteriorated depending on the combination of the foreground color and background color selected by the selection processing unit 117. Therefore, the adjustment processing unit 118 executes a process to improve the visibility of the text by adjusting the foreground color and background color selected by the selection processing unit 117.

[0069] For example, when one of the foreground color and the background color is a high saturation color and the other is a dark color, the adjustment processing unit 118 increases the luminance values ​​of the red and green components of the foreground color. Also, when one of the foreground color and the background color is a high saturation color and the other is a light color, the adjustment processing unit 118 decreases the luminance values ​​of the red and green components of the foreground color.

[0070] FIG. 12 is a flowchart showing an example of a procedure of image processing executed in the image processing device 1 according to the second embodiment.

[0071] In the flowchart shown in FIG. 12, step S3 in the flowchart shown in FIG. 4 is replaced with steps S31 and S32.

[0072] After determining the text and character font (steps S1 and S2), the control unit 11 selects the foreground color and background color in step S31. Specifically, the foreground color is FGC=(Fr, Fg, Fb), the background color is BGC=(Br, Bg, Bb), and Fr, Fg, Fb, Br, Bg, and Bb are randomly determined. Here, in order to generate character images with various color combinations more efficiently, the control unit 11 defines at least the following three color groups and assigns them to the FGC and BGC, respectively.

[0073] For example, 8-bit (0 to 255) image data is used as an example. All dark colors (DK) are set to a dark threshold value (thDK) or less (for example, thDK=50), and each color channel is set to 0 to 50. All light colors (BR) are set to a light threshold value (thBR) or more (for example, thBR=200), and each color channel is set to 200 to 255. For high saturation colors (ST), at least one of the red component (Fr), green component (Fg), and blue component (Fb) of the foreground color is set to a dark threshold value (thDK) or less, and at least one of the red component (Fr), green component (Fg), and blue component (Fb) of the foreground color is set to a light threshold value (thBR) or more.

[0074] The control unit 11 randomly determines each color channel within the corresponding range. The method of determining the high saturation color (ST) is to obtain a random number between 0 and 5 based on the light color (BR), divide it by 3 to obtain the quotient Q and the remainder R, and when Q=0, set the channel [R] to a dark color (DK), and when Q=1, set the channels other than [R] to dark colors (DK).

[0075] These are defined as a desired combination, for example, FGC=DK, BGC=BR. Also, the combination of [FGC, BGC] may be written in multiple arrays and selected randomly.

[0076] For example, the control unit 11 randomly assigns 0 to 5 to six elements {[DK,BR], [BR,DK], [ST,DK], [DK,ST], [BR,ST], [ST,BR]} and selects a combination of [FGC,BGC]. FIG. 13A shows an example of a combination of a foreground color FGC and a background color BGC [ST,DK] for text drawn on a background pattern. FIG. 13B shows an example of a combination of a foreground color FGC and a background color BGC [DK,ST]. FIG. 13C shows an example of a combination of a reversed character text and a foreground color FGC and a background color BGC [ST,DK].

[0077] As colors with other characteristics, each color channel may be set to be equal to or greater than the dark threshold (thDK) and equal to or less than the light threshold (thBR) as a mid-tone color (MD), or may be set to be white (WP) as a 255 color. The control unit 11 may select a combination of foreground and background colors including mid-tones and white colors.

[0078] In step S32, the control unit 11 adjusts the foreground color and background color. Depending on the combination of color characteristics selected in step S31 and the colors selected, it is assumed that the combination may be poor in visibility and may be unrealistic. For example, in a combination of a foreground (light foreground) of colored characters and a dark background, or in a combination of a dark character (dark foreground) and a colored background (light background), if the colored characters are bluish, the visibility may be poor (see FIG. 14A). Also, in a combination of a foreground (light foreground) of colored characters and a light background, or in a combination of a light character (light foreground) and a colored background (light background), if the colored characters and the colored background are yellowish, the visibility may be poor (see FIG. 15A).

[0079] For example, when one of the foreground color FGC and the background color BGC is a high saturation color (ST) and the other is a dark color (DK), the control unit 11 increases the luminance values ​​of Fr and Fg (for example, increases the luminance value of the character by 75) (see FIG. 14B) because the color becomes bluish when ST's Q=1 and R=2 (see FIG. 14A). Also, when one of the foreground color FGC and the background color BGC is a high saturation color (ST) and the other is a light color (BR), the control unit 11 decreases the luminance values ​​of Fr and Fg (for example, multiplies the luminance value by 1 / 2) (see FIG. 15B and FIG. 15C) because the color becomes yellowish when ST's Q=0 and R=2 (see FIG. 15A). Note that FIG. 15B shows an example in which the luminance value of the background color is decreased, and FIG. 15C shows an example in which the luminance value of the foreground color is decreased.

[0080] After adjusting the foreground color and background color, the control unit 11 generates a first character image Ps1 in the same manner as in the first embodiment (S4 to S7).

[0081] Furthermore, in parallel with the processes of steps S3 (S31, S32) to S7, the control unit 11 executes the processes of steps S21 to S23 in the same manner as in the first embodiment. Note that, as another embodiment of the second embodiment, the control unit 11 may omit the processes of steps S21 to S23.

[0082] As described above, the image processing device 1 of the second embodiment selects a combination of character drawing information of a specified character string and background drawing information of the character string, generates a character image based on the selected combination of the character drawing information and the background drawing information, and generates learning data including the generated character image.

[0083] That is, the image processing device 1 defines a plurality of color property groups (e.g., dark colors, light colors, high saturation colors), randomly determines colors within a range that satisfies the properties, and determines combinations of these according to the type of text to be drawn (e.g., reversed characters, background patterns, etc.), thereby generating data (first character images Ps1) with a variety of color combinations. This makes it possible to generate learning data with a variety of colors. This makes it possible to efficiently generate a variety of character images as learning data.

[0084] The image processing device 1 also includes a configuration for adjusting the selected foreground color and background color. For example, when one of the foreground color and the background color is a high saturation color and the other is a dark color, the image processing device 1 increases the luminance values ​​of the red and green components of the foreground color. When one of the foreground color and the background color is a high saturation color and the other is a light color, the image processing device 1 decreases the luminance values ​​of the red and green components of the foreground color.

[0085] According to the above configuration, it is possible to generate text with good visibility (first character image Ps1) by excluding text with poor visibility (first character image Ps1).

[0086] [Disclosure Note 1] Below, an outline of the disclosure extracted from the above-mentioned first embodiment will be described. Note that the configurations and processing functions described in the following notes can be selected and combined as desired.

[0087] <Appendix 1> a first image generation processing unit that generates a first character image of a predetermined character string based on at least one of a character font, a foreground color, a background color, and a background pattern; a second image generation processing unit that generates a second character image for detecting a coordinate of the character string based on the predetermined character string; a calculation processing unit that calculates, based on the second character image generated by the second image generation processing unit, a layout coordinate of a character region of the character string with respect to an entire image corresponding to learning data; a learning data generation processing unit that generates the learning data including the first character image generated by the first image generation processing unit and coordinate information of the arrangement coordinates calculated by the calculation processing unit; An image processing device comprising:

[0088] <Appendix 2> the calculation processing unit calculates a position coordinate of the character string with respect to a predetermined rectangular area set for the character string, and calculates a layout coordinate of the character area with respect to the entire image based on the calculated position coordinate; 2. An image processing device as described in claim 1.

[0089] <Appendix 3> a determination processing unit that determines a placement coordinate of the rectangular area relative to the entire image, the calculation processing unit calculates a layout coordinate of the character region with respect to the entire image based on a layout coordinate of the rectangular region determined by the determination processing unit and a position coordinate of the character string with respect to the rectangular region; 3. An image processing device as described in claim 2.

[0090] <Appendix 4> the learning data generation processing unit generates the learning data including the whole image in which each of the plurality of first character images is arranged based on the arrangement coordinates, and the coordinate information of each of the plurality of first character images. 4. The image processing device according to claim 2 or 3.

[0091] <Appendix 5> the second image generation processing unit fixes the foreground color and the background color of the first character image to predetermined colors, and generates, as the second character image, an image obtained by excluding the background pattern of the first character image. 5. The image processing device according to claim 1,

[0092] <Appendix 6> the first image generation processing unit randomly determines at least one of the character font, the foreground color, the background color, and the background pattern to generate the first character image; 6. The image processing device according to claim 1,

[0093] <Appendix 7> An image processing device according to any one of Supplementary Notes 1 to 6, A learning device that generates a trained model by performing machine learning using the learning data generated by the image processing device; An image processing system comprising:

[0094] <Appendix 8> A detection device that detects character regions of a string of characters from an input image using the trained model generated by the learning device described in appendix 7.

[0095] [Disclosure Note 2] The following is an outline of the disclosure extracted from the above-described second embodiment. Note that the configurations and processing functions described in the following supplementary notes can be arbitrarily combined by selecting them.

[0096] <Appendix 1> a selection processing unit that selects a combination of character drawing information of a predetermined character string and background drawing information of the character string; an image generation processing unit that generates a character image based on a combination of the character drawing information and the background drawing information selected by the selection processing unit; a learning data generation processing unit that generates learning data including the character image generated by the image generation processing unit; An image processing device comprising:

[0097] <Appendix 2> the selection processing unit selects a combination of a foreground color, which is a character color of the character string, and a background color of the character string. 2. An image processing device as described in claim 1.

[0098] <Appendix 3> the selection processing unit selects at least two pieces of color information from among a dark color, a light color, and a high saturation color, and determines the selected pieces of color information as the foreground color and the background color; 3. An image processing device as described in claim 2.

[0099] <Appendix 4> the selection processing unit sets the dark color to a luminance value equal to or less than a predetermined dark color threshold, sets the light color to a luminance value equal to or more than a predetermined light color threshold, sets at least one of the red component, the green component, and the blue component of the high saturation color to the dark color threshold or less, and sets at least one of the red component, the green component, and the blue component of the foreground color to the light color threshold or more. 4. The image processing device according to claim 1,

[0100] <Appendix 5> an adjustment processing unit that adjusts the foreground color and the background color selected by the selection processing unit, 5. The image processing device according to claim 2,

[0101] <Appendix 6> the adjustment processing unit increases luminance values ​​of red and green components of the foreground color when one of the foreground color and the background color is a high saturation color and the other is a dark color. 6. The image processing device according to claim 5.

[0102] <Appendix 7> the adjustment processing unit reduces luminance values ​​of red and green components of the foreground color when one of the foreground color and the background color is a high saturation color and the other is a light color. 7. The image processing device according to claim 5 or 6.

[0103] <Appendix 8> An image processing device according to any one of claims 1 to 7, A learning device that generates a trained model by performing machine learning using the learning data generated by the image processing device; An image processing system comprising:

[0104] <Appendix 9> A detection device that detects character regions of a string of characters from an input image using the trained model generated by the learning device described in appendix 8. [Explanation of symbols]

[0105] 1: Image processing device 11: Control section 12: Storage section 13: Operation display section 14: Communications Department 111: First image generation processing unit 112: second image generation processing unit 113: Calculation processing unit 114: Decision processing unit 115: Learning data generation processing unit 116: Learning processing unit 117: Selection processing unit 118: Adjustment processing unit A1: Overall image Ps1: First character image Ps2: Second character image

Claims

1. a selection processing unit that selects a combination of character drawing information of a predetermined character string and background drawing information of the character string; an image generation processing unit that generates a character image based on a combination of the character drawing information and the background drawing information selected by the selection processing unit; a learning data generation processing unit that generates learning data including the character image generated by the image generation processing unit; An image processing device comprising:

2. the selection processing unit selects a combination of a foreground color, which is a character color of the character string, and a background color of the character string. The image processing device according to claim 1 .

3. the selection processing unit selects at least two pieces of color information from among a dark color, a light color, and a high saturation color, and determines the selected pieces of color information as the foreground color and the background color; The image processing device according to claim 2 .

4. the selection processing unit sets the dark color to a luminance value equal to or less than a predetermined dark color threshold, sets the light color to a luminance value equal to or more than a predetermined light color threshold, sets at least one of the red component, the green component, and the blue component of the high saturation color to the dark color threshold or less, and sets at least one of the red component, the green component, and the blue component of the foreground color to the light color threshold or more. The image processing device according to claim 1 .

5. an adjustment processing unit that adjusts the foreground color and the background color selected by the selection processing unit, The image processing device according to claim 2 .

6. the adjustment processing unit increases luminance values ​​of red and green components of the foreground color when one of the foreground color and the background color is a high saturation color and the other is a dark color. The image processing device according to claim 5 .

7. the adjustment processing unit reduces luminance values ​​of red and green components of the foreground color when one of the foreground color and the background color is a high saturation color and the other is a light color. The image processing device according to claim 5 .

8. An image processing device according to any one of claims 1 to 7, A learning device that generates a trained model by performing machine learning using the learning data generated by the image processing device; An image processing system comprising:

9. A detection device that detects a character region of a character string from an input image by using the trained model generated by the learning device according to claim 8.

10. selecting a combination of character drawing information of a predetermined character string and background drawing information of the character string; generating a character image based on a combination of the character drawing information and the background drawing information; generating learning data including the character image; An image processing method executed by one or more processors.

11. selecting a combination of character drawing information of a predetermined character string and background drawing information of the character string; generating a character image based on a combination of the character drawing information and the background drawing information; generating learning data including the character image; An image processing program for causing one or more processors to execute the above.

Images