Image correction device, character recognition device, image correction method, character recognition method, program, and recording medium

Abstract

To provide an image correction device capable of easily generating a facing image, a character recognition device, an image correction method, a character recognition method, a program and a recording medium.SOLUTION: In an image correction device 1, an image acquisition unit acquires a correction object image including an object. A terminal information acquisition unit acquires terminal posture information being information of a posture of an imaging device in acquiring the correction object image, and a distance information acquisition unit acquires object distance information being information of distance from the imaging terminal to the object. A reference posture information acquisition unit acquires reference posture information being a posture of the object in acquiring the correction object image. A reference surface setting unit sets a facing reference surface on the basis of the terminal posture information, the object distance information and the reference posture information. An image correction unit corrects the correction object image to a facing image seen from a vertical direction from the facing reference surface on the basis of the terminal posture information and the reference posture information.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image correction device, a character recognition device, an image correction method, a character recognition method, a program, and a recording medium.

Background Art

[0002] Regarding products flowing on a production line, it is necessary to manage the products individually in order to prevent mis-shipment, mix-up, etc. At this time, it is generally carried out to manage by attaching tags such as barcodes and RFID. On the other hand, for products such as steel products where the environment during product processing is harsh, the durability of the tags is insufficient and they cannot be attached. In this case, management is performed by directly printing characters (for example, engraving printing, stamp printing, stencil spraying, etc.) on the product (for example, Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In such a situation, in order to obtain identification information of a product by image recognition, an image of the product is captured. However, engraving and the like are difficult to appear in a photograph due to lighting conditions. Also, it is not always possible to capture the product from the front. And in the case of an oblique image, there is a problem that the accuracy when performing character recognition or the like using image recognition decreases. This is the same in various fields where articles are managed by engraving or the like. For this reason, a technique for easily normalizing an image is demanded.

[0005] Therefore, an object of the present invention is to provide an image correction device that can easily generate a normalized image.

Means for Solving the Problems

[0006] To achieve the above objective, the image correction apparatus of the present invention is It includes an image acquisition unit, a terminal information acquisition unit, a distance information acquisition unit, a reference posture information acquisition unit, a reference plane setting unit, and an image correction unit. The image acquisition unit acquires the image to be corrected, The image to be corrected is an image that includes an object, The terminal information acquisition unit acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The distance information acquisition unit acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The aforementioned reference posture information acquisition unit acquires reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The reference plane setting unit sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction unit corrects the image to be corrected to an orthogonal image viewed from a direction perpendicular to the orthogonal reference plane, based on the terminal orientation information and the reference orientation information.

[0007] To achieve the above objective, the character recognition device of the present invention is Including an image correction unit, a character candidate selection unit, and a character recognition unit, The image correction unit is the image correction device of the present invention, The character candidate selection unit selects a character candidate region from the image corrected by the image correction unit, The character recognition unit performs character recognition on the image corrected by the image correction procedure.

[0008] The image correction method of the present invention is This process includes an image acquisition step, a terminal information acquisition step, a distance information acquisition step, a reference orientation information acquisition step, a reference plane setting step, and an image correction step. The aforementioned image acquisition step acquires the image to be corrected, The image to be corrected is an image that includes an object, The terminal information acquisition step acquires terminal posture information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The distance information acquisition step acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above-mentioned reference posture information acquisition step acquires reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The aforementioned reference plane setting step sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction step corrects the image to be corrected to an orthogonalized image viewed from a direction perpendicular to the orthogonalized reference plane, based on the terminal orientation information and the reference orientation information.

[0009] To achieve the above objective, the character recognition method of the present invention is This process includes an image correction step, a character candidate selection step, and a character recognition step. The image correction step is the image correction method of the present invention, The character candidate selection step involves selecting a character candidate region from the image corrected by the image correction step, The character recognition step involves performing character recognition on the image corrected by the image correction procedure.

[0010] The first program of the present invention includes an image acquisition procedure, a terminal information acquisition procedure, a distance information acquisition procedure, a reference posture information acquisition procedure, a reference plane setting procedure, and an image correction procedure. The aforementioned image acquisition procedure involves acquiring the image to be corrected, The image to be corrected is an image that includes an object, The aforementioned terminal information acquisition procedure acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The aforementioned distance information acquisition procedure acquires object distance information, The object distance information is information on the distance from the imaging terminal to the object, The reference posture information acquisition procedure acquires reference posture information, The reference posture information is information on the posture of the object at the time of acquisition of the correction target image, The reference plane setting procedure sets a front-facing reference plane based on the terminal posture information, the object distance information, and the reference posture information, The image correction procedure corrects the correction target image into a front-facing image viewed from the vertical direction from the front-facing reference plane based on the terminal posture information and the reference posture information, Each of the above procedures is a program for causing a computer to execute them.

[0011] The second program of the present invention includes an image correction procedure, a character candidate selection procedure, and a character recognition procedure, The image correction procedure is executed by the first program, The character candidate selection procedure selects a character candidate area from the image corrected by the image correction procedure, The character recognition procedure performs character recognition on the image corrected by the image correction procedure, and each of the above procedures is a program for causing a computer to execute them.

Advantages of the Invention

[0012] According to the present invention, a front-facing image can be easily generated.

Brief Description of the Drawings

[0013] [Figure 1] FIG. 1 is a block diagram showing a configuration example of an image correction device according to Embodiment 1. [Figure 2] FIG. 2 is a block diagram showing an example of the hardware configuration of the image correction device according to Embodiment 1. [Figure 3] FIG. 3 is a flowchart showing an example of processing in the image correction device according to Embodiment 1. [Figure 4]Figure 4 is an explanatory diagram illustrating an example of the use of the image correction device of Embodiment 1. [Figure 5] Figure 5 is a block diagram showing the configuration of an example of a character recognition device according to Embodiment 2. [Figure 6] Figure 6 is a block diagram showing an example of the hardware configuration of the character recognition device according to Embodiment 2. [Figure 7] Figure 7 is a flowchart showing an example of processing in the character recognition device of Embodiment 2. [Modes for carrying out the invention]

[0014] Next, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiments. In the following drawings, the same parts are denoted by the same reference numerals. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably with those of the others, and unless otherwise specified, the configurations of each embodiment can be combined.

[0015] [Embodiment 1] The image correction device of this embodiment will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of an example of the image correction device 10 of this embodiment. As shown in Figure 1, the image correction device 1 (hereinafter also referred to as "this device 1") includes an image acquisition unit 2, a terminal information acquisition unit 3, a distance information acquisition unit 4, a reference orientation information acquisition unit 5, a reference plane setting unit 6, and an image correction unit 7. Although not shown, this device 1 may also include, for example, a storage unit.

[0016] The device 1 may be, for example, a single device including the aforementioned parts, or it may be a device in which the aforementioned parts can be connected via a communication network. Furthermore, the device 1 can be connected to external devices described later via a communication network. The communication network is not particularly limited and can use any known network, such as wired or wireless. Examples of communication networks include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), Local 5G, LPWA, etc. The wireless communication may be in the form of direct communication between devices (Ad Hoc communication), infrastructure communication, or indirect communication via an access point. The device 1 may, for example, be incorporated into a server as part of a system. Furthermore, the device 1 may be, for example, a personal computer (PC, e.g., desktop or notebook type), smartphone, tablet terminal, etc., on which the program of the present invention is installed. The device 1 may also be an imaging terminal capable of imaging an object (e.g., a smartphone or tablet terminal with a camera), or a device capable of communicating with the imaging terminal. Moreover, the device 1 may be in the form of cloud computing or edge computing, for example, in which at least one of the above parts is on a server and the other parts are on a terminal.

[0017] Figure 2 illustrates a block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a CPU 101, memory 102, bus 103, storage device 104, input device 105, output device 106, communication device (communication unit) 107, etc. Each part of the device 10 is interconnected via the bus 103 through its respective interface (I / F).

[0018] The CPU 101 operates in conjunction with other components, such as a controller (system controller, I / O controller, etc.), and is responsible for the overall control of the device 1. In the device 1, the CPU 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. Specifically, for example, the CPU 101 functions as an image acquisition unit 2, a terminal information acquisition unit 3, a distance information acquisition unit 4, a reference posture information acquisition unit 5, a reference plane setting unit 6, and an image correction unit 7. The device 1 is equipped with a CPU as its computing device, but it may also be equipped with other computing devices such as a GPU (Graphics Processing Unit) or an APU (Accelerated Processing Unit), or a combination of the CPU and these.

[0019] Bus 103 can also be connected to external devices, for example. Examples of such external devices include external storage devices (external databases, etc.), printers, external input devices, external display devices, audio output devices such as speakers, external imaging devices such as cameras, and various sensors such as acceleration sensors, geomagnetic sensors, and direction sensors. Device 1 can be connected to an external network (the aforementioned communication network) via a communication device 107 connected to bus 103, for example, and can also be connected to other devices such as a user's terminal via the external network.

[0020] Memory 102 may be, for example, main memory. When the CPU 101 performs processing, memory 102 reads various operational programs, such as the program of the present invention, stored in the storage device 104 (described later), and the CPU 101 receives data from memory 102 and executes the program. The main memory may be, for example, RAM (random access memory). Alternatively, memory 102 may be, for example, ROM (read-only memory).

[0021] The storage device 104 is also called an auxiliary storage device, for example, in relation to the main memory (primary memory). As described above, the storage device 104 stores an operating program including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing to the recording medium. The recording medium is not particularly limited and may be internal or external, for example, an HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD). If the device 1 includes, for example, the storage device 104 functions as the storage unit. The storage device 104 may store, for example, at least one of the correction target image, reference posture information, object distance information, alignment reference plane, and alignment image, which will be described later.

[0022] In this device 1, the memory 102 and storage device 104 can also store various types of information, such as log information, information obtained from an external database (not shown) or external devices, information generated by this device 1, and information used by this device 1 when executing processing. At least some of the information may be stored on an external server other than the memory 102 and storage device 104, or it may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

[0023] The device 10 further includes, for example, an input device 105 and an output device 106. The input device 105 may include, for example, a pointing device such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; an audio input means such as a microphone; and so on. The output device 106 may include, for example, a display device such as an LED display or liquid crystal display; an audio output device such as a speaker; a printer; and so on. In this embodiment 1, the input device 105 and the output device 106 are configured separately, but the input device 105 and the output device 106 may be configured as an integrated unit, such as a touch panel display.

[0024] Next, an example of the image correction method of this embodiment will be described based on the flowchart in Figure 3. The image correction method of this embodiment is carried out as follows, for example, using the image correction device 1 shown in Figures 1 and 2. Note that the image correction method of this embodiment is not limited to the use of the image correction device 1 shown in Figures 1 and 2.

[0025] First, the image acquisition unit 2 of the image correction device 1 acquires the image to be corrected (S1, image acquisition step). The image to be corrected is, for example, an image that includes an object. The image acquisition unit 2 may acquire the image to be corrected using an imaging device such as a camera provided in the device 1, or it may acquire the image to be corrected from an imaging device outside the device via a communication network. The image to be corrected may be, for example, a video or a still image, or it may be an image that has already been captured or an image preview image. If the image to be corrected is an image preview image, the image acquisition unit 2 acquires the image preview image in real time, for example. The image acquisition unit 2 may store the acquired image to be corrected in, for example, a storage device 104 or memory 102.

[0026] Next, the terminal information acquisition unit 3 acquires terminal orientation information (S2, terminal information acquisition step). The terminal orientation information is information about the orientation of the imaging terminal at the time of acquiring the image to be corrected, and can be estimated from, for example, a gyro sensor, acceleration sensor, geomagnetic sensor, distance sensor (for example, optical sensors such as 3D-Lidar, millimeter-wave sensors, ultrasonic sensors, etc.) equipped in the imaging terminal. Alternatively, the terminal orientation information may also be information obtained by, for example, imaging the imaging terminal from the outside at the time of acquiring the image to be corrected, and estimating the orientation of the imaging terminal from the captured image. Preferably, the terminal orientation information includes, for example, information from the gyro sensor. The orientation information is, for example, information about the orientation coordinate system of the imaging terminal in three axes: the X axis (for example, also called the Roll axis), the Y axis (for example, also called the Pitch axis), and the Z axis (for example, also called the Yaw axis). The terminal information acquisition unit 3 may, for example, store the acquired terminal orientation information in the storage device 104 or memory 102.

[0027] The terminal orientation information may include, for example, other information. This other information may include, for example, information about the shooting location, shooting date and time, and user identification information (name, ID, terminal identification information, etc.).

[0028] Next, the distance information acquisition unit 4 acquires object distance information (S3, distance information acquisition step). The object distance information may be, for example, a predetermined distance (for example, also called a provisional shooting distance), a distance measured from the imaging terminal to the object by a distance sensor provided by the imaging terminal (for example, an optical sensor such as a 3D-Lidar, a millimeter-wave sensor, an ultrasonic sensor, etc.), or a distance estimated from the size of the object included in the image to be corrected. The distance from the size of the object can be estimated, for example, by using distance conversion information that associates the actual distance with the number of pixels in the image, thereby calculating the distance at which the object exists. The distance conversion information may be stored in, for example, the storage unit, or it may be stored in an external database.

[0029] Next, the reference posture information acquisition unit 5 acquires reference posture information (S4, reference posture information acquisition step). The reference posture information is information about the posture of the object at the time the image to be corrected is acquired, and may be, for example, a predetermined value set in advance, or information estimated from a gyro sensor, acceleration sensor, geomagnetic sensor, etc., provided by the object. Alternatively, the reference posture information may be, for example, information obtained by taking an image of the object from the outside at the time the image to be corrected is acquired, and estimating the posture of the object from the image taken. The image may be, for example, the image to be corrected taken by the imaging terminal, or an image taken by another device. The reference posture information is, for example, information about the posture coordinate system of the object in three axes: the X axis (for example, also called the Roll axis), the Y axis (for example, also called the Pitch axis), and the Z axis (for example, also called the Yaw axis). The reference posture information acquisition unit 5 may, for example, store the acquired reference posture information in the storage device 104 or memory 102.

[0030] Next, the reference surface setting unit 6 sets a reference surface for alignment based on the terminal orientation information, the object distance information, and the reference orientation information (S5, reference surface setting step). The reference surface for alignment may be, for example, any plane on the object. If the object is, for example, a steel plate on a production line, the reference surface for alignment may be, for example, the surface on the steel plate on which an identification number or the like is printed.

[0031] Then, the image correction unit 7 corrects the image to be corrected to an orthogonal image viewed from a direction perpendicular to the orthogonal reference plane based on the terminal orientation information and the reference orientation information (S6, image correction step). The image correction unit 7, for example, estimates the coordinates of four points corresponding to four arbitrarily specified points in the image to be corrected, based on the terminal orientation information and the reference orientation information, when the object is viewed from a direction perpendicular to the orthogonal reference plane, and corrects the image to be corrected to an orthogonal image by projection transformation. The coordinates of the four points are not particularly limited, and for example, any coordinates in the image to be corrected can be specified, but it is preferable that they be the coordinates of the area surrounding the feature points of the object included in the image to be corrected. The feature points are, for example, identification information of the object (for example, product management numbers with markings, stencil printing, stamp printing, etc.). In addition, if the image to be corrected is a shooting preview image, the image correction unit 7 corrects the shooting preview image to the orthogonal image in real time. Furthermore, the image correction unit 7 may, for example, crop a predetermined range from the shooting preview image and correct the cropped image to an orthogonal image. The predetermined range may, for example, be the area containing characters in the shooting preview image. In this case, the image correction unit 7 can, for example, extract character candidate areas from the shooting preview image using known character recognition technology, crop a rectangular range based on the character candidate areas, and correct the cropped image to an orthogonal image. In extracting character candidate areas, for example, a trained model created by machine learning using orthogonal images generated by this device 1 may be used to extract character candidate areas from the preview image.

[0032] A specific example of image correction using the device 1 will be explained using Figure 4. In the following explanation, the image correction device 1 is a tablet terminal with a camera function, and the example will be given of capturing images of steel products on a production line using the tablet terminal, but the present invention is not limited in any way to the following example.

[0033] First, as shown in Figure 4(A), the camera function of the tablet terminal 1, which is the device 1, captures the object 30 and acquires a preview image from the camera as the image to be corrected. Next, as terminal orientation information, the orientation coordinate system of the mobile terminal (tablet terminal), indicated by the solid arrow in Figure 4(A), is acquired from the gyro sensor of the tablet terminal 1. In addition, as reference orientation information, the orientation coordinate system of the object, indicated by the dashed arrow in Figure 4(A), is acquired. Next, the device 1 detects the object included in the preview image and estimates the distance between the device 1 and the object from the size of the detected object. Next, based on the terminal orientation information, the object distance information, and the reference orientation information, the device 1 identifies the surface of the object in the preview image and sets the surface as the reference plane for orientation correction. Next, the device 1 specifies the coordinates of four arbitrarily designated points in the preview image, indicated by black circles in Figure 4(B). Then, based on the terminal orientation information and the reference orientation information, the coordinates of the four points corresponding to the object 30 when viewed from the direction perpendicular to the orienting reference plane (shown as white circles in Figure 4(B)) are estimated, and the image to be corrected is corrected to an orienting image by projection transformation.

[0034] According to the image correction device 1 of this embodiment, based on the terminal orientation information, it is possible to easily generate an orthogonalized image in which the image to be corrected is orthogonalized with respect to the orthogonalization reference plane.

[0035] [Embodiment 2] This embodiment is an example of a character recognition device of the present invention. The character recognition device of the present invention includes an image correction unit, a character candidate selection unit, and a character recognition unit, wherein the image correction unit is the image correction device of the present invention, the character candidate selection unit selects a character candidate region from the image corrected by the image correction unit, and the character recognition unit performs character recognition from the image corrected by the image correction unit.

[0036] The character recognition device of this embodiment will be described with reference to Figure 5. Figure 5 is a block diagram showing an example configuration of the character recognition device 20 of this embodiment. As shown in Figure 5, the character recognition device 20 includes an image correction unit 1, a character candidate selection unit 21, and a character recognition unit 22. The character recognition device 20 may also include, for example, a storage unit (not shown). The character recognition device 20 of this embodiment may be incorporated into a server as a system. The character recognition device 20 of this embodiment may also be a personal computer (PC) on which the program of the present invention is installed. Although not shown, the character recognition device 20 may also be connectable to at least one of an external device and an external terminal of a device administrator via a communication network. If the character recognition device 20 is connectable to the external terminal of the device administrator, the device administrator may manage the character recognition device 20 from the external terminal.

[0037] The aforementioned communication network is not particularly limited and can use any known network, such as a wired or wireless network. Examples of such communication networks include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), WiFi (Wireless Fidelity), LPWA (Low Power Wide Area), etc.

[0038] Examples of the external devices include PCs; mobile devices such as mobile phones, smartphones, and tablet terminals; and smartwatches, smart glasses, and wearable devices. The external devices may also include imaging means such as cameras and scanners, and audio input means such as IC (integrated circuit) card readers and microphones.

[0039] Figure 6 illustrates a block diagram of the hardware configuration of the character recognition device 20. The character recognition device 20 includes, for example, a CPU (Central Processing Unit) 101, memory 202, bus 203, storage device 204, input device 205, output device 206, communication device 207, etc. Each part of the character recognition device 20 is connected via the bus 203 by its respective interface (I / F).

[0040] The CPU 201 operates in conjunction with other components, such as a controller (system controller, I / O controller, etc.), and is responsible for the overall control of the character recognition device 20. In the character recognition device 20, the CPU 201 executes, for example, the program of the present invention or other programs, and also reads and writes various types of information. Specifically, for example, the CPU 201 functions as an image correction unit 1, a character candidate selection unit 21, and a character recognition unit 22. The character recognition device 20 is equipped with a CPU as its processing unit, but it may also be equipped with other processing units such as a GPU (Graphics Processing Unit) or an APU (Accelerated Processing Unit), or a combination of the CPU and these. If, for example, the character candidate selection unit 21 performs processing using deep learning, it is preferable that the character recognition device 20 is equipped with a GPU.

[0041] Memory 202 includes, for example, main memory. This main memory is also called primary memory. When the CPU 201 performs processing, memory 202 reads various operational programs, such as the program 105 of the present invention, which are stored in the storage device 204 (auxiliary storage device) described later. The CPU 201 then reads and decodes the data from memory 202 and executes the program. This main memory is, for example, RAM (random access memory). Memory 202 also includes, for example, ROM (read-only memory).

[0042] Bus 203 can also be connected to external devices, for example. Examples of such external devices include imaging devices such as cameras, external storage devices, printers, and audio output devices such as speakers. The character recognition device 20 can be connected to the communication network, for example, by a communication device 208 connected to bus 203, and can also be connected to the external devices via the communication network.

[0043] The storage device 204 is also called an auxiliary storage device, for example, in relation to the main memory (primary memory). As described above, the storage device 204 stores an operating program including the program of the present invention. The storage device 204 includes, for example, a storage medium and a drive for reading from and writing to the storage medium. The storage medium is not particularly limited and may be internal or external, and examples include HD (hard disk), FD (floppy disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc., and the drive is not particularly limited. The storage device 204 may be, for example, a hard disk drive (HDD) in which the storage medium and the drive are integrated.

[0044] The character recognition device 20 further includes, for example, an input device 206 and a display 207. The input device 206 may include, for example, a pointing device such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; and voice input means such as a microphone. The display 207 may include, for example, an LED display or a liquid crystal display. In this second embodiment, the input device 206 and the display 207 are configured separately, but the input device 206 and the display 207 may be configured as an integrated unit, such as a touch panel display.

[0045] In the character recognition device 20, the memory 202 and the storage device 204 can also store user access information and log information, as well as information obtained from an external database (not shown).

[0046] Next, the character recognition method of this embodiment will be explained using the flowchart in Figure 7. The character recognition method of this embodiment can be implemented, for example, using the character recognition device 20 of this embodiment shown in Figures 5 and 6. However, the character recognition method of the present invention is not limited to the use of the character recognition device 20.

[0047] First, the image correction unit 1 of the character recognition device 20 performs S1 to S6 in the same manner as S1 to S6 in the image correction method of Embodiment 1, acquires the image to be corrected, and corrects the image to be corrected into an image that is aligned as viewed from a direction perpendicular to the alignment reference plane.

[0048] Next, the character candidate selection unit 21 of the character recognition device 20 selects character candidate regions from the image corrected by the image correction unit (S11, character candidate selection step). The character candidate selection unit 21 selects, for example, regions (character rectangular regions) that have been determined to contain characters in the image corrected by the image correction unit 1, i.e., the orthogonalized image, as the character candidate regions. The determination of whether or not the orthogonalized image contains characters may be performed, for example, by processing using deep learning, or by using known image processing methods. The selection of character candidate regions using deep learning is preferably performed, for example, by the GPU included in the character recognition device 20.

[0049] Next, the character recognition unit 22 of the character recognition device 20 performs character recognition from the image corrected by the image correction unit 1 (aligned image) (S12, character recognition step). The character recognition process by the character recognition device of the present invention is not particularly limited and can be carried out in the same manner as general character recognition processes (for example, optical character recognition (OCR)).

[0050] According to the character recognition device of the present invention, character recognition is possible from the aligned image corrected by the image correction unit 1. For example, as a result, the accuracy of character recognition can be improved.

[0051] [Embodiment 3] The first program of this embodiment is a program that causes a computer to execute each step of the image correction method described above. Specifically, the first program of this embodiment is a program that causes a computer to execute an image acquisition procedure, a terminal information acquisition procedure, a distance information acquisition procedure, a reference orientation information acquisition procedure, a reference plane setting procedure, and an image correction procedure.

[0052] The aforementioned image acquisition procedure involves acquiring the image to be corrected, The image to be corrected is an image that includes an object, The aforementioned terminal information acquisition procedure acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The aforementioned distance information acquisition procedure acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above procedure for acquiring reference posture information involves acquiring reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The above-mentioned reference plane setting procedure sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information, The image correction procedure corrects the image to be corrected to an orthogonalized image viewed from a direction perpendicular to the orthogonalized reference plane, based on the terminal orientation information and the reference orientation information.

[0053] Furthermore, the first program of this embodiment can also be described as a program that causes the computer to function as an image acquisition procedure, a terminal information acquisition procedure, a distance information acquisition procedure, a reference orientation information acquisition procedure, a reference plane setting procedure, and an image correction procedure.

[0054] The first program of this embodiment can be derived from the description in the image correction apparatus and image correction method of the present invention. Each of the above steps can be read as, for example, "step" or "process". The program of this embodiment may also be recorded on, for example, a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited and includes, for example, random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, floppy disk (FD), and the like.

[0055] [Embodiment 4] The second program of this embodiment is a program that causes a computer to execute each step of the character recognition method described above. Specifically, the second program of this embodiment is a program that causes a computer to execute an image correction procedure and a character recognition procedure.

[0056] The image correction procedure is performed by the first program, The character recognition procedure performs character recognition on the image corrected by the image correction procedure.

[0057] Furthermore, the second program of this embodiment can also be described as a program that causes the computer to function as an image correction procedure and a character recognition procedure.

[0058] The second program of this embodiment can be derived from the description in the character recognition device and character recognition method of the present invention. Each of the above steps can be read as, for example, "step" instead of "process". The program of this embodiment may also be recorded on, for example, a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited and includes, for example, random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, floppy disk (FD), and the like.

[0059] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above embodiments. Various modifications to the configuration and details of the present invention can be understood by those skilled in the art within the scope of the present invention.

[0060] <Note> Some or all of the above embodiments may be described as follows, but are not limited to the following: (Note 1) It includes an image acquisition unit, a terminal information acquisition unit, a distance information acquisition unit, a reference posture information acquisition unit, a reference plane setting unit, and an image correction unit. The image acquisition unit acquires the image to be corrected, The image to be corrected is an image that includes an object, The terminal information acquisition unit acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The distance information acquisition unit acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The aforementioned reference posture information acquisition unit acquires reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The reference plane setting unit sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction unit corrects the image to be corrected to an orthogonal image viewed from a direction perpendicular to the orthogonal reference plane, based on the terminal orientation information and the reference orientation information. Image correction device. (Note 2) The image correction device described in Appendix 1, wherein the terminal orientation information includes information from the gyro sensor of the imaging terminal. (Note 3) The image acquisition unit acquires the shooting preview image in real time as the image to be corrected. The image correction unit corrects the captured preview image to the aligned image in real time, as described in Appendix 1 or 2. (Note 4) The image correction unit crops a predetermined range from the shooting preview image and corrects the cropped image into an orthogonal image. Image correction device as described in Appendix 3. (Note 5) The image correction unit is an image correction device as described in Appendix 4, which crops the area containing characters from the shooting preview image to the predetermined range. (Note 6) Including an image correction unit, a character candidate selection unit, and a character recognition unit, The image correction unit is an image correction device as described in any of appendices 1 to 5. The character candidate selection unit selects a character candidate region from the image corrected by the image correction unit, The character recognition unit is a character recognition device that performs character recognition from an image corrected by the image correction unit. (Note 7) This process includes an image acquisition step, a terminal information acquisition step, a distance information acquisition step, a reference orientation information acquisition step, a reference plane setting step, and an image correction step. The aforementioned image acquisition step acquires the image to be corrected, The image to be corrected is an image that includes an object, The terminal information acquisition step acquires terminal posture information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The distance information acquisition step acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above-mentioned reference posture information acquisition step acquires reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The aforementioned reference plane setting step sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction step corrects the image to be corrected to an orthogonal image viewed from a direction perpendicular to the orthogonal reference plane, based on the terminal orientation information and the reference orientation information. Image correction methods. (Note 8) The image correction method described in Appendix 7, wherein the terminal orientation information includes information from the gyro sensor of the imaging terminal. (Note 9) The aforementioned image acquisition step acquires a shooting preview image in real time as the image to be corrected, The image correction step is the image correction method according to Appendix 7 or 8, which corrects the captured preview image to the orthogonalized image in real time. (Note 10) The image correction step involves cropping a predetermined range from the captured preview image and correcting the cropped image to an orthogonal image. Image correction method as described in Appendix 9. (Note 11) The image correction step is the image correction method described in Appendix 10, which involves cropping the area containing text from the captured preview image to the predetermined range. (Note 12) This process includes an image correction step, a character candidate selection step, and a character recognition step. The aforementioned image correction step is an image correction method described in any of appendices 7 to 11. The character candidate selection step involves selecting a character candidate region from the image corrected by the image correction step, The character recognition step is a character recognition method that performs character recognition on an image corrected by the image correction step. (Note 13) This includes an image acquisition procedure, a terminal information acquisition procedure, a distance information acquisition procedure, a reference orientation information acquisition procedure, a reference plane setting procedure, and an image correction procedure. The aforementioned image acquisition procedure involves acquiring the image to be corrected, The image to be corrected is an image that includes an object, The aforementioned terminal information acquisition procedure acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The aforementioned distance information acquisition procedure acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above procedure for acquiring reference posture information involves acquiring reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The above-mentioned reference plane setting procedure sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information, The image correction procedure corrects the image to be corrected to an orthogonalized image viewed from a direction perpendicular to the orthogonalized reference plane, based on the terminal orientation information and the reference orientation information. A program that causes a computer to perform each of the above steps. (Note 14) The program described in Appendix 13, wherein the terminal orientation information includes information from the gyro sensor of the imaging terminal. (Note 15) The aforementioned image acquisition procedure acquires the shooting preview image in real time as the image to be corrected, The image correction procedure is a program according to Appendix 13 or 14 that corrects the captured preview image to the orthogonalized image in real time. (Note 16) The aforementioned image correction procedure involves cropping a predetermined range from the captured preview image and correcting the cropped image to an orthogonal image. The program described in Appendix 15. (Note 17) The image correction procedure is the program described in Appendix 16, which crops the area containing the text from the captured preview image as the predetermined range. (Note 18) This includes an image correction procedure, a character candidate selection procedure, and a character recognition procedure. The aforementioned image correction procedure is performed by the program described in any of appendices 13 to 17. The character candidate selection procedure involves selecting a character candidate region from the image corrected by the image correction procedure, The character recognition procedure performs character recognition on the image corrected by the image correction procedure, A program that causes a computer to perform each of the above steps. (Note 19) This includes an image acquisition procedure, a terminal information acquisition procedure, a distance information acquisition procedure, a reference orientation information acquisition procedure, a reference plane setting procedure, and an image correction procedure. The aforementioned image acquisition procedure involves acquiring the image to be corrected, The image to be corrected is an image that includes an object, The aforementioned terminal information acquisition procedure acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The aforementioned distance information acquisition procedure acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above procedure for acquiring reference posture information involves acquiring reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The above-mentioned reference plane setting procedure sets a aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information, The image correction procedure corrects the image to be corrected to an orthogonalized image viewed from a direction perpendicular to the orthogonalized reference plane, based on the terminal orientation information and the reference orientation information. A computer-readable recording medium containing a program that causes a computer to perform each of the aforementioned steps. (Note 20) The recording medium described in Appendix 19, wherein the terminal orientation information includes information from the gyro sensor of the imaging terminal. (Note 21) The aforementioned image acquisition procedure acquires the shooting preview image in real time as the image to be corrected, The recording medium described in Appendix 19 or 20 corrects the captured preview image to the orthogonalized image in real time, as described in the image correction procedure. (Note 22) The aforementioned image correction procedure involves cropping a predetermined range from the captured preview image and correcting the cropped image to an orthogonal image. Recording medium as described in Appendix 21. (Note 23) The recording medium described in Appendix 22 is used for the image correction procedure, in which the area containing the characters is cropped from the shooting preview image to the predetermined range. (Note 24) This includes an image correction procedure, a character candidate selection procedure, and a character recognition procedure. The aforementioned image correction procedure is performed by the program described in any of appendices 13 to 17. The character candidate selection procedure involves selecting a character candidate region from the image corrected by the image correction procedure, The character recognition procedure performs character recognition on the image corrected by the image correction procedure, A computer-readable recording medium containing a program that causes a computer to perform each of the aforementioned steps. [Industrial applicability]

[0061] According to the present invention, orthogonalized images can be easily generated. Therefore, the present invention is widely useful in fields that utilize images. [Explanation of Symbols]

[0062] 1. Image Correction Device 2 Image acquisition unit 3. Terminal Information Acquisition Unit 4 Distance information acquisition section 5 Reference attitude information acquisition unit 6 Reference plane setting section 7 Image Correction Unit 101 CPU 102 memory 103 Bus 104 Storage device 105 Input device 106 Output device 107 Communication devices 20 Character recognition device 21 Character Candidate Selection Department 22 Character recognition section 201 CPU 202 memory Bus 203 204 Storage device 205 Input device 206 Output device 207 Communication devices

Claims

1. It includes an image acquisition unit, a terminal information acquisition unit, a distance information acquisition unit, a reference posture information acquisition unit, a reference plane setting unit, and an image correction unit. The image acquisition unit acquires the image to be corrected, The image to be corrected is an image that includes an object, The terminal information acquisition unit acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The distance information acquisition unit acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The aforementioned reference posture information acquisition unit acquires reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The reference plane setting unit sets at least one face of the object as the aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction unit corrects the image to be corrected to an orthogonal image viewed from a direction perpendicular to the orthogonal reference plane, based on the terminal orientation information and the reference orientation information. Image correction device.

2. The image correction device according to claim 1, wherein the terminal orientation information includes information from the gyro sensor of the imaging terminal.

3. The image acquisition unit acquires the shooting preview image in real time as the image to be corrected. The image correction device according to claim 1 or 2, wherein the image correction unit corrects the captured preview image to the aligned image in real time.

4. The image correction unit crops a predetermined range from the shooting preview image and corrects the cropped image into an orthogonal image. The image correction device according to claim 3.

5. The image correction device according to claim 4, wherein the image correction unit crops the area containing characters from the captured preview image to the predetermined range.

6. Including an image correction unit, a character candidate selection unit, and a character recognition unit, The image correction unit is an image correction device according to claim 1 or 2. The character candidate selection unit selects a character candidate region from the image corrected by the image correction unit, The character recognition unit is a character recognition device that performs character recognition from an image corrected by the image correction unit.

7. This process includes an image acquisition step, a terminal information acquisition step, a distance information acquisition step, a reference orientation information acquisition step, a reference plane setting step, and an image correction step. The aforementioned image acquisition step acquires the image to be corrected, The image to be corrected is an image that includes an object, The terminal information acquisition step acquires terminal posture information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The distance information acquisition step acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above-mentioned reference posture information acquisition step acquires reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The aforementioned reference plane setting step sets at least one face of the object as the aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction step corrects the image to be corrected to an orthogonalized image viewed from a direction perpendicular to the orthogonalization reference plane, based on the terminal orientation information and the reference orientation information. An image correction method in which each step is performed by a computer.

8. This process includes an image correction step, a character candidate selection step, and a character recognition step. The image correction step is the image correction method described in claim 7, The character candidate selection step involves selecting a character candidate region from the image corrected by the image correction step, The character recognition step is a character recognition method in which character recognition is performed from an image corrected by the image correction step, with each step being performed by a computer.

9. This includes an image acquisition procedure, a terminal information acquisition procedure, a distance information acquisition procedure, a reference orientation information acquisition procedure, a reference plane setting procedure, and an image correction procedure. The aforementioned image acquisition procedure involves acquiring the image to be corrected, The image to be corrected is an image that includes an object, The aforementioned terminal information acquisition procedure acquires terminal orientation information, The terminal orientation information is information about the orientation of the imaging terminal at the time the image to be corrected is acquired. The aforementioned distance information acquisition procedure acquires object distance information, The object distance information is information about the distance from the imaging terminal to the object, The above procedure for acquiring reference posture information involves acquiring reference posture information, The aforementioned reference posture information is information about the posture of the object at the time the image to be corrected was acquired. The above-mentioned reference plane setting procedure sets at least one face of the object as the aligned reference plane based on the terminal orientation information, the object distance information, and the reference orientation information. The image correction procedure corrects the image to be corrected to an orthogonalized image viewed from a direction perpendicular to the orthogonalization reference plane, based on the terminal orientation information and the reference orientation information. A program that instructs a computer to perform each step.

10. This includes an image correction procedure, a character candidate selection procedure, and a character recognition procedure. The image correction procedure is performed by the program described in claim 9. The character candidate selection procedure involves selecting a character candidate region from the image corrected by the image correction procedure, The character recognition procedure performs character recognition on the image corrected by the image correction procedure, A program that instructs a computer to perform each step.

Images