Image processing device, image processing method, and image processing program

Abstract

The present invention addresses the problem of performing image conversion suitable for character recognition for a query image in which a plurality of regions are captured. An image processing device 1 comprises: a query image acquisition unit 21 that acquires a query image in which a plurality of objects are captured; a normal line inference unit 25 that infers, for each of the plurality of objects captured in the query image, a normal direction indicating the front direction of the object; a representative normal line determination unit 26 that determines a representative normal direction representing the entire query image, on the basis of the plurality of normal directions inferred for the plurality of objects; and a conversion unit 27 that generates a corrected query image by geometrically deforming the query image on the basis of the representative normal direction.

Description

Image processing device, image processing method, and image processing program

[0001] The present disclosure relates to image processing techniques.

[0002] Conventionally, a recognition device has been proposed that includes a vehicle detection unit that cuts out an image surrounding the license plate from an image acquired from an imaging device, a license plate detection unit that extracts a front image of the license plate from the image surrounding the license plate, a character recognition unit that recognizes the vehicle registration number from the front image of the license plate, a reliability determination unit that determines the ability of the license plate detection unit to extract a front image based on the recognition difficulty of the character recognition unit, and a model modification unit that modifies the ability of the license plate detection unit to extract a front image based on the determination result of the reliability determination unit (see Patent Document 1).

[0003] Further, a conventional technique is known for estimating camera displacement between two images of a planar object by decomposing a homography matrix (see Non-Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2020-160814

[0005] Ezio Malis, Manuel Vargas, “Deeper understanding of the homography decomposition for vision-based control”, INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE, September 25, 2007, RR-6303

[0006] In the past, various techniques have been proposed for recognizing predetermined targets (imaged objects, characters, etc.) contained in images, but there is room for improvement in the accuracy of recognizing targets in images. In view of the above-mentioned problems, an object of the present disclosure is to improve the accuracy of recognizing predetermined targets contained in images.

[0007] An example of the present disclosure is an image processing device including: a query image acquisition means for acquiring a query image in which a plurality of objects are captured; a normal estimation means for estimating, for each of the plurality of objects captured in the query image, a normal direction indicating the forward direction of the object; a representative normal determination means for determining a representative normal direction that represents the entire query image based on the plurality of normal directions estimated for the plurality of objects; and a conversion means for generating a corrected query image by geometrically transforming the query image based on the representative normal direction.

[0008] The present disclosure can be understood as an image processing device, a system, a method executed by a computer, or a program executed by a computer. The present disclosure can also be understood as such a program recorded on a recording medium readable by a computer or other device, machine, etc. Here, a recording medium readable by a computer, etc. refers to a recording medium that stores information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer, etc.

[0009] According to the present disclosure, it is possible to improve the accuracy of recognizing a predetermined object contained in an image.

[0010] 1 is a schematic diagram showing the configuration of a system according to an embodiment; FIG. 2 is a diagram showing an outline of the functional configuration of an image processing device according to an embodiment; FIG. 3 is a diagram showing an example of a query image according to an embodiment; FIG. 4 is a diagram showing an example of a query image according to an embodiment in which geometric transformation according to a conventional method has been performed on the query image; FIG. 5 is a flowchart showing the flow of image transformation and character recognition processing according to an embodiment; FIG. 6 is a diagram showing an example of normal vectors estimated for each of two objects relating to different orientations according to an embodiment; FIG. 7 is a diagram showing examples of minimum ratio normal vectors and maximum ratio normal vectors estimated for each of a plurality of objects in a query image according to an embodiment; FIG. 8 is a diagram showing a concept of calculating a representative normal vector based on a plurality of minimum ratio normal vectors and a plurality of maximum ratio normal vectors estimated for each of two objects relating to different orientations according to an embodiment; and FIG. 9 is a diagram showing an example of a query image according to an embodiment in which geometric transformation based on a representative normal vector has been performed on the query image according to an embodiment.

[0011] Hereinafter, embodiments of an image processing device, method, and program according to the present disclosure will be described with reference to the drawings. However, the embodiments described below are merely examples, and the image processing device, method, and program according to the present disclosure are not limited to the specific configurations described below. In implementing the present disclosure, specific configurations according to the embodiment may be appropriately adopted, and various improvements and modifications may be made.

[0012] Conventionally, character recognition is performed after performing geometric transformation to orient a region to be subjected to character recognition in an image acquired by imaging or the like, so that the region is oriented to face forward. However, if an image contains multiple regions to be subjected to character recognition, and the writing surfaces (fronts) of the characters to be recognized in these multiple regions face in different directions, or if the projections differ depending on the positional relationship with the camera used for imaging, even if geometric transformation is performed to orient any of the regions to face forward, the region oriented to face forward becomes suitable for character recognition, but the other regions do not become suitable for character recognition. In other words, conventional techniques have room for improvement in terms of improving recognition accuracy when multiple regions to be subjected to character recognition are located apart within the same image. In view of the above-mentioned problems, the present disclosure aims to perform image transformation suitable for character recognition on a query image in which multiple regions are captured.

[0013] In this embodiment, the technology according to the present disclosure will be described in terms of an embodiment in which a query image in which multiple objects are captured is subjected to image conversion suitable for character recognition, and then character recognition processing is performed on the converted image. However, the technology according to the present disclosure can be widely used to perform image conversion suitable for character recognition on a query image in which multiple regions are captured, and the application of the present disclosure is not limited to the examples shown in the embodiments. For example, the object may not be a tangible object, but may be an object drawn in a virtual space or an object drawn using drawing software. Furthermore, for example, the image may not be an image of the real world, but may be an image of a virtual world or an image drawn using drawing software.

[0014] 1 is a schematic diagram showing the configuration of a system according to this embodiment. The system according to this embodiment includes an image processing device 1, an imaging device 81, and a user terminal 9, which are connected to a network and can communicate with each other.

[0015] The image processing device 1 is a computer including a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage device 14 such as an EEPROM (Electrically Erasable and Programmable Read Only Memory) or an HDD (Hard Disk Drive), and a communication unit 15 such as a NIC (Network Interface Card). However, the specific hardware configuration of the image processing device 1 can be omitted, replaced, or added as appropriate depending on the embodiment. Furthermore, the image processing device 1 is not limited to a device consisting of a single housing. The image processing device 1 may be realized by a plurality of devices using so-called cloud or distributed computing technology.

[0016] The imaging device 81 captures an image of a target to obtain a query image, which will be described later. The imaging device 81 may be a general digital camera or any other device capable of recording light incident from a target, and the specific configuration thereof is not limited.

[0017] The user terminal 9 is a terminal device used by a user. The user terminal 9 is a computer equipped with a CPU, ROM, RAM, a storage device, a communication unit, an input device, an output device, etc. (not shown). However, the specific hardware configuration of the user terminal 9 can be omitted, replaced, or added as appropriate depending on the embodiment. Furthermore, the user terminal 9 is not limited to a device consisting of a single housing. The user terminal 9 may be realized by multiple devices using so-called cloud or distributed computing technology, etc. Users use various services provided by the system according to this embodiment via these user terminals 9.

[0018] 2 is a diagram showing an outline of the functional configuration of the image processing device 1 according to this embodiment. The image processing device 1 functions as an image processing device including a query image acquisition unit 21, an object detection unit 22, a shape data acquisition unit 23, a homography matrix calculation unit 24, a normal estimation unit 25, a representative normal determination unit 26, a conversion unit 27, and a character recognition unit 28, by a program recorded in the storage device 14 being read into the RAM 13 and executed by the CPU 11, which controls each piece of hardware included in the image processing device 1. Note that in this embodiment and other embodiments described below, each function included in the image processing device 1 is executed by the CPU 11, which is a general-purpose processor, but some or all of these functions may be executed by one or more dedicated processors.

[0019] The query image acquisition unit 21 acquires a query image in which multiple objects to be subjected to image processing and / or character recognition are simultaneously captured. Although the method for acquiring the query image is not limited, in this embodiment, an example will be described in which a query image captured using the imaging device 81 is acquired via the user terminal 9.

[0020] 3 is a diagram showing an example of a query image in this embodiment. This diagram shows a query image that captures multiple smartphones placed on tables in a store with product guides displayed on their touch-panel displays, and explanatory media (hereinafter simply referred to as "product information cards") that contain information such as descriptions and prices for each of the smartphones and are placed near the smartphones. A staff member takes a photo of multiple smartphones and product information cards, which are the objects to be subjected to image processing and character recognition in this embodiment, in a location where these objects are present, so that they are included in the same image, and this photo is used as the query image.

[0021] FIG. 4 shows an example of a conventional geometric transformation performed on a query image in this embodiment. According to this figure, for example, if image transformation is performed so that the product information card in the center of the query image faces the camera, the transformation results in a state suitable for character recognition for the central product information card facing the camera. However, other objects, particularly smartphones at the edges of the screen whose front direction (since the object is the target of character recognition in this embodiment, the front direction is defined as the normal direction of the writing surface of the character to be recognized displayed on the object) differ significantly from the central product information card, resulting in increased distortion and reduced character recognition accuracy. Furthermore, before image transformation, the smartphone and its corresponding product information card are positioned vertically side by side (see FIG. 3 ). However, after image transformation, the smartphone and its corresponding product information card are misaligned vertically, making it difficult to associate the smartphone with the product information card.

[0022] To solve the above problem, it is conceivable to perform image transformation on each of the multiple objects included in the query image so that the object faces forward and then perform character recognition on the object. However, if such a method is adopted, the number of image transformations and character recognition processes required will be the same as the number of objects in the query image. For this reason, the system according to this embodiment is designed to increase the accuracy of character recognition while reducing the number of image transformations and character recognition processes performed on one query image.

[0023] The object detection unit 22 detects multiple object images from the query image. In this embodiment, the object detection unit 22 detects multiple objects captured in the query image using a machine learning model that, in response to an input of the type or attribute of an object to be subjected to character recognition, outputs the position or range in the query image in which an object related to the type or attribute is captured. However, the type of object detection technology used for object detection from an image is not limited, and any currently known object detection technology or any object detection technology developed in the future may be used.

[0024] The shape data acquisition unit 23 acquires shape data relating to the front shape of an object. For example, the shape data may be the aspect ratio, size, design data, front image, etc. of the object stored in advance in the RAM 13 or the storage device 14. Furthermore, the shape data may be data that uniquely determines at least the front shape of objects of the same type or attribute.

[0025] However, even for objects of the same type or attribute, the front-view shape of the object may not be uniquely determined depending on the specified type or attribute. For example, if an object is specified as a "smartphone manufactured by Company X, model number XX-XX" or a "B8-size card," its shape can be uniquely determined. However, if the object is simply specified as a "smartphone," a "smartphone manufactured by Company X," or a "product information card," its shape may vary. Therefore, in this embodiment, the shape data may be shape data with a range for objects of the same type or attribute. Shape data with a range may be, for example, a range of aspect ratios, a range of sizes, or multiple front images of the object. In this embodiment, an example in which an aspect ratio range of an object is used as shape data will be described.

[0026] The homography matrix calculation unit 24 calculates, for each of the plurality of objects, a homography matrix between the front shape of the object specified in accordance with the shape data and the object image of the object in the query image. Here, when the shape data has a range as described above, the homography matrix calculation unit 24 calculates, for each of the plurality of objects, a homography matrix associated with the minimum value and a homography matrix associated with the maximum value in the range of the shape data.

[0027] The normal estimator 25 estimates a normal direction indicating the front direction of each of the plurality of objects captured in the query image. In this embodiment, the normal estimator 25 estimates the normal direction (normal vector) of each of the plurality of objects based on the homography matrix calculated by the homography matrix calculator 24, thereby estimating the normal direction of the object in accordance with the comparison result between the front shape of the object specified in accordance with the shape data and the object image in the query image.

[0028] However, a specific method for estimating the normal direction of an object is not limited to calculating a normal vector from a homography matrix. Furthermore, when the shape data has a range as described above, the normal estimation unit 25 calculates, for each of the multiple objects, a normal direction associated with the minimum value and a normal direction associated with the maximum value in the range of the shape data.

[0029] The representative normal determination unit 26 determines a representative normal direction representing the entire query image based on multiple normal directions estimated for multiple objects. In this embodiment, the representative normal determination unit 26 determines the average value of multiple normal vectors estimated for multiple objects as a representative normal vector and determines the direction of the representative normal vector as a representative normal direction. However, the representative normal direction may be obtained by a method that obtains a representative direction using a statistical method, and may be calculated by a method other than calculating the average value of normal vectors. Furthermore, when the shape data has a range as described above, the representative normal determination unit 26 determines, as the representative normal direction, a normal direction that represents multiple normal directions associated with minimum values ​​in the range of shape data estimated for the multiple objects and multiple normal directions associated with maximum values ​​in the range of shape data estimated for each of the multiple objects.

[0030] The transformation unit 27 generates a corrected query image by geometrically transforming the query image based on the representative normal direction. In this embodiment, the transformation unit 27 generates the corrected query image by performing projective transformation on the query image so that the representative normal direction is the front direction of the transformed image.

[0031] The character recognition unit 28 recognizes characters written in an object to be subjected to character recognition by performing character recognition processing on the corrected query image. The technology disclosed herein aims to provide image conversion suitable for character recognition, and the specific algorithm used for character recognition is not limited. Any character recognition technology currently used or to be developed in the future may be used for character recognition.

[0032] <Processing Flow> Next, the processing flow executed by the image processing device according to this embodiment will be described. Note that the specific content and processing order of the processing described below are an example for implementing the present disclosure. The specific processing content and processing order may be selected as appropriate depending on the embodiment of the present disclosure.

[0033] 5 is a flowchart showing the flow of image conversion and character recognition processing according to this embodiment. The processing shown in this flowchart is executed when a user's instruction to start the processing is accepted.

[0034] In step S101, a query image is acquired. The worker captures an image of an object using the imaging device 81 and inputs image data of the acquired query image into the image processing device 1. In this embodiment, the imaged object is a table in a store on which a smartphone and a product information card are placed as predetermined objects. The imaging method and the method of inputting image data into the image processing device 1 are not limited. However, in this embodiment, the object is captured using the imaging device 81, and the image data transferred from the imaging device 81 to the user terminal 9 via communication or a recording medium is further transferred to the image processing device 1 via a network, thereby inputting the image data of the query image into the image processing device 1. When the query image is acquired by the query image acquisition unit 21, the process proceeds to step S102.

[0035] In step S102, an object to be subjected to character recognition is designated. The object detection unit 22 acquires object designation information input by the operator, which indicates the type or attributes of an object to be subjected to character recognition among the objects captured in the query image. In this embodiment, an example is described in which, among the objects captured in the query image of a table in a store, a smartphone displayed as a commodity and a product information card for the smartphone are subjected to character recognition. Therefore, in this embodiment, for example, "smartphone" and "tag" are input as object identification information. Note that in this embodiment, an example is described in which the object designation information is acquired as text data entered by the user via a UI prompt. However, the method for acquiring the object identification information is not limited to the example described in this embodiment. For example, the object identification information may be acquired in a format other than text data (e.g., a code indicating the type or attributes of the object), or the object identification information may be acquired by being set in the system in advance before the query image is acquired. Then, the process proceeds to step S103.

[0036] In steps S103 and S104, an object is detected from the query image, and the area in which the object is captured in the query image is cut out as an object image. The object detection unit 22 detects, from the query image acquired in step S101, multiple objects of the type or attribute specified in the object designation information acquired in step S102 (step S103). It is preferable that all objects of the type or attribute specified in the object designation information are detected from the objects captured in the query image. In this embodiment, a zero-shot object detector using a convolutional neural network (CNN) may be used to detect objects from the query image. However, the type of algorithm used to detect objects from an image is not limited, and any currently known algorithm or any algorithm developed in the future may be used.

[0037] When multiple objects are detected from the query image, the object detection unit 22 extracts, from the query image, the range in which the objects are captured, as an object image for each of the detected objects (step S104). However, image data for the object image does not need to be newly generated; it is sufficient that the range of the object image in the query image is identified. Then, the process proceeds to step S105.

[0038] In step S105, the aspect ratio (width-to-height ratio) of the object is set. For each of the multiple objects detected in step S103, the shape data acquisition unit 23 sets an aspect ratio according to the type or attribute of the object. Specifically, the shape data acquisition unit 23 selects an aspect ratio related to a part or the entire configuration of the object when viewed from the front (when viewed in a plan view), which is pre-stored for each type or attribute of the object, based on the type or attribute of the detected object, and sets the selected aspect ratio as the aspect ratio of the object. Here, the type or attribute of the object can be identified based on the object designation information input in step S102 or the output obtained from the object detector in step S103. In this embodiment, for the object "smartphone" detected from the query image, the aspect ratio pre-stored as the aspect ratio of the smartphone is set, and for the object "product information card," the aspect ratio pre-stored as the aspect ratio of the product information card is set.

[0039] The aspect ratio set here may have a range. For example, the aspect ratio of smartphones varies depending on the model. For this reason, in this embodiment, the aspect ratio of a smartphone is set within a range from a minimum value to a maximum value, such as "between 0.479 and 0.486." Once the aspect ratio (width-to-height ratio) of the object is set, the process proceeds to step S106.

[0040] In steps S106 and S107, the homography matrix and normal vector of the object are calculated. The homography matrix calculation unit 24 calculates a homography matrix for each of the multiple objects detected in step S103 using the aspect ratio range set in step S105. First, for each object, the homography matrix calculation unit 24 detects specific components of the object from the object image and matches them with the specific components related to the set aspect ratio. In this embodiment, the objects are smartphones and product information cards, both of which generally have an approximately rectangular shape. Therefore, the homography matrix calculation unit 24 detects the vertices (corners) of the rectangle related to the object and matches the detected vertices with the vertices of the rectangle related to the set aspect ratio.

[0041] Here, the means for detecting predetermined components of an object (in this embodiment, the vertices of a rectangle) from the object image may be, for example, an image analysis technique for detecting feature points in an image. However, the type of image analysis technique used for detecting predetermined components from an object image is not limited, and any image analysis technique that is currently known or that will be developed in the future may be used.

[0042] Once the matching of the specified components detected from the object with the specified components related to the set aspect ratio is completed, the homography matrix calculation unit 24 calculates the homography matrix of the object by comparing the rectangle of the object in the matched object image with the set aspect ratio (step S106).

[0043] Once the homography matrix has been calculated, the normal estimation unit 25 obtains the normal vector of each of the multiple object images detected in step S103 by decomposing the homography matrix obtained in step S106 (step S107. For details on the method for estimating the normal vectors of two planar objects by decomposing the homography matrix, see Non-Patent Document 1).

[0044] FIG. 6 shows the normal vectors N estimated for two objects A and B in different orientations in this embodiment. Ａ and N Ｂ 10 is a diagram illustrating an example of a normal vector calculated in this embodiment, which is a normal vector indicating the front direction of the object, because the aspect ratio used in calculating the homography matrix is ​​the aspect ratio when the object is viewed from the front. Furthermore, since the normal vector calculated in this embodiment is calculated based on an aspect ratio having a range, it is a normal vector having a range from a normal vector calculated based on the minimum aspect ratio (hereinafter referred to as a "minimum ratio normal vector") to a normal vector calculated based on the maximum aspect ratio (hereinafter referred to as a "maximum ratio normal vector").

[0045] 7 is a diagram showing an example of minimum ratio normal vectors (shown by dashed lines) and maximum ratio normal vectors (shown by solid lines) estimated for each of a plurality of objects in a query image in this embodiment. Once the normal vectors for the detected plurality of objects have been calculated, the process proceeds to step S108.

[0046] In step S108, a representative normal vector is determined. Based on the multiple normal vectors calculated for the multiple objects in step S107, the representative normal determination unit 26 determines a representative normal vector that represents the query image including these objects. In this embodiment, the representative normal determination unit 26 determines the average of the normal vectors calculated for each of the multiple objects as the representative normal vector. However, the method for determining the representative normal vector is not limited to calculating the average value, and a method of obtaining a representative vector using a statistical method may also be adopted.

[0047] FIG. 8 shows a plurality of minimum ratio normal vectors N estimated for two objects A and B having different orientations in this embodiment. Ａ１ and N Ｂ１ and a plurality of maximum ratio normal vectors N Ａ２ and N Ｂ２As described above, the normal vectors calculated in step S107 are in the range from the minimum ratio normal vector to the maximum ratio normal vector (for object A in the figure, the minimum ratio normal vector N is Ａ１ from the maximum ratio normal vector N Ａ２ For object B in the figure, the minimum ratio normal vector N Ｂ１ from the maximum ratio normal vector N Ｂ２ . ) is a normal vector having a range of . Therefore, the representative normal determination unit 26 calculates a representative value of a plurality of minimum ratio normal vectors (N1 in the figure, hereinafter referred to as the "minimum ratio representative normal vector") and a representative value of a plurality of maximum ratio normal vectors (N2 in the figure, hereinafter referred to as the "maximum ratio representative normal vector"), and determines the representative normal vector by calculating a representative value (N in the figure, for example, an average value) of the minimum ratio representative normal vector and the maximum ratio representative normal vector. Once the representative normal vector has been determined, the process proceeds to step S109.

[0048] In step S109, the query image is projectively transformed based on the representative normal vector. The transformation unit 27 performs projective transformation (perspective correction using homography transformation in this embodiment) on the query image so that the representative normal vector determined in step S108 faces forward, thereby obtaining a corrected query image. In other words, based on the representative normal vector, the transformation unit 27 calculates a homography matrix such that the X and Y axis components of the representative normal vector in the corrected query image after projective transformation are zero (i.e., only the Z axis component) when the corrected query image after projective transformation is set to the XY plane, and then performs homography transformation on the query image using the homography matrix.

[0049] FIG. 9 is a diagram showing an example of a query image subjected to geometric transformation based on a representative normal vector in this embodiment. As can be seen from this diagram, by performing geometric transformation based on a representative normal vector representing multiple objects in the query image, the query image as a whole, including multiple objects, is in a state suitable for character recognition, and the accuracy of character recognition for the entire query image is likely to be improved. It can also be seen that even after image transformation, the vertical positional relationship between the smartphone and the product information card corresponding to the smartphone is generally maintained. Once projective transformation of the query image based on the representative normal vector is complete, processing proceeds to step S110.

[0050] In step S110, characters captured in the corrected query image are recognized. The character recognition unit 28 recognizes the characters captured in the corrected query image by performing optical character recognition (OCR) processing on the corrected query image obtained in step S109. The character recognition unit 28 then outputs the character recognition result, and the processing shown in this flowchart ends. According to the processing shown in this flowchart, by having the above-described processing flow, it is possible to improve the recognition accuracy of characters written on multiple objects captured in the query image in one image conversion process for the query image and one character recognition process.

[0051] <Effects> According to the image processing device, method, and program of this embodiment, it is possible to perform image conversion suitable for character recognition on a query image in which multiple regions are captured, and ultimately to improve the accuracy of character recognition on a query image in which multiple regions are captured.

[0052] While the above-described embodiment has been described with reference to an example of processing when the shape data has a range, as mentioned above, the shape data may be data that uniquely determines at least the shape of the front of an object of the same type or attribute. When such shape data is used, calculation of the minimum ratio normal vector and the maximum ratio normal vector for each object is omitted, and a representative normal vector is determined based on the normal vector obtained for each of the multiple objects.

[0053] 1. Image processing device

Claims

1. An image processing device comprising: a query image acquisition means for acquiring a query image in which a plurality of objects are captured; a normal estimation means for estimating a normal direction indicating a forward direction of each of the plurality of objects captured in the query image; a representative normal determination means for determining a representative normal direction representative of the entire query image based on a plurality of normal directions estimated for the plurality of objects; and a conversion means for generating a corrected query image by geometrically transforming the query image based on the representative normal direction.

2. The image processing device of claim 1, further comprising a shape data acquisition means for acquiring shape data relating to a front shape of the object, wherein the normal estimation means estimates the normal direction of each of the plurality of objects according to a comparison result between the front shape of the object identified according to the shape data and an object image in the query image.

3. The image processing device of claim 2, wherein the shape data is shape data having a range for objects of the same type or attribute, the normal estimation means calculates the normal direction associated with the minimum value and the normal direction associated with the maximum value in the range of the shape data for each of the multiple objects, and the representative normal determination means determines a normal direction representative of the multiple normal directions associated with the minimum value in the range of the shape data estimated for the multiple objects and the multiple normal directions associated with the maximum value in the range of the shape data estimated for each of the multiple objects as the representative normal direction.

4. The image processing device of claim 2, further comprising a homography matrix calculation means for calculating, for each of the plurality of objects, a homography matrix between the front shape of the object specified in accordance with the shape data and an object image in the query image, wherein the normal estimation means estimates, for each of the plurality of objects, a normal direction of the object based on the homography matrix calculated by the homography matrix calculation means.

5. The image processing device of claim 4, wherein the shape data is shape data having a range for objects of the same type or attribute, the homography matrix calculation means calculates, for each of the plurality of objects, the homography matrix relating to the minimum value and the homography matrix relating to the maximum value in the range of the shape data, the normal estimation means calculates, for each of the plurality of objects, the normal direction relating to the minimum value and the normal direction relating to the maximum value in the range of the shape data, and the representative normal determination means determines, as the representative normal direction, a normal direction representative of the multiple normal directions relating to the minimum value in the range of the shape data estimated for the plurality of objects and the multiple normal directions relating to the maximum value in the range of the shape data estimated for each of the plurality of objects.

6. The image processing device according to claim 3 or 5, wherein the shape data includes an aspect ratio of the object stored in advance, and the range is a range of aspect ratios.

7. The image processing device of claim 1, further comprising: an object detection means for detecting a plurality of objects captured in a query image using a machine learning model that, in response to an input of a type or attribute of an object to be subjected to character recognition, outputs a position or range in the query image in which an object related to the type or attribute is captured; and a character recognition means for recognizing characters written on the object to be subjected to character recognition by performing character recognition processing on the corrected query image.

8. The image processing device of claim 1, wherein the normal estimation means estimates a normal vector indicating the forward direction of each of a plurality of objects captured in the query image, and the representative normal determination means determines the average of the plurality of normal vectors estimated for the plurality of objects as the representative normal direction.

9. The image processing device according to claim 1, wherein the transformation means performs projective transformation on the query image so that the representative normal direction coincides with the front direction of the transformed image.

10. An image processing method in which a computer executes the following steps: a query image acquisition step of acquiring a query image in which a plurality of objects are captured; a normal estimation step of estimating a normal direction indicating the forward direction of each of the plurality of objects captured in the query image; a representative normal determination step of determining a representative normal direction representative of the entire query image based on a plurality of normal directions estimated for the plurality of objects; and a transformation step of generating a corrected query image by geometrically transforming the query image based on the representative normal direction.

11. An image processing program that causes a computer to function as: a query image acquisition means for acquiring a query image in which a plurality of objects are captured; a normal estimation means for estimating a normal direction indicating the forward direction of each of the plurality of objects captured in the query image; a representative normal determination means for determining a representative normal direction representative of the entire query image based on a plurality of normal directions estimated for the plurality of objects; and a conversion means for generating a corrected query image by geometrically transforming the query image based on the representative normal direction.

Images