Information processing apparatus, information processing method, and information processing program

Abstract

To accurately estimate a rotation angle of an image including a character string.SOLUTION: An information processing apparatus according to the present application includes an extraction unit that extracts, from an image including a character string, character string area information regarding a character string area that is a rectangular area indicating a position of the character string by a machine learning model, a calculation unit that calculates, for each character string area, an inclination of the character string area with respect to a reference direction based on the character string area information, a determination unit that determines, for each character string area, whether an orientation of the character string located in the character string area is in an upright state or an inverted state, and an estimation unit that estimates, for each character string, a rotation angle of the character string based on a calculation result of the calculation unit and a determination result of the determination unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, OCR (Optical Character Recognition) technology has been known for recognizing characters contained in images. In recent years, technology related to AI-OCR, which is OCR using deep learning, has become known. However, it is known that if the character string contained in the image is rotated on the medium, or if the character string is facing the medium but the medium itself is rotated, it becomes difficult for OCR to recognize the character string.

[0003] In response to this, a technique for estimating the rotation angle of a character string included in an image is known. For example, a technique related to a learning model is known in which, when a rotated character image, which is an image showing rotated characters, which are characters rotated at a predetermined angle, is input to an input layer, a recognition rate of the characters included in the rotated character image is calculated through calculations in an intermediate layer, and the highest recognition rate, character information corresponding to the highest recognition rate, and rotation angle information indicating the predetermined angle are output from an output layer. [Prior art documents] [Non-patent literature]

[0004] [Patent Document 1] Patent Publication No. 2023-85076 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned conventional technology simply uses a learning model to calculate the recognition rate of characters contained in a rotated character image, and outputs the highest recognition rate value, character information corresponding to the highest recognition rate value, and rotation angle information indicating a predetermined angle, so it is not necessarily possible to accurately estimate the rotation angle of an image containing a character string.

[0006] An object of the present application is to provide an information processing device, an information processing method, and an information processing program that can accurately estimate the rotation angle of an image that includes a character string. [Means for solving the problem]

[0007] The information processing device of the present application includes an extraction unit that extracts, from an image including character strings, character string area information related to character string areas, which are rectangular areas indicating the position of the character strings, using a machine learning model; a calculation unit that calculates, for each character string area, the inclination of the character string area with respect to a reference direction based on the character string area information; a determination unit that determines, for each character string area, whether the orientation of the character string located within the character string area is upright or inverted; and an estimation unit that estimates, for each character string, the rotation angle of the character string based on the calculation result of the calculation unit and the determination result of the determination unit. [Effects of the Invention]

[0008] According to one aspect of the embodiment, it is possible to accurately estimate the rotation angle of an image including a character string. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. [Figure 4]FIG. 4 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. [Figure 5] FIG. 5 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted. Furthermore, hereinafter, AI (artificial intelligence) and machine learning model will be treated as synonymous terms.

[0011] (Embodiment) 1. Introduction Conventionally, OCR (Optical Character Recognition) technology has been known for recognizing characters contained in images. In recent years, AI-OCR technology, which is OCR using deep learning, has become known. For example, in identity verification for various services, AI-OCR technology is used to extract character strings contained in an image of an identification card submitted by the user, and then authenticate the user.

[0012] However, AI-OCR is not a panacea. Specifically, most machine learning models have low robustness against character string rotation. As a result, even a slight tilt of the character string can result in misrecognition or a significant drop in recognition accuracy. Here, tilted character strings include both cases where the character string contained in the image is rotated on the medium, and cases where the character string is facing the medium but the medium itself is rotated.

[0013] In response to this, the inventors of the present invention recognized that there is a limit to improving the accuracy of AI-OCR itself. Furthermore, the inventors of the present invention focused on improving the accuracy of AI-OCR character string recognition by accurately estimating the rotation angle of an image containing character strings and rotating the image based on the accurately estimated rotation angle. Based on this recognition of the problem, the present invention relates to a method for accurately estimating the rotation angle of each character string from an image containing character strings, such as on an identification card.

[0014] Specifically, the information processing device 100 according to the embodiment improves the rotation robustness of an open-source AI that extracts character string region information related to a character string region (e.g., a bounding box), which is a rectangular region indicating the position of the character string, from an image containing the character string. The AI ​​is pre-trained by providing training data including multiple images containing character strings tilted at multiple different rotation angles. For example, the open-source AI may be CRAFT (Non-Patent Document 1; Youngmin Baek et al., "Character region awareness for text detection," [online], [searched July 13, 2024], 2019, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, https: / / arxiv.org / pdf / 1904.01941). Furthermore, the information processing device 100 calculates the tilt and aspect ratio of the character string region relative to a reference direction based on the character string region information output by the AI ​​with improved rotation robustness, and accurately estimates the rotation angle of each character string based on the tilt and aspect ratio of the character string region relative to the reference direction. This allows the information processing device 100 to accurately estimate the rotation angle of an image including a character string, based on the average of the rotation angles of a plurality of character strings estimated with high accuracy.

[0015] 2. Configuration of Information Processing Device An example of the configuration of an information processing device 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0016] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC). The communication unit 110 is connected to various networks via wired or wireless connections, and transmits and receives information to and from external information processing devices (for example, terminal devices used by users).

[0017] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various data. The storage unit 120 also stores various programs. For example, the storage unit 120 stores an information processing program according to the embodiment.

[0018] (control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in a storage device inside the information processing device 100 using RAM as a work area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0019] The control unit 130 has an extraction unit 131, a calculation unit 132, a rotation correction unit 133, a determination unit 134, an estimation unit 135, and a provision unit 136 as functional units, and may realize or execute the information processing actions described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 1, and may have other configurations as long as they perform the information processing described below. Furthermore, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0020] (Extraction part 131) The extraction unit 131 generates an image including a character string. For example, the extraction unit 131 acquires an image of a medium on which a character string is written from a user's terminal device via the communication unit 110. For example, the medium may be a card such as an identification card or a document. Next, the extraction unit 131 generates an image including the character string based on the image of the medium on which the character string is written. FIG. 2 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. On the left side of FIG. 2, the extraction unit 131 acquires an image D1 of a medium on which horizontally written character strings T11 to T13, "text box," are written. Next, the extraction unit 131 generates an image G1 including the character strings T11 to T13 based on the medium image D1. Specifically, the extraction unit 131 generates an image G1 indicating the position of the medium image D1 in a two-dimensional Cartesian coordinate system (e.g., an XY coordinate system). For example, the extraction unit 131 generates an image G1 indicating the position of the medium image D1 in a plane including an X axis indicating a reference direction and a Y axis perpendicular to the X axis. For example, the extraction unit 131 generates an image G1 in which one pair of opposite sides of the rectangular image G1 are parallel to the X-axis and the other opposite side of the image G1 is parallel to the Y-axis. For example, the extraction unit 131 generates an image D1 in which one corner of the rectangular image G1 corresponds to the origin O of the XY coordinate system. In FIG. 2, for ease of viewing, the lower left corner of the rectangular image G1 is shown slightly offset from the origin O of the XY coordinate system. In reality, the extraction unit 131 generates an image G1 in which the lower left corner of the rectangular image G1 corresponds to the origin O of the XY coordinate system.

[0021] Furthermore, the extraction unit 131 extracts, from an image including a character string, character string region information related to a character string region, which is a rectangular region indicating the position of the character string, using a machine learning model. For example, the character string region is a bounding box surrounding the character string. The character string region information includes the character string region itself. The character string region information also includes coordinate information indicating the positions of each of the points corresponding to the rectangle of the character string region. Specifically, when an image including a character string is input, the extraction unit 131 extracts character string region information corresponding to the image to be processed from the image to be processed using a machine learning model M1 that has been trained to output character string region information. More specifically, the extraction unit 131 inputs the image to be processed to the machine learning model M1 and obtains the character string region information output from the machine learning model M1.

[0022] Furthermore, when a plurality of images including character strings tilted at a plurality of different rotation angles are input, extraction unit 131 extracts character string region information using a machine learning model trained to output character string region information regarding character string regions corresponding to each of the character strings tilted at a plurality of different rotation angles. Specifically, when an image including character strings tilted counterclockwise at a rotation angle θ (θ is any angle between 0°≦θ<360°) with respect to a reference direction is input, extraction unit 131 extracts character string region information corresponding to the image to be processed from the image to be processed using machine learning model M1 trained to output character string region information regarding character string regions corresponding to the character strings tilted at the rotation angle θ.

[0023] In FIG. 2, the extraction unit 131 inputs the generated image G1 to the machine learning model M1 and extracts, from the image G1, character string region information for each of character string regions B11 to B13, which indicates the position of each of character strings T11 to T13 included in the image G1. For example, the extraction unit 131 extracts each of the character string regions B11 to B13 as the character string region information. Furthermore, the extraction unit 131 extracts coordinate information indicating the position of each square point of the character string regions B11 to B13 as the character string region information. For example, the extraction unit 131 extracts the character string region B11 itself as the character string region information for the character string region B11. Furthermore, the extraction unit 131 extracts XY coordinate information indicating the position of each square point P11 to P14 of the character string region B11 as the character string region information for the character string region B11.

[0024] (Calculation section 132) Calculation unit 132 excludes from the calculation any character string region whose width is zero based on the character string region information extracted by extraction unit 131. Calculation unit 132 excludes linear character string regions based on the character string region information. For example, calculation unit 132 excludes from the calculation any character string region for which XY coordinate information indicating the positions of four points corresponding to a square in the character string region was not extracted by extraction unit 131. In the following, calculation unit 132 performs calculations on character string regions whose width is not zero.

[0025] Calculation unit 132 calculates the inclination of each character string region with respect to the reference direction based on the character string region information. Specifically, calculation unit 132 calculates the inclination of the direction of at least one side of the character string region with respect to the reference direction as the inclination of the character string region with respect to the reference direction. Calculation unit 132 calculates the inclination of the direction of at least one side of the character string region with respect to the reference direction based on the character string region information for each character string region.

[0026] For example, calculation unit 132 calculates the inclination of the direction of at least one side of the character string region with respect to a reference direction based on coordinate information indicating the positions of points corresponding to the squares of the character string region extracted by extraction unit 131. For example, the reference direction is the same direction as the X-axis direction. Note that the reference direction may also be the same direction as the Y-axis direction. For example, calculation unit 132 calculates a first value, which is a value obtained by subtracting the Y-coordinate value indicating the position of a point corresponding to one end of at least one side of the character string region from the Y-coordinate value indicating the position of a point corresponding to the other end of the character string region. Calculation unit 132 also calculates a second value, which is a value obtained by subtracting the X-coordinate value indicating the position of a point corresponding to one end of at least one side of the character string region from the X-coordinate value indicating the position of a point corresponding to the other end of the character string region. Calculation unit 132 also calculates a value obtained by dividing the first value by the second value as the inclination of the direction of at least one side of the character string region with respect to the reference direction. Hereinafter, the inclination of the direction of at least one side of the character string region with respect to the reference direction may also be referred to as the inclination of the character string region with respect to the reference direction.

[0027] 2, the calculation unit 132 calculates the inclination of the side of the character string region B11, whose ends are points P11 and P12, with respect to the X-axis direction. The calculation unit 132 calculates the inclination of the side of the character string region B11, whose ends are points P11 and P12, with respect to the X-axis direction, based on the coordinates (x1, y1) indicating the position of point P11 and the coordinates (x2, y2) indicating the position of point P12, which correspond to the square of the character string region B11. For example, the calculation unit 132 calculates a first value (y2-y1) which is the value obtained by subtracting the Y-coordinate value y1 of point P11 from the Y-coordinate value y2 of point P12. The calculation unit 132 also calculates a second value (x2-x1) which is the value obtained by subtracting the X-coordinate value x1 of point P11 from the X-coordinate value x2 of point P12. Furthermore, the calculation unit 132 calculates the value obtained by dividing the first value (y2-y1) by the second value (x2-x1) as the inclination of the side having the points P11 and P12 as its two ends with respect to the X-axis direction.

[0028] Note that the calculation unit 132 may calculate the inclination of the direction of any side of the character string region with respect to the reference direction, as long as the inclination is the inclination of the direction of at least one side of the character string region with respect to the reference direction. For example, in FIG. 2, the calculation unit 132 may calculate the inclination of the side of the character string region B11 with points P13 and P14 as its both ends with respect to the X-axis direction, instead of calculating the inclination of the side with points P11 and P12 as its both ends with respect to the X-axis direction. Furthermore, the calculation unit 132 may calculate the inclination of the side with points P11 and P14 as its both ends with respect to the X-axis direction, instead of calculating the inclination of the side with points P11 and P14 as its both ends with respect to the X-axis direction. Furthermore, the calculation unit 132 may calculate the inclination of the side with points P12 and P13 as its both ends with respect to the X-axis direction, instead of calculating the inclination of the side with points P11 and P14 as its both ends with respect to the X-axis direction.

[0029] Furthermore, calculation unit 132 calculates the aspect ratio of each character string region based on the character string region information extracted by extraction unit 131. The aspect ratio can also be referred to as the ratio of width to height. Specifically, calculation unit 132 calculates the aspect ratio of each character string region based on coordinate information indicating the positions of the square points of the character string region extracted by extraction unit 131. More specifically, calculation unit 132 calculates the aspect ratio of the character string region by dividing the length of at least one side of the character string region by the length of the side intersecting with that side. Note that the shape of the character string region is not limited to a geometrically defined rectangular shape (rectangle). Specifically, the shape of the character string region also includes shapes with an allowable degree of difference (error / distortion) and shapes similar to such shapes. For example, the shape of the character string region may be a trapezoid. For example, calculation unit 132 calculates the magnitude of the inclination of the direction of at least one side of the character string region with respect to a reference direction and the magnitude of the inclination of the direction of the side intersecting with that side with respect to the reference direction. Next, the calculation unit 132 identifies the side corresponding to the direction with the smaller tilt with respect to the reference direction based on a comparison of the magnitudes of tilt. The calculation unit 132 may calculate the aspect ratio of the character string region by dividing the length of the side corresponding to the direction with the smaller tilt with respect to the reference direction by the length of the side intersecting with that side. Note that the calculation unit 132 may identify the direction with the smaller tilt with respect to the reference direction as the horizontal direction and the direction of the side intersecting with that side as the vertical direction. The calculation unit 132 may calculate the aspect ratio of the character string region by dividing the length of the vertical side by the length of the horizontal side.

[0030] In FIG. 2, the calculation unit 132 calculates the magnitude of the slope Δ1 of the side of the character string region B11 having points P11 and P12 as its both ends with respect to the X-axis direction, and the magnitude of the slope Δ2 of the side of the character string region B11 having points P11 and P14 as its both ends with respect to the X-axis direction. The calculation unit 132 compares the magnitude of the slope Δ1 with the magnitude of the slope Δ2, and because the magnitude of the slope Δ1 is smaller, the calculation unit 132 identifies the direction of the side having points P11 and P12 as its both ends as the direction with the smaller magnitude of the slope with respect to the reference direction. Because the magnitude of the slope Δ1 is smaller, the calculation unit 132 may identify the direction of the side having points P11 and P12 as the horizontal direction. Furthermore, because the magnitude of the slope Δ2 is larger than the magnitude of the slope Δ1, the calculation unit 132 may identify the direction of the side having points P11 and P14 as the vertical direction. The calculation unit 132 also calculates the length L2 of the side having points P11 and P12 as its ends (also referred to as the horizontal side length L2). The calculation unit 132 calculates the horizontal side length L2 based on the coordinates (x1, y1) indicating the position of point P11 corresponding to the square in the character string area B11 and the coordinates (x2, y2) indicating the position of point P12. The calculation unit 132 calculates the positive square root of the sum of the squares of the first value (y2-y1) and the second value (x2-x1) as the horizontal side length L2. Similarly to the horizontal side length L2, the calculation unit 132 also calculates the length L1 of the side having points P11 and P14 as the side intersecting with the side having points P11 and P12 as its ends. Calculation unit 132 calculates length L1 of the side having points P11 and P14 as both ends (also referred to as length L1 of the vertical side) based on coordinates (x1, y1) indicating the position of point P11 corresponding to the square of character string area B11 and coordinates (x4, y4) indicating the position of point P14. Furthermore, calculation unit 132 calculates the aspect ratio of character string area B11 by dividing length L1 of the side having points P11 and P14 as both ends (also referred to as length L1 of the vertical side) by length L2 of the side having points P11 and P12 as both ends (also referred to as length L2 of the horizontal side).

[0031] (Rotation correction unit 133) Rotation correction unit 133 performs a process of rotating the image. Specifically, calculation unit 132 calculates the average of the aspect ratios of multiple character string regions (hereinafter also referred to as multiple aspect ratios) based on the aspect ratio of the character string region calculated for each character string region. Rotation correction unit 133 compares the average of the multiple aspect ratios calculated by calculation unit 132 with an aspect threshold, and determines whether the average of the multiple aspect ratios satisfies a predetermined condition based on the comparison with the aspect threshold.

[0032] Generally, in an image containing a character string, the length of the character string region in the direction in which the characters are arranged is longer than the direction perpendicular to the direction in which the characters are arranged. Furthermore, if the direction in which the characters are arranged in the image is nearly parallel to the reference direction, the length of the horizontal side of the character string region is greater than the length of the vertical side, and the average of the multiple aspect ratios is equal to or less than the aspect threshold. On the other hand, if the direction in which the characters are arranged in the image is nearly perpendicular to the reference direction, the length of the vertical side of the character string region is greater than the length of the horizontal side, and the average of the multiple aspect ratios exceeds the aspect threshold. Therefore, the rotation correction unit 133 determines whether the direction in which the characters are arranged in the image is nearly perpendicular to the reference direction. Specifically, the rotation correction unit 133 determines whether the average of the multiple aspect ratios exceeds the aspect threshold. If the average of the multiple aspect ratios exceeds the aspect threshold, the rotation correction unit 133 determines that the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with the aspect threshold. If the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with an aspect threshold, the rotation correction unit 133 performs a process of rotating the image by 90 degrees. For example, the rotation correction unit 133 performs a process of rotating the image by 90 degrees clockwise from its original state. In other words, the rotation correction unit 133 performs a process of rotating the image by 90 degrees clockwise with respect to the reference direction. As a result, if the direction in which characters included in the image are arranged is nearly perpendicular to the reference direction, the information processing device 100 can, for example, rotate the image by 90 degrees clockwise with respect to the reference direction to make the direction in which the characters included in the image are arranged nearly parallel to the reference direction.

[0033] FIG. 3 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. The diagram on the left side of FIG. 3 illustrates a case where the direction in which characters included in an image are arranged is nearly perpendicular to a reference direction. Specifically, the diagram on the left side of the top row of FIG. 3 illustrates a case where the orientation of character strings included in an image is rotated 90 degrees counterclockwise with respect to the reference direction (the direction corresponding to the horizontal or lateral direction in FIG. 3). The diagram on the left side of the top row of FIG. 3 also illustrates a state in which a character string region B31 corresponding to a character string included in an image has a length H of a vertical side greater than a length W of a horizontal side. The rotation correction unit 133 determines that the aspect ratio of the character string region B31 exceeds the aspect threshold. Because the aspect ratio of the character string region B31 exceeds the aspect threshold, the rotation correction unit 133 performs a process of rotating the image including the character string region B31 clockwise by 90 degrees from its original state. The diagram on the right side of the upper row of FIG. 3 shows how the image including character string region B31 has been rotated 90 degrees clockwise from its original state by rotation correction unit 133, causing the orientation of character string region B31 (hereinafter sometimes referred to as rotated character string region B31) included in the rotated image to become parallel to the reference direction. Furthermore, the orientation of the character strings located within rotated character string region B31 has changed to a state where they are not rotated with respect to the reference direction. In other words, the orientation of the character strings located within rotated character string region B31 has changed to a state where they are rotated 0 degrees counterclockwise with respect to the reference direction. Hereinafter, the state where the orientation of the character strings located within the character string region is not rotated with respect to the reference direction (in other words, a state where they are rotated 0 degrees counterclockwise with respect to the reference direction) may be referred to as an "upright state."

[0034] The diagram on the left side of the lower row of Figure 3 differs from the diagram on the left side of the upper row of Figure 3 in that the orientation of the character string included in the image is rotated 270° counterclockwise with respect to the reference direction. Similarly to the diagram on the left side of the upper row of Figure 3, the diagram on the left side of the lower row of Figure 3 shows that the length H of the vertical side of character string region B41 corresponding to the character string included in the image is greater than the length W of the horizontal side. The rotation correction unit 133 determines that the aspect ratio of character string region B41 exceeds the aspect threshold. Because the aspect ratio of character string region B41 exceeds the aspect threshold, the rotation correction unit 133 performs processing to rotate the image including character string region B41 by 90° clockwise from its original state. The diagram on the right side of the lower row in FIG. 3 shows how the image including character string region B41 has been rotated 90° clockwise from its original state by rotation correction unit 133, causing the orientation of character string region B41 included in the rotated image (hereinafter sometimes referred to as rotated character string region B41) to become parallel to the reference direction. Furthermore, the orientation of the character string located within character string region B41 after rotation has changed to a state where it has been rotated 180° counterclockwise with respect to the reference direction. Hereinafter, the state in which the orientation of the character string located within the character string region has been rotated 180° counterclockwise with respect to the reference direction may be referred to as the "inverted state."

[0035] Furthermore, extraction unit 131 generates a rotated image after rotation by rotation correction unit 133. Extraction unit 131 may extract character string region information related to character string regions indicating the positions of character strings from the rotated image using machine learning model M1. Calculation unit 132 calculates the tilt of each character string region included in the rotated image with respect to a reference direction, based on the character string region information extracted from the rotated image.

[0036] (Judgment unit 134) As described by comparing the upper and lower diagrams in Figure 3, even if the inclination of a character string area is the same, the orientation of the character string located within the character string area can be either upright or inverted. Therefore, the determination unit 134 determines, for each character string area, whether the orientation of the character string located within the character string area is upright or inverted. The determination unit 134 determines whether the orientation of the character string located within the character string area is upright or inverted relative to a reference direction. Here, the upright orientation of the character string refers to a state in which the orientation of the character string is not rotated relative to the reference direction. In other words, the upright orientation of the character string refers to a state in which the orientation of the character string is rotated 0° counterclockwise relative to the reference direction. Furthermore, the inverted orientation of the character string refers to a state in which the orientation of the character string is rotated 180° counterclockwise relative to the reference direction.

[0037] Specifically, when an image including a character string region is input, the determination unit 134 determines whether the orientation of the character string located in the character string region to be processed is upright or inverted, using a machine learning model M2 that has been trained to determine, for each character string region, whether the orientation of the character string located in the character string region is upright or inverted. More specifically, the machine learning model M2 may be a machine learning model that has been trained to classify the orientation of the character string located in the character string region into two values, whether upright or inverted, and output a classification result as a determination result. For example, when an image including a character string region is input, the determination unit 134 determines the orientation of the character string located in the character string region to be processed using the machine learning model M2 that has been trained to output a determination result of "1" (or "0") if the orientation of the character string located in the character string region is upright and "0" (or "1") if the orientation of the character string is inverted. The determination unit 134 inputs an image including the character string region to be processed into the machine learning model M2 and obtains a determination result output from the machine learning model M2.

[0038] 4 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. In FIG. 4, the determination unit 134 inputs the image G1 generated by the extraction unit 131 to the machine learning model M2, and obtains a determination result indicating whether the orientation of each of the character strings T11 to T13 located within each of the character string regions B11 to B13 is upright or inverted.

[0039] (Estimation part 135) Estimation unit 135 estimates the rotation angle of each character string based on the calculation results of calculation unit 132 and the determination results of determination unit 134. Specifically, estimation unit 135 estimates the rotation angle of each character string based on the tilt of the character string region calculated by calculation unit 132. More specifically, if the tilt of the character string region calculated by calculation unit 132 is represented as a and the counterclockwise rotation angle of the character string region relative to the reference direction is represented as θ, then a and θ are related by a trigonometric function, "a = tan(θ)," and an inverse function, "θ = arctan(a)." For example, estimation unit 135 calculates the counterclockwise rotation angle (θ) of the character string region relative to the reference direction by substituting the tilt (a) of the character string region calculated by calculation unit 132 into the inverse function, "θ = arctan(a)."

[0040] Furthermore, the estimation unit 135 estimates the rotation angle of the character string located in the character string area based on the counterclockwise rotation angle (θ) of the character string area relative to the reference direction and the orientation of the character string located in the character string area determined by the determination unit 134. More specifically, the estimation unit 135 estimates the counterclockwise rotation angle (φ) of the character string located in the character string area relative to the reference direction. For example, if the orientation of the character string determined by the determination unit 134 is upright, the estimation unit 135 estimates that the counterclockwise rotation angle (φ) of the character string located in the character string area relative to the reference direction is the same angle (θ) as the counterclockwise rotation angle (θ) of the character string located in the character string area relative to the reference direction. Furthermore, if the orientation of the character string determined by the determination unit 134 is inverted, the estimation unit 135 estimates that the counterclockwise rotation angle (φ) of the character string located in the character string area relative to the reference direction is an angle (θ + 180°) obtained by adding 180° to the counterclockwise rotation angle (θ) of the character string area relative to the reference direction.

[0041] The estimation unit 135 also calculates the average (ρ) of the rotation angles (φ) of the multiple character strings. The estimation unit 135 estimates the rotation angle of the image based on the average of the rotation angles of the multiple character strings and outputs the estimated rotation angle of the image. For example, the estimation unit 135 selects one angle from four or more discretized angles and outputs the selected angle as the rotation angle of the image. For example, the estimation unit 135 selects one angle from four discretized angles, 0°, 90°, 180°, and 270°, and outputs the selected angle as the rotation angle of the image. For example, if the average (ρ) of the rotation angles of the multiple character strings satisfies 0°≦ρ<90°, the estimation unit 135 selects 0° from the four discretized angles and outputs 0° as the rotation angle of the image. If the average (ρ) of the rotation angles of the multiple character strings satisfies 90°≦ρ<180°, the estimation unit 135 selects 90° from the four discretized angles and outputs 90° as the rotation angle of the image. Furthermore, when the average (ρ) of the rotation angles of the multiple character strings satisfies 180°≦ρ<270°, the estimation unit 135 selects 180° from the four discretized angles and outputs 180° as the rotation angle of the image. Furthermore, when the average (ρ) of the rotation angles of the multiple character strings satisfies 270°≦ρ<360°, the estimation unit 135 selects 270° from the four discretized angles and outputs 270° as the rotation angle of the image. Note that the estimation unit 135 may select one angle from five or more discretized angles and output the selected angle as the rotation angle of the image.

[0042] (Provider 136) The providing unit 136 rotates the image based on the rotation angle (μ) of the image estimated by the estimating unit 135. Specifically, the providing unit 136 generates a rotated image by rotating the image clockwise by the rotation angle (μ) with respect to the reference direction based on the rotation angle (μ) of the image estimated by the estimating unit 135. The providing unit 136 also performs AI-OCR on the generated rotated image and provides information about character strings in the rotated image recognized by the AI-OCR. For example, the providing unit 136 transmits information about character strings in the rotated image to a user's terminal device.

[0043] [3. Modifications] The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0044] In the above-described embodiment, the case where the character string included in the image is written horizontally and the reference direction is the same as the X-axis direction has been described, but the character string included in the image may be written vertically. If the character string included in the image is written vertically, the reference direction may be the same as the Y-axis direction.

[0045] In the above-described embodiment, the rotation correction unit rotates the image 90° clockwise with respect to the reference direction when the average aspect ratio of the character string region exceeds the aspect threshold. However, the rotation correction unit 133 may also rotate the image 90° clockwise with respect to the reference direction when the average aspect ratio of the character string region is below the aspect threshold. For example, consider a case where the character string included in the image is written vertically and the reference direction is the same as the Y-axis direction. In this case, if the average of multiple aspect ratios is below the aspect threshold, the rotation correction unit 133 determines that the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with the aspect threshold. If the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with the aspect threshold, the rotation correction unit 133 rotates the image 90°. For example, the rotation correction unit 133 rotates the image 90° clockwise from its original state. In other words, the rotation correction unit 133 rotates the image 90° clockwise with respect to the reference direction.

[0046] [4. Effects] As described above, the information processing device 100 according to the embodiment includes the extraction unit 131, the calculation unit 132, the determination unit 134, and the estimation unit 135. The extraction unit 131 extracts character string region information related to character string regions, which are rectangular regions indicating the positions of character strings, from an image including character strings using a machine learning model. The calculation unit 132 calculates the inclination of each character string region with respect to a reference direction based on the character string region information. The determination unit 134 determines, for each character string region, whether the orientation of the character string located within the character string region is upright or inverted. The estimation unit 135 estimates the rotation angle of each character string based on the calculation result of the calculation unit 132 and the determination result of the determination unit 134.

[0047] This allows the information processing device 100 to accurately estimate the rotation angle of each character string included in the image. Therefore, the information processing device 100 can accurately estimate the rotation angle of the image based on the rotation angles of multiple character strings that have been accurately estimated.

[0048] Furthermore, calculation unit 132 calculates the inclination of at least one side of the character string region with respect to the reference direction as the inclination of the character string region with respect to the reference direction.

[0049] This allows the information processing device 100 to accurately estimate the rotation angle of each character string included in the image, based on the inclination of the direction of at least one side of the character string region with respect to the reference direction.

[0050] Information processing device 100 further includes rotation correction unit 133 that performs a process of rotating an image. Calculation unit 132 calculates the aspect ratio of each character string region based on the character string region information. Rotation correction unit 133 performs a process of rotating the image by 90° when the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with an aspect threshold.

[0051] As a result, when the direction in which characters included in an image are lined up is nearly perpendicular to the reference direction, the information processing device 100 can make the direction in which the characters included in the image are lined up nearly parallel to the reference direction by, for example, rotating the image by 90° clockwise with respect to the reference direction. Furthermore, since the information processing device 100 can make the direction in which the characters included in the image are lined up nearly parallel to the reference direction, it can accurately determine whether the orientation of the character string located in the character string area is upright or inverted with respect to the reference direction.

[0052] Furthermore, the estimation unit 135 estimates the rotation angle of the image based on the average of the rotation angles of a plurality of character strings, and outputs the estimated rotation angle of the image.

[0053] This allows the information processing device 100 to accurately estimate the rotation angle of the image based on the average of the rotation angles of a plurality of character strings that have been accurately estimated.

[0054] Furthermore, the estimation unit 135 selects one angle from the four or more discretized angles, and outputs the selected angle as the rotation angle of the image.

[0055] This allows the information processing device 100 to roughly estimate the rotation angle of the image.

[0056] Furthermore, when a plurality of images each including a character string tilted at a plurality of different rotation angles are input, the extraction unit 131 extracts character string area information using a machine learning model trained to output character string area information regarding the character string area corresponding to each of the character strings tilted at a plurality of different rotation angles.

[0057] This allows the information processing device 100 to extract character string region information with high accuracy for each character string using a machine learning model with improved robustness against character string rotation.

[0058] [5. Hardware Configuration] The information processing device 100 according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 5. Fig. 5 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0059] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, programs that depend on the hardware of the computer 1000, and the like.

[0060] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0061] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0062] Media interface 1700 reads a program or data stored in recording medium 1800 and provides it to CPU 1100 via RAM 1200. CPU 1100 loads the program or data from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0063] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0064] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have undergone various modifications and improvements based on the knowledge of those skilled in the art.

[0065] [6. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0066] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0067] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content. [Explanation of symbols]

[0068] 100 Information processing device 110 Communications Department 120 Storage section 130 Control Unit 131 Extraction part 132 Calculation section 133 Rotation correction unit 134 Judgment section 135 Estimation Department 136 Provision Department

Claims

1. an extraction unit that extracts character string region information related to a character string region, which is a rectangular region indicating the position of the character string, from an image including the character string using a machine learning model; a calculation unit that calculates, for each character string region, an inclination of the character string region with respect to a reference direction based on the character string region information; a determination unit that determines, for each character string area, whether the orientation of the character string located within the character string area is upright or inverted; an estimation unit that estimates a rotation angle of each of the character strings based on the calculation result of the calculation unit and the determination result of the determination unit; An information processing device comprising:

2. The calculation unit As the inclination of the character string region with respect to the reference direction, the inclination of the direction of at least one side of the character string region with respect to the reference direction is calculated. The information processing device according to claim 1 .

3. a rotation correction unit that performs processing to rotate the image, The calculation unit calculating an aspect ratio of each character string region based on the character string region information; The rotation correction unit If the average of the plurality of aspect ratios satisfies a predetermined condition based on a comparison with an aspect threshold, the image is rotated by 90 degrees. The information processing device according to claim 1 .

4. The estimation unit The rotation angle of the image is estimated based on an average of the rotation angles of the plurality of character strings, and the estimated rotation angle of the image is output. The information processing device according to claim 1 .

5. The estimation unit One angle is selected from the four or more discretized angles, and the selected angle is output as the rotation angle of the image. The information processing device according to claim 3 .

6. The extraction unit When a plurality of images each including a character string tilted at a plurality of different rotation angles are input, the character string region information is extracted by the machine learning model that has been trained to output character string region information regarding a character string region corresponding to each of the character strings tilted at the plurality of different rotation angles. The information processing device according to claim 1 .

7. An information processing method realized by a program executed by an information processing device, an extraction step of extracting character string region information relating to a character string region, which is a rectangular region indicating the position of the character string, from an image including the character string using a machine learning model; a calculation step of calculating an inclination of each character string region with respect to a reference direction based on the character string region information; a determining step of determining, for each character string region, whether the orientation of the character strings located within the character string region is upright or inverted; an estimation step of estimating a rotation angle of each of the character strings based on the calculation result of the calculation step and the determination result of the determination step; An information processing method including:

8. an extraction step of extracting character string region information relating to a character string region, which is a rectangular region indicating the position of the character string, from an image including the character string using a machine learning model; a calculation step of calculating an inclination of each character string region with respect to a reference direction based on the character string region information; a determination step of determining, for each character string region, whether the orientation of the character string located within the character string region is upright or inverted; an estimation step of estimating a rotation angle of each of the character strings based on a calculation result of the calculation step and a determination result of the determination step; An information processing program that causes a computer to execute the above.

Images