Information processing device, information processing method, and information processing program

Abstract

To accurately estimate the rotation angle of an image including a character string. [Solution] The information processing device of the present application includes an extraction unit that extracts string region information regarding string regions, which are rectangular regions that indicate the position of strings, from an image including strings using a machine learning model; a calculation unit that calculates the inclination of each string region with respect to a reference direction based on the string region information; a determination unit that determines for each string region whether the orientation of the string located within the string region is upright or inverted; and an estimation unit that estimates the rotation angle of each string based on the calculation results of the calculation unit and the determination results of the determination unit.

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, OCR (Optical Character Recognition) technology that recognizes characters included in an image has been known. In recent years, technology related to AI-OCR, which is OCR using deep learning, has become known. Here, it is known that if a character string included in an image is rotated on a medium, or if the character string is facing the medium but the medium itself is rotated, it becomes difficult for the OCR to recognize the character string.

[0003] In response to this, a technique for estimating the rotation angle of a character string included in an image is known. For example, a technique related to a learning model is known in which, when a rotated character image, which is an image showing rotated characters, which are characters rotated at a predetermined angle, is input to an input layer, a recognition rate of characters included in the rotated character image is calculated through calculations in an intermediate layer, and the highest recognition rate, character information corresponding to the highest recognition rate, and rotation angle information showing the predetermined angle are output from an output layer. [Prior art documents] [Non-patent literature]

[0004] [Patent Document 1] Patent Publication No. 2023-85076 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned conventional technology merely uses a learning model to calculate the recognition rate of characters contained in a rotated character image, and outputs the highest recognition rate value, character information corresponding to the highest recognition rate value, and rotation angle information indicating a predetermined angle, so it is not necessarily possible to accurately estimate the rotation angle of an image containing a character string.

[0006] An object of the present application is to provide an information processing device, an information processing method, and an information processing program capable of accurately estimating the rotation angle of an image including a character string. [Means for solving the problem]

[0007] The information processing device of the present application includes an extraction unit that extracts string region information regarding string regions, which are rectangular regions indicating the position of the string, from an image including the strings using a machine learning model; a calculation unit that calculates the inclination of the string region with respect to a reference direction for each of the string regions based on the string region information; a determination unit that determines for each of the string regions whether the orientation of the string located within the string region is upright or inverted; and an estimation unit that estimates the rotation angle of the string for each of the strings based on the calculation result of the calculation unit and the determination result of the determination unit. Effect of the Invention

[0008] According to one aspect of the embodiment, it is possible to accurately estimate the rotation angle of an image including a character string. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. [Figure 4]FIG. 4 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. [Diagram 5] FIG. 5 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application (hereinafter, referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. In addition, the same parts in the following embodiments are given the same reference numerals, and duplicated descriptions are omitted. In addition, hereinafter, AI (artificial intelligence) and machine learning model are treated as synonymous terms.

[0011] (Embodiment) 1. Introduction Conventionally, OCR (Optical Character Recognition) technology that recognizes characters contained in an image has been known. In recent years, technology related to AI-OCR, which is OCR using deep learning, has become known. For example, in identity verification in various services, AI-OCR technology is used to extract character strings contained in an image of an identification document submitted by a user, and identity authentication is performed.

[0012] However, AI-OCR is not an all-purpose tool. Specifically, most machine learning models have low robustness against character string rotation. Therefore, even a slight tilt of the character string can lead to misrecognition or a significant drop in recognition accuracy. Here, tilted character strings include both cases where the character string in the image is rotated on the medium, and cases where the character string is facing the medium but the medium itself is rotated.

[0013] In response to this, the inventor of the present invention considered that there is a limit to improving the accuracy of AI-OCR itself. The inventor of the present invention also focused on improving the accuracy of character string recognition by AI-OCR by accurately estimating the rotation angle of an image including character strings and rotating the image based on the accurately estimated rotation angle. Based on this problem recognition, the present invention relates to a method for estimating the rotation angle of each character string with high accuracy from an image including character strings, such as an identification card.

[0014] Specifically, the information processing device 100 according to the embodiment improves the rotation robustness of an open source AI that extracts character string region information on a character string region (e.g., a bounding box) that is a rectangular region indicating the position of a character string from an image including the character string by providing learning data including each of a plurality of images including each of character strings tilted at a plurality of different rotation angles and pre-learning the AI. For example, the open source AI may be CRAFT (Non-Patent Document 1; Youngmin Baek et al., "Character region awareness for text detection.", [online], [searched July 13, 2024], 2019, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition., https: / / arxiv.org / pdf / 1904.01941). In addition, the information processing device 100 calculates the inclination and aspect ratio of the character string region with respect to a reference direction based on the character string region information output by the AI ​​with improved rotation robustness, and estimates the rotation angle of each character string with high accuracy based on the inclination and aspect ratio of the character string region with respect to the reference direction. This allows the information processing device 100 to accurately estimate the rotation angle of an image including a character string, based on the average of the rotation angles of a plurality of character strings estimated with high accuracy.

[0015] 2. Configuration of Information Processing Device An example of the configuration of an information processing device 100 according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. The information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

[0016] (Communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC). The communication unit 110 is connected to various networks by wire or wirelessly, and transmits and receives information to and from an external information processing device (for example, a terminal device used by a user).

[0017] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various data. The storage unit 120 also stores various programs. For example, the storage unit 120 stores an information processing program according to the embodiment.

[0018] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., executing various programs stored in a storage device inside the information processing device 100 using a RAM as a working area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0019] The control unit 130 has an extraction unit 131, a calculation unit 132, a rotation correction unit 133, a determination unit 134, an estimation unit 135, and a provision unit 136 as functional units, and may realize or execute the information processing action described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 1, and may be other configurations as long as they perform the information processing described below. Also, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0020] (Extraction part 131) The extraction unit 131 generates an image including a character string. For example, the extraction unit 131 acquires an image of a medium on which a character string is written from a terminal device of a user via the communication unit 110. For example, the medium may be a card such as an identification card or a document. Next, the extraction unit 131 generates an image including a character string based on the image of the medium on which the character string is written. FIG. 2 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. On the left side of FIG. 2, the extraction unit 131 acquires an image D1 of a medium on which horizontally written character strings T11 to T13, "text box," are written. Next, the extraction unit 131 generates an image G1 including the character strings T11 to T13 based on the image D1 of the medium. Specifically, the extraction unit 131 generates an image G1 indicating the position of the image D1 of the medium in a two-dimensional orthogonal coordinate system (for example, an XY coordinate system). For example, the extraction unit 131 generates an image G1 indicating the position of the image D1 of the medium in a plane including an X axis indicating a reference direction and a Y axis perpendicular to the X axis. For example, the extraction unit 131 generates an image G1 in which one pair of opposite sides of the rectangular image G1 are parallel to the X-axis, and the other pair of opposite sides of the image G1 are parallel to the Y-axis. For example, the extraction unit 131 generates an image D1 in which one corner of the rectangular image G1 corresponds to the origin O of the XY coordinate system. In FIG. 2, in order to make the figure easier to see, the lower left corner of the rectangular image G1 is shown slightly offset from the origin O of the XY coordinate system. In reality, the extraction unit 131 generates an image G1 in which the lower left corner of the rectangular image G1 corresponds to the origin O of the XY coordinate system.

[0021] Moreover, the extraction unit 131 extracts, from an image including a character string, character string region information related to a character string region, which is a rectangular region indicating the position of a character string by a machine learning model. For example, the character string region is a bounding box surrounding a character string. Moreover, the character string region information includes the character string region itself. Moreover, the character string region information includes coordinate information indicating the positions of each of the points corresponding to the rectangle of the character string region. Specifically, when an image including a character string is input, the extraction unit 131 extracts character string region information corresponding to the image to be processed from the image to be processed using the machine learning model M1 that has been trained to output character string region information. More specifically, the extraction unit 131 inputs the image to be processed to the machine learning model M1 and acquires character string region information output from the machine learning model M1.

[0022] Furthermore, when a plurality of images including each of character strings tilted at a plurality of different rotation angles are input, the extraction unit 131 extracts character string region information using a machine learning model trained to output character string region information regarding each of the character strings tilted at a plurality of different rotation angles. Specifically, when an image including a character string tilted at a rotation angle θ (θ is any angle between 0°≦θ<360°) counterclockwise with respect to a reference direction is input, the extraction unit 131 extracts character string region information corresponding to the image to be processed from the image to be processed using a machine learning model M1 trained to output character string region information regarding a character string region corresponding to the character string tilted at the rotation angle θ.

[0023] In FIG. 2, the extraction unit 131 inputs the generated image G1 to the machine learning model M1, and extracts, from the image G1, character string region information related to each of the character string regions B11 to B13 indicating the respective positions of the character strings T11 to T13 included in the image G1. For example, the extraction unit 131 extracts each of the character string regions B11 to B13 as the character string region information. Furthermore, the extraction unit 131 extracts coordinate information indicating the positions of the square points of each of the character string regions B11 to B13 as the character string region information. For example, the extraction unit 131 extracts the character string region B11 itself as the character string region information related to the character string region B11. Furthermore, the extraction unit 131 extracts XY coordinate information indicating the positions of each of the square points P11 to P14 of the character string region B11 as the character string region information related to the character string region B11.

[0024] (Calculation section 132) Calculation unit 132 excludes from the calculation any character string region whose width is zero based on the character string region information extracted by extraction unit 131. Calculation unit 132 excludes linear character string regions from the calculation based on the character string region information. For example, calculation unit 132 excludes from the calculation any character string region from which XY coordinate information indicating the positions of four points corresponding to the squares of the character string region was not extracted by extraction unit 131. In the following, calculation unit 132 performs calculations on character string regions whose width is not zero.

[0025] Calculation unit 132 calculates the inclination of each character string region with respect to a reference direction based on the character string region information. Specifically, calculation unit 132 calculates the inclination of the direction of at least one side of the character string region with respect to the reference direction as the inclination of the character string region with respect to the reference direction. Calculation unit 132 calculates the inclination of the direction of at least one side of the character string region with respect to the reference direction for each character string region based on the character string region information.

[0026] For example, the calculation unit 132 calculates the inclination of the direction of at least one side of the character string region with respect to the reference direction based on the coordinate information indicating the position of the point corresponding to the square of the character string region extracted by the extraction unit 131. For example, the reference direction is the same direction as the X-axis direction. The reference direction may be the same direction as the Y-axis direction. For example, the calculation unit 132 calculates a first value which is a value obtained by subtracting the value of the Y coordinate indicating the position of the point corresponding to one end from the value of the Y coordinate indicating the position of the point corresponding to the other end among both ends of at least one side of the character string region. Furthermore, the calculation unit 132 calculates a second value which is a value obtained by subtracting the value of the X coordinate indicating the position of the point corresponding to the other end from the value of the X coordinate indicating the position of the point corresponding to one end among both ends of at least one side of the character string region. Furthermore, the calculation unit 132 calculates a value obtained by dividing the first value by the second value as the inclination of the direction of at least one side of the character string region with respect to the reference direction. Hereinafter, the inclination of the direction of at least one side of the character string region with respect to the reference direction may be described as the inclination of the character string region with respect to the reference direction.

[0027] In FIG. 2, the calculation unit 132 calculates the inclination of the side of the character string region B11, the ends of which are the points P11 and P12, in the X-axis direction. The calculation unit 132 calculates the inclination of the side of the character string region B11, the ends of which are the points P11 and P12, in the X-axis direction, based on the coordinates (x1, y1) indicating the position of the point P11 corresponding to the square of the character string region B11 and the coordinates (x2, y2) indicating the position of the point P12. For example, the calculation unit 132 calculates a first value (y2-y1) which is a value obtained by subtracting the Y coordinate value y1 of the point P11 from the Y coordinate value y2 of the point P12. The calculation unit 132 also calculates a second value (x2-x1) which is a value obtained by subtracting the X coordinate value x1 of the point P11 from the X coordinate value x2 of the point P12. Furthermore, the calculation unit 132 calculates the value obtained by dividing the first value (y2-y1) by the second value (x2-x1) as the inclination of the side having the points P11 and P12 as both ends with respect to the X-axis direction.

[0028] Note that the calculation unit 132 may calculate the inclination of the direction of any side of the character string region with respect to the reference direction as long as the inclination is the inclination of the direction of at least one side of the character string region with respect to the reference direction. For example, in FIG. 2, the calculation unit 132 may calculate the inclination of the side with points P13 and P14 with respect to the X-axis direction, instead of calculating the inclination of the side with points P11 and P12 with respect to the X-axis direction, among the sides of the character string region B11. Also, the calculation unit 132 may calculate the inclination of the side with points P11 and P12 with respect to the X-axis direction, instead of calculating the inclination of the side with points P11 and P12 with respect to the X-axis direction. Also, the calculation unit 132 may calculate the inclination of the side with points P12 and P13 with respect to the X-axis direction, instead of calculating the inclination of the side with points P11 and P14 with respect to the X-axis direction.

[0029] Further, calculation unit 132 calculates the aspect ratio of each character string region based on the character string region information extracted by extraction unit 131. The aspect ratio can also be said to be the length to width ratio. Specifically, calculation unit 132 calculates the aspect ratio of the character string region based on coordinate information indicating the positions of the square points of the character string region extracted by extraction unit 131. More specifically, calculation unit 132 calculates a value obtained by dividing the length of at least one side of the character string region by the length of a side intersecting with the side, as the aspect ratio of the character string region. Note that the shape of the character string region is not limited to a geometrically defined rectangular shape (rectangle). Specifically, the shape of the character string region includes a shape having an allowable degree of difference (error / distortion) and a shape similar to that shape. For example, the shape of the character string region may be a trapezoid. For example, calculation unit 132 calculates the magnitude of the inclination of the direction of at least one side of the character string region with respect to a reference direction, and the magnitude of the inclination of the direction of the side intersecting with the side with respect to the reference direction. Next, the calculation unit 132 identifies the side corresponding to the direction in which the magnitude of the tilt is smaller with respect to the reference direction based on a comparison between the magnitudes of the tilt. The calculation unit 132 may calculate a value obtained by dividing the length of the side corresponding to the direction in which the magnitude of the tilt is smaller with respect to the reference direction by the length of the side intersecting with the side, as the aspect ratio of the character string region. The calculation unit 132 may specify the direction in which the magnitude of the tilt is smaller with respect to the reference direction as the horizontal direction, and the direction of the side intersecting with the side, as the vertical direction. The calculation unit 132 may calculate a value obtained by dividing the length of the vertical side by the length of the horizontal side, as the aspect ratio of the character string region.

[0030] In FIG. 2, the calculation unit 132 calculates the magnitude of the inclination Δ1 of the side having points P11 and P12 at both ends with respect to the X-axis direction, and the magnitude of the inclination Δ2 of the side having points P11 and P14 at both ends with respect to the X-axis direction, among the sides of the character string region B11. The calculation unit 132 compares the magnitude of the inclination Δ1 with the magnitude of the inclination Δ2, and since the magnitude of the inclination Δ1 is smaller, specifies the direction of the side having points P11 and P12 at both ends as the direction with the smaller magnitude of the inclination with respect to the reference direction. Since the magnitude of the inclination Δ1 is smaller, the calculation unit 132 may specify the direction of the side having points P11 and P12 at both ends as the horizontal direction. Furthermore, since the magnitude of the inclination Δ2 is larger than the magnitude of the inclination Δ1, the calculation unit 132 may specify the direction of the side having points P11 and P14 at both ends as the vertical direction. The calculation unit 132 also calculates the length L2 of the side having the points P11 and P12 as its two ends (also referred to as the length L2 of the horizontal side). The calculation unit 132 calculates the length L2 of the horizontal side based on the coordinates (x1, y1) indicating the position of the point P11 corresponding to the square of the character string area B11 and the coordinates (x2, y2) indicating the position of the point P12. The calculation unit 132 calculates the positive square root of the sum of the value obtained by squaring the first value (y2-y1) and the value obtained by squaring the second value (x2-x1) as the length L2 of the horizontal side. Similarly to the length L2 of the horizontal side, the calculation unit 132 also calculates the length L1 of the side having the points P11 and P14 as the side intersecting with the side having the points P11 and P12 as its two ends. Calculation unit 132 calculates length L1 (also referred to as length L1 of the vertical side) of the side having points P11 and P14 as both ends based on coordinates (x1, y1) indicating the position of point P11 corresponding to the square of character string area B11 and coordinates (x4, y4) indicating the position of point P14. Calculation unit 132 also calculates the aspect ratio of character string area B11 by dividing length L1 (also referred to as length L1 of the vertical side) of the side having points P11 and P14 as both ends by length L2 (also referred to as length L2 of the horizontal side) of the side having points P11 and P12 as both ends.

[0031] (Rotation correction unit 133) Rotation correction unit 133 performs a process of rotating an image. Specifically, calculation unit 132 calculates an average of the aspect ratios of a plurality of character string regions (hereinafter also referred to as a plurality of aspect ratios) based on the aspect ratio of the character string region calculated for each character string region. Rotation correction unit 133 compares the average of the plurality of aspect ratios calculated by calculation unit 132 with an aspect threshold value, and determines whether or not the average of the plurality of aspect ratios satisfies a predetermined condition based on the comparison with the aspect threshold value.

[0032] Generally, in an image including a character string, the length of the character string region in the direction in which the characters included in the character string are arranged is longer than the direction perpendicular to the direction in which the characters included in the character string are arranged. In addition, when the direction in which the characters included in the image are arranged is nearly parallel to the reference direction, the length of the horizontal side of the character string region is greater than the length of the vertical side, so that the average of the multiple aspect ratios is equal to or less than the aspect threshold. On the other hand, when the direction in which the characters included in the image are arranged is nearly perpendicular to the reference direction, the length of the vertical side of the character string region is greater than the length of the horizontal side, so that the average of the multiple aspect ratios exceeds the aspect threshold. Therefore, the rotation correction unit 133 determines whether the direction in which the characters included in the image are arranged is nearly perpendicular to the reference direction. Specifically, the rotation correction unit 133 determines whether the average of the multiple aspect ratios exceeds the aspect threshold. When the average of the multiple aspect ratios exceeds the aspect threshold, the rotation correction unit 133 determines that the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with the aspect threshold. When the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with an aspect threshold, the rotation correction unit 133 performs a process of rotating the image by 90°. For example, the rotation correction unit 133 performs a process of rotating the image by 90° clockwise from the original state. In other words, the rotation correction unit 133 performs a process of rotating the image by 90° clockwise with respect to the reference direction. As a result, when the direction in which characters included in the image are lined up is nearly perpendicular to the reference direction, the information processing device 100 can make the direction in which characters included in the image are lined up nearly parallel to the reference direction by, for example, rotating the image by 90° clockwise with respect to the reference direction.

[0033] FIG. 3 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. The diagram on the left side of FIG. 3 shows a case where the direction in which characters included in an image are arranged is nearly perpendicular to a reference direction. Specifically, the diagram on the left side of the upper part of FIG. 3 shows a case where the orientation of a character string included in an image is rotated 90° counterclockwise with respect to a reference direction (a direction corresponding to the horizontal direction or the lateral direction in FIG. 3). Also, the diagram on the left side of the upper part of FIG. 3 shows a state in which a character string area B31 corresponding to a character string included in an image has a length H of a vertical side greater than a length W of a horizontal side. The rotation correction unit 133 determines that the aspect ratio of the character string area B31 exceeds the aspect threshold. Since the aspect ratio of the character string area B31 exceeds the aspect threshold, the rotation correction unit 133 performs a process of rotating the image including the character string area B31 clockwise by 90° from the original state. The diagram on the right side of the upper row of FIG. 3 shows how the orientation of character string region B31 (hereinafter may be referred to as rotated character string region B31) included in the rotated image has changed to a state parallel to the reference direction as a result of rotation correction unit 133 rotating the image including character string region B31 90° clockwise from the original state. The orientation of the character string located within rotated character string region B31 has changed to a state where it is not rotated with respect to the reference direction. In other words, the orientation of the character string located within rotated character string region B31 has changed to a state where it is rotated 0° counterclockwise with respect to the reference direction. Hereinafter, the state where the orientation of the character string located within the character string region is not rotated with respect to the reference direction (in other words, a state where it is rotated 0° counterclockwise with respect to the reference direction) may be referred to as an "upright state."

[0034] The diagram on the left side of the lower part of Fig. 3 differs from the diagram on the left side of the upper part of Fig. 3 in that the orientation of the character string included in the image is rotated 270° counterclockwise with respect to the reference direction. Similarly to the diagram on the left side of the upper part of Fig. 3, the diagram on the left side of the lower part of Fig. 3 illustrates how the length H of the vertical side of character string region B41 corresponding to the character string included in the image is greater than the length W of the horizontal side. Rotation correction unit 133 determines that the aspect ratio of character string region B41 exceeds the aspect threshold. Because the aspect ratio of character string region B41 exceeds the aspect threshold, rotation correction unit 133 performs a process of rotating the image including character string region B41 90° clockwise from the original state. The diagram on the right side of the lower row of FIG. 3 shows how the image including character string region B41 has been rotated 90° clockwise from its original state by rotation correction section 133, causing the orientation of character string region B41 included in the rotated image (hereinafter sometimes referred to as rotated character string region B41) to change to a state parallel to the reference direction. In addition, the orientation of the character string located within character string region B41 after rotation has changed to a state rotated 180° counterclockwise with respect to the reference direction. Hereinafter, the state in which the orientation of the character string located within the character string region has been rotated 180° counterclockwise with respect to the reference direction may be referred to as the "inverted state."

[0035] Furthermore, extraction unit 131 generates a rotated image after rotation by rotation correction unit 133. Extraction unit 131 may extract character string region information related to a character string region indicating a position of a character string from the rotated image by machine learning model M1. Calculation unit 132 calculates the inclination of the character string region with respect to a reference direction for each character string region included in the rotated image, based on the character string region information extracted from the rotated image.

[0036] (Judgment unit 134) As described by comparing the upper and lower diagrams of FIG. 3, even if the inclination of the character string area is the same, the orientation of the character string located in the character string area can be in two ways: upright and inverted. Therefore, the determination unit 134 determines whether the orientation of the character string located in the character string area is upright or inverted for each character string area. The determination unit 134 determines whether the orientation of the character string located in the character string area is upright or inverted with respect to the reference direction. Here, the upright orientation of the character string refers to a state in which the orientation of the character string is not rotated with respect to the reference direction. In other words, the upright orientation of the character string refers to a state in which the orientation of the character string is rotated 0° counterclockwise with respect to the reference direction. Moreover, the inverted orientation of the character string refers to a state in which the orientation of the character string is rotated 180° counterclockwise with respect to the reference direction.

[0037] Specifically, when an image including a character string region is input, the determination unit 134 uses the machine learning model M2 that has been trained to determine for each character string region whether the orientation of the character string located in the character string region is upright or inverted to determine whether the orientation of the character string located in the character string region to be processed is upright or inverted. More specifically, the machine learning model M2 may be a machine learning model trained to output a classification result obtained by classifying the orientation of the character string located in the character string region into a binary state, whether the orientation of the character string located in the character string region is upright or inverted, as a determination result. For example, when an image including a character string region is input, the determination unit 134 uses the machine learning model M2 that has been trained to output "1" (or "0") as a determination result when the orientation of the character string located in the character string region is upright, and output "0" (or "1") as a determination result when the orientation of the character string located in the character string region to be processed. The determination unit 134 inputs an image including the character string region to be processed to the machine learning model M2 and acquires a determination result output from the machine learning model M2.

[0038] Fig. 4 is a diagram for explaining an example of information processing by the information processing device according to the embodiment. In Fig. 4, a determination unit 134 inputs an image G1 generated by an extraction unit 131 to a machine learning model M2, and obtains a determination result indicating whether the orientation of each of the character strings T11-T13 located in each of the character string regions B11-B13 is upright or inverted.

[0039] (Estimation part 135) Estimation unit 135 estimates the rotation angle of each character string based on the calculation result of calculation unit 132 and the determination result of determination unit 134. Specifically, estimation unit 135 estimates the rotation angle of each character string based on the inclination of the character string region calculated by calculation unit 132. More specifically, if the inclination of the character string region calculated by calculation unit 132 is represented as a and the counterclockwise rotation angle of the character string region with respect to the reference direction is represented as θ, a trigonometric function relational expression "a=tan(θ)" and an inverse function relational expression "θ=arctan(a)" hold between a and θ. For example, estimation unit 135 substitutes the inclination (a) of the character string region calculated by calculation unit 132 into the inverse function relational expression "θ=arctan(a)" to calculate the counterclockwise rotation angle (θ) of the character string region with respect to the reference direction.

[0040] Furthermore, the estimation unit 135 estimates the rotation angle of the character string located in the character string area based on the counterclockwise rotation angle (θ) of the character string area relative to the reference direction and the orientation of the character string located in the character string area determined by the determination unit 134. More specifically, the estimation unit 135 estimates the counterclockwise rotation angle (φ) of the character string located in the character string area relative to the reference direction. For example, when the orientation of the character string determined by the determination unit 134 is upright, the estimation unit 135 estimates that the counterclockwise rotation angle (φ) of the character string located in the character string area relative to the reference direction is the same angle (θ) as the counterclockwise rotation angle (θ) of the character string located in the character string area relative to the reference direction. Furthermore, when the orientation of the character string determined by the determination unit 134 is inverted, the estimation unit 135 estimates that the counterclockwise rotation angle (φ) of the character string located in the character string area relative to the reference direction is an angle (θ+180°) obtained by adding 180° to the counterclockwise rotation angle (θ) of the character string area relative to the reference direction.

[0041] Furthermore, the estimation unit 135 calculates the average (ρ) of the rotation angles (φ) of the multiple character strings. The estimation unit 135 estimates the rotation angle of the image based on the average of the rotation angles of the multiple character strings, and outputs the estimated rotation angle of the image. For example, the estimation unit 135 selects one angle from four or more discretized angles, and outputs the selected angle as the rotation angle of the image. For example, the estimation unit 135 selects one angle from four discretized angles of 0°, 90°, 180°, and 270°, and outputs the selected angle as the rotation angle of the image. For example, when the average (ρ) of the rotation angles of the multiple character strings is an angle that satisfies 0°≦ρ<90°, the estimation unit 135 selects 0° from the four discretized angles, and outputs 0° as the rotation angle of the image. Furthermore, when the average (ρ) of the rotation angles of the multiple character strings is an angle that satisfies 90°≦ρ<180°, the estimation unit 135 selects 90° from the four discretized angles, and outputs 90° as the rotation angle of the image. Furthermore, when the average (ρ) of the rotation angles of the multiple character strings satisfies 180°≦ρ<270°, the estimation unit 135 selects 180° from the four discretized angles and outputs 180° as the rotation angle of the image. Furthermore, when the average (ρ) of the rotation angles of the multiple character strings satisfies 270°≦ρ<360°, the estimation unit 135 selects 270° from the four discretized angles and outputs 270° as the rotation angle of the image. Note that the estimation unit 135 may select one angle from five or more discretized angles and output the selected one angle as the rotation angle of the image.

[0042] (Provider 136) The providing unit 136 rotates the image based on the rotation angle (μ) of the image estimated by the estimating unit 135. Specifically, the providing unit 136 generates a rotated image by rotating the image by the rotation angle (μ) clockwise with respect to the reference direction based on the rotation angle (μ) of the image estimated by the estimating unit 135. The providing unit 136 also applies AI-OCR to the generated rotated image, and provides information on character strings in the rotated image recognized by AI-OCR. For example, the providing unit 136 transmits information on character strings in the rotated image to the user's terminal device.

[0043] 3. Modifications The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0044] In the above embodiment, the case where the character string included in the image is written horizontally and the reference direction is the same as the X-axis direction has been described, but the character string included in the image may be written vertically. If the character string included in the image is written vertically, the reference direction may be the same as the Y-axis direction.

[0045] In the above embodiment, the rotation correction unit performs a process of rotating the image 90° clockwise with respect to the reference direction when the average aspect ratio of the character string region exceeds the aspect threshold. However, the rotation correction unit 133 may perform a process of rotating the image 90° clockwise with respect to the reference direction when the average aspect ratio of the character string region falls below the aspect threshold. For example, consider a case where the character string included in the image is written vertically and the reference direction is the same as the Y-axis direction. In this case, when the average of the multiple aspect ratios falls below the aspect threshold, the rotation correction unit 133 determines that the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with the aspect threshold. When the average of the multiple aspect ratios satisfies a predetermined condition based on a comparison with the aspect threshold, the rotation correction unit 133 performs a process of rotating the image 90°. For example, the rotation correction unit 133 performs a process of rotating the image 90° clockwise from the original state. In other words, the rotation correction unit 133 performs a process of rotating the image 90° clockwise with respect to the reference direction.

[0046] 4. Effects As described above, the information processing device 100 according to the embodiment includes an extraction unit 131, a calculation unit 132, a determination unit 134, and an estimation unit 135. The extraction unit 131 extracts character string region information related to a character string region, which is a rectangular region indicating the position of a character string, from an image including a character string by a machine learning model. The calculation unit 132 calculates the inclination of the character string region with respect to a reference direction for each character string region based on the character string region information. The determination unit 134 determines for each character string region whether the orientation of the character string located in the character string region is upright or inverted. The estimation unit 135 estimates the rotation angle of the character string for each character string based on the calculation result of the calculation unit 132 and the determination result of the determination unit 134.

[0047] This allows the information processing device 100 to accurately estimate the rotation angle of each character string included in the image. Therefore, the information processing device 100 can accurately estimate the rotation angle of the image based on the rotation angles of multiple character strings that have been accurately estimated.

[0048] Furthermore, calculation unit 132 calculates the inclination of at least one side of the character string region with respect to the reference direction as the inclination of the character string region with respect to the reference direction.

[0049] This allows information processing device 100 to accurately estimate the rotation angle of each character string included in the image, based on the inclination of the direction of at least one side of the character string region with respect to the reference direction.

[0050] Information processing device 100 further includes rotation correction unit 133 that performs a process of rotating an image. Calculation unit 132 calculates the aspect ratio of each character string region based on the character string region information. Rotation correction unit 133 performs a process of rotating the image by 90° when the average of a plurality of aspect ratios satisfies a predetermined condition based on a comparison with an aspect threshold value.

[0051] As a result, when the direction in which the characters included in the image are lined up is nearly perpendicular to the reference direction, the information processing device 100 can make the direction in which the characters included in the image are lined up nearly parallel to the reference direction by, for example, rotating the image 90° clockwise with respect to the reference direction. Also, since the information processing device 100 can make the direction in which the characters included in the image are lined up nearly parallel to the reference direction, it can accurately determine whether the orientation of the character string located in the character string area is upright or inverted with respect to the reference direction.

[0052] Furthermore, the estimation unit 135 estimates the rotation angle of the image based on the average of the rotation angles of a plurality of character strings, and outputs the estimated rotation angle of the image.

[0053] This enables information processing device 100 to accurately estimate the rotation angle of the image based on the average of the rotation angles of a plurality of character strings that have been accurately estimated.

[0054] Furthermore, the estimation unit 135 selects one angle from the four or more discretized angles, and outputs the selected angle as the rotation angle of the image.

[0055] This enables the information processing device 100 to roughly estimate the rotation angle of the image.

[0056] In addition, when each of a plurality of images including each of character strings tilted at a plurality of different rotation angles is input, the extraction unit 131 extracts character string region information using a machine learning model trained to output character string region information regarding each of the character strings corresponding to each of the character strings tilted at a plurality of different rotation angles.

[0057] This enables the information processing device 100 to extract character string region information with high accuracy for each character string using a machine learning model with improved robustness against rotation of the character string.

[0058] [5. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized by a computer 1000 having a configuration as shown in Fig. 5, for example. Fig. 5 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I / F) 1500, an input / output interface (I / F) 1600, and a media interface (I / F) 1700.

[0059] The CPU 1100 operates and controls each unit based on a program stored in the ROM 1300 or the HDD 1400. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, programs that depend on the hardware of the computer 1000, and the like.

[0060] The HDD 1400 stores programs executed by the CPU 1100, data used by such programs, etc. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

[0061] The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse, via the input / output interface 1600. The CPU 1100 acquires data from the input devices via the input / output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input / output interface 1600.

[0062] The media interface 1700 reads a program or data stored in the recording medium 1800 and provides it to the CPU 1100 via the RAM 1200. The CPU 1100 loads the program from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0063] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 130. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via a predetermined communication network.

[0064] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0065] [6. Other] Furthermore, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0066] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0067] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory. [Explanation of symbols]

[0068] 100 Information processing device 110 Communications Department 120 Storage section 130 Control section 131 Extraction part 132 Calculation section 133 Rotation correction section 134 Judgment section 135 Estimation Department 136 Provision Department

Claims

1. an extraction unit that extracts character string region information related to a character string region, which is a rectangular region indicating a position of the character string, from an image including the character string using a machine learning model; a calculation unit that calculates an inclination of each character string region with respect to a reference direction based on the character string region information; a determination unit that, when an image including the character string region is input, determines whether the orientation of the character string located in the character string region is upright or inverted for each of the character string regions using a machine learning model and outputs a classification result; an estimation unit that estimates a rotation angle of each of the character strings based on a calculation result of the calculation unit and a determination result of the determination unit; An information processing device comprising:

2. an extraction unit that extracts character string region information related to a character string region, which is a rectangular region indicating a position of the character string, from an image including the character string using a machine learning model; a calculation unit that calculates an inclination of each character string region with respect to a reference direction based on the character string region information; a determination unit that determines, for each character string region, whether a character string located within the character string region is oriented upright or inverted; an estimation unit that estimates a rotation angle of each of the character strings based on a calculation result of the calculation unit and a determination result of the determination unit; Equipped with The estimation unit selects one angle from four or more discretized angles based on the calculated rotation angles of the inclinations of the plurality of character strings, and outputs the selected one angle as a rotation angle of the image. Information processing device.

3. The estimation unit estimates, for each of the character strings, if the character string is determined to be in the upright state, a rotation angle of the calculated inclination as the rotation angle of the character string, and if the character string is determined to be in the inverted state, a rotation angle of the character string that is a rotation angle obtained by adding 180° to the calculated inclination as the rotation angle of the character string. The information processing device according to claim 1 .

4. The calculation unit is As the inclination of the character string region with respect to the reference direction, the inclination of at least one side of the character string region with respect to the reference direction is calculated.

3. The information processing device according to claim 1 or 2.

5. A rotation correction unit that performs processing to rotate the image is further provided. The calculation unit is calculating an aspect ratio of each of the character string regions based on the character string region information; The rotation correction unit is If the average of the aspect ratios satisfies a predetermined condition based on a comparison with an aspect threshold, the image is rotated by 90 degrees.

3. The information processing device according to claim 1 or 2.

6. The estimation unit is A rotation angle of the image is estimated based on an average of the rotation angles of the plurality of character strings, and the estimated rotation angle of the image is output.

3. The information processing device according to claim 1 or 2.

7. The estimation unit is One angle is selected from the four or more discretized angles, and the selected angle is output as the rotation angle of the image. The information processing device according to claim 5 .

8. The extraction unit is When a plurality of images each including a character string tilted at a plurality of different rotation angles are input, the machine learning model is trained to output character string region information regarding a character string region corresponding to each of the character strings tilted at the plurality of different rotation angles, and extracts the character string region information by the machine learning model.

3. The information processing device according to claim 1 or 2.

9. An information processing method implemented by a program executed by an information processing device, comprising: An extraction step of extracting character string region information related to a character string region, which is a rectangular region indicating a position of the character string, from an image including the character string using a machine learning model; a calculation step of calculating an inclination of each character string region with respect to a reference direction based on the character string region information; a determining step of, when an image including the character string region is input, determining, for each of the character string regions, whether the orientation of the character strings located within the character string region is upright or inverted, using a machine learning model that outputs a classification result; an estimation step of estimating a rotation angle of each of the character strings based on a calculation result of the calculation step and a determination result of the determination step; An information processing method comprising:

10. An information processing method implemented by a program executed by an information processing device, comprising: An extraction step of extracting character string region information related to a character string region, which is a rectangular region indicating a position of the character string, from an image including the character string using a machine learning model; a calculation step of calculating an inclination of each character string region with respect to a reference direction based on the character string region information; a determining step of determining, for each character string region, whether the orientation of the character string located within the character string region is upright or inverted; an estimation step of estimating a rotation angle of each of the character strings based on a calculation result of the calculation step and a determination result of the determination step; Equipped with The estimation step selects one angle from four or more discretized angles based on the calculated rotation angles of the inclinations of the plurality of character strings, and outputs the selected angle as the rotation angle of the image. Information processing methods.

11. An extraction step of extracting character string region information regarding a character string region, which is a rectangular region indicating a position of the character string, from an image including the character string using a machine learning model; a calculation step of calculating an inclination of each character string region with respect to a reference direction based on the character string region information; a determination step in which, when an image including the character string region is input, a machine learning model is used to determine whether the orientation of the character string located within the character string region is upright or inverted for each of the character string regions and output a classification result; an estimation step of estimating a rotation angle of each of the character strings based on a calculation result of the calculation step and a determination result of the determination step; An information processing program that causes a computer to execute the above.

12. An extraction step of extracting character string region information regarding a character string region, which is a rectangular region indicating a position of the character string, from an image including the character string using a machine learning model; a calculation step of calculating an inclination of each character string region with respect to a reference direction based on the character string region information; a determination step of determining, for each character string region, whether a character string located within the character string region is in an upright state or an inverted state; an estimation step of estimating a rotation angle of each of the character strings based on a calculation result of the calculation step and a determination result of the determination step; on the computer, The estimation step includes selecting one angle from four or more discretized angles based on the calculated rotation angles of the slant of the plurality of character strings, and outputting the selected angle as a rotation angle of the image. Information processing program.

Images