Information processing device

Abstract

To provide an information processor capable of improving the character recognition accuracy of a character string arranged in a curved shape.SOLUTION: An information processor 100 includes: an IF 105 or a network IF 106 to which an image of an object (cylinder β) having a curved surface on which a character string arranged in a curved shape is described; and a CPU 101 that detects an ellipse E1 passing through a plurality of upper end points of the character string from the image and an ellipse E4 passing through a plurality of lower end points of the character string from the image, and converts the image of the curved surface of the object into a plane image α on the basis of center coordinates of the ellipse E1 and the ellipse E4, radii A1 and A2 of major axes of the ellipse E1 and the ellipse E4, and radii B1 and B2 of minor axes of the ellipse E1 and the ellipse E4.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Conventionally, techniques for detecting character strings aligned in the X-axis direction or the Y-axis direction contained in a captured image have been known (see, for example, Patent Documents 1 to 5). Also known is a technique for rotating a character string tilted relative to the X-axis or Y-axis direction so that it is parallel to the X-axis or Y-axis direction, and reading the rotated character string. Furthermore, a technique for performing image processing such as edge extraction before character recognition is known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-103755 [Patent Document 2] Japanese Patent Application Publication No. 7-121658 [Patent Document 3] Japanese Patent Application Publication No. 9-237318 [Patent Document 4] Japanese Patent Application Publication No. 2019-128727 [Patent Document 5] Japanese Patent Publication No. 2022-52716 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, if two lines of text are written on a rectangular piece of paper, the paper is wrapped around a cylinder, and an image of the cylinder is taken, the characters will be lower in alignment, the characters closer to the left and right edges will be smaller, and the space between the characters will be narrower. In this state, character recognition may not be able to accurately read each character in the text.

[0005] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an information processing device that can improve the accuracy of character recognition of character strings arranged in a curved shape. [Means for solving the problem]

[0006] The information processing device of the present invention is characterized by comprising an input means for inputting an image of an object having a curved surface on which a character string arranged in a curved shape is written, a detection means for detecting a first ellipse that passes through multiple upper endpoints of the character string from the image and a second ellipse that passes through multiple lower endpoints of the character string from the image, and a conversion means for converting the image of the curved surface of the object into an image of a plane based on the central coordinates of the first ellipse and the second ellipse, the radii of the major axes of the first ellipse and the second ellipse, and the radii of the minor axes of the first ellipse and the second ellipse.

[0007] In the above configuration, the object is a cylinder, and the conversion means defines a difference between the Y coordinates of the center coordinates of the first ellipse and the second ellipse as a length H in the Y direction of the image of the plane, defines half the length of the perimeter of either the first ellipse or the second ellipse as a length W in the X direction of the image of the plane, defines the lower right end of the image of the plane as the origin, defines upward and leftward directions as positive, defines the coordinates of an arbitrary point R in the image of the plane as (x, y), and defines an angle by which the radii of the major axes of the first ellipse and the second ellipse rotate clockwise from the horizontal direction as θ, calculates the angle θ from the coordinate x of the point R and the length W, and defines the upper and lower endpoints of a vertical line segment passing through the point R as points P , and point Q, the coordinates of points P' and Q' on the cylinder corresponding to the coordinates of point P and point Q may be calculated based on the central coordinates of the first ellipse and the second ellipse, the radius of the major axis, the radius of the minor axis, and the angle θ, the coordinates of the point that divides the Y coordinates of points P' and Q' internally into y:(Hy) may be set as the Z coordinate of point R' on the cylinder corresponding to the coordinate of point R, and the X coordinate and Y coordinate of either point P' or point Q' may be set as the X coordinate and Y coordinate of point R', thereby calculating the X coordinate, Y coordinate, and Z coordinate of point R', and the color of point R' may be detected from surrounding pixels by bilinear interpolation, and the color of point R' may be set as the color of point R.

[0008] In the above configuration, when the character string is made up of multiple lines of character strings, the first ellipse may be an ellipse that passes through multiple upper endpoints of a topmost character string among the multiple lines of character strings, and the second ellipse may be an ellipse that passes through multiple lower endpoints of a bottommost character string among the multiple lines of character strings.

[0009] The information processing device according to the present invention may acquire the result of character recognition of a character string by using a learning model that receives an input of a captured image of a cylinder having a curved surface on which a character string arranged in a curved shape is written, and outputs the result of character recognition of the character string. The learning model is generated by using a known machine learning algorithm such as a neural network. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the accuracy of character recognition of a character string arranged in a curved shape. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing an information processing device according to the present embodiment. [Figure 2] FIG. 2 is a flowchart showing the processing executed by the information processing device. [Figure 3] 3A and 3B are explanatory diagrams of the ellipse fitting process in step S8 of FIG. [Figure 4] FIG. 4 is an explanatory diagram of the image conversion process in step S9 of FIG. [Figure 5] FIG. 5 is a diagram showing an example of a character string after the image conversion process in step S9 of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] FIG. 1 is a block diagram showing an information processing device according to this embodiment. The information processing device 100 in FIG. 1 includes a CPU (Central Processing Unit) 101 as a processor, a RAM (Random Access Memory) 102 and a ROM (Read Only Memory) 103 as memories, a HDD (Hard Disk Drive) 104, an interface (IF) 105, and a network interface (IF) 106. The information processing device 100 can be realized by, for example, a computer, a server, or a smartphone. The CPU 101 is connected to the RAM 102, the ROM 103, the HDD 104, the IF 105, and the network IF 106 via a bus 107. The IF 105 or the network IF 106 functions as an input unit, and the CPU 101 functions as a detection unit and a conversion unit.

[0014] The IF 105 is, for example, a USB (Universal Serial Bus), and is connected to a camera 110 serving as an imaging device. In this embodiment, the camera 110 is connected to the outside of the information processing device 100, but may also be built into the information processing device 100. The network IF 106 is an interface for connecting the information processing device 100 to an external network. The camera 110 may be connected to the information processing device 100 via the network IF 106.

[0015] The HDD 104 stores images captured by the camera 110 and programs for executing various types of image processing and character recognition processing. These programs correspond to the flowcharts described below. The HDD 104 also stores the results of various types of image processing and character recognition processing. An SSD (Solid State Drive) may be used instead of the HDD 104.

[0016] 2 is a flowchart showing the processing executed by the information processing device 100. The processing in FIG.

[0017] First, an image captured by camera 110 is input (step S1). In this embodiment, the object captured by camera 110 is a cylinder with a curved surface on which a character string arranged in a curved shape is written. The character string includes letters, numbers, signs, symbols, etc. The curved cylinder is, for example, a component such as a rotating shaft, a bolt, or a plug. For example, the character string arranged in a curved shape is a character string such as a product name or model number written in a straight line along the curved surface of the cylinder. In the captured image of the cylinder, the character string is arranged in a curved shape, with the arrangement of the character string descending or ascending as it approaches the left or right end (see, for example, FIG. 3(A)).

[0018] Next, CPU 101 detects an object from the captured image (step S2) and extracts a curved surface area where a character string arranged in a curved shape is written (step S3). Next, CPU 101 performs a process to remove the α channel from the image of the area extracted in S3 (step S4). The α channel is auxiliary data provided for each pixel in addition to color representation data, and generally represents the opacity of the pixel.

[0019] Next, CPU 101 performs binarization processing on the image after the alpha channel removal (step S5), and performs processing to enlarge or reduce the line width of the character string included in the image after the binarization processing (step S6). If the line width of the character string is larger than the range of a preset threshold, the line width of the character string is reduced so that it falls within the range of the preset threshold, and if the line width of the character string is smaller than the range of the preset threshold, the line width of the character string is enlarged so that it falls within the range of the preset threshold.

[0020] Next, CPU 101 removes the outer frame corresponding to the outer shape of the object from the image after the line width enlargement or reduction process (step S7). The outer frame corresponding to the outer shape of the object is removed because it is information unnecessary for character recognition.

[0021] Next, CPU 101 performs ellipse fitting processing on the image after outer frame removal, that is, the image of the character string arranged in a curved shape (step S8). Details of the ellipse fitting processing will be described later.

[0022] Next, based on the information on the multiple ellipses detected in the ellipse fitting process, CPU 101 executes an image conversion process to convert the image of the curved surface of the object into a flat image so that the character strings written on the curved surface of the object, which are arranged in a curved manner, are arranged in a straight line (step S9). The image conversion process will be described in detail later.

[0023] Next, CPU 101 performs optical character recognition processing on the planar image after the image conversion processing (step S10), outputs the processing result of the optical character recognition processing (step S11), and ends the processing in Fig. 2. The processing result of the optical character recognition processing may be stored in HDD 104, or may be output to an external device (not shown) via network IF 106 or IF 105.

[0024] The processing of steps S2 to S10 in Fig. 2 may be performed by machine learning. In this case, the information processing device 100 receives, as input, a captured image of a cylinder having a curved surface on which character strings arranged in a curved manner are written, and acquires the result of character recognition of the character string using a learning model that outputs the result of character recognition of the character string. The learning model is generated using a known machine learning algorithm such as a neural network. The training data to be input to the machine learning algorithm can be created, for example, by using the camera 110 to capture images of a cylinder having a curved surface on which character strings arranged in a curved manner are written under various conditions, and linking the result of character recognition of the character string to the captured image.

[0025] Figures 3(A) and (B) are explanatory diagrams of the ellipse fitting process in step S8 in Figure 2. As shown in Figure 3(A), it is assumed that two lines of text, "ABCD 1234-EF," are written on the curved surface of a cylinder.

[0026] In the ellipse fitting process, CPU 101 detects each of the blocks "A," "B," "C," "D," "1," "2," "3," "4," "E," and "F," and extracts the top and bottom endpoints of each block. The state where the top and bottom endpoints of each block have been extracted is shown in the right diagram of FIG. 3(A).

[0027] If two lines of text, "ABCD 1234-EF," are written on the curved surface of a cylinder, four point clouds will be created: the multiple upper endpoints of "ABCD" on the first line, the multiple lower endpoints of "ABCD" on the first line, the multiple upper endpoints of "1234-EF" on the second line, and the multiple lower endpoints of "1234-EF" on the second line. However, the "-" is omitted as a connection, resulting in six point clouds, separated by "1234" and "EF." Additionally, flat parts such as "E" and "F" have multiple endpoints instead of just one.

[0028] Next, as shown in FIG. 3B, CPU 101 detects ellipses E1 to E4 from the point cloud of the multiple upper endpoints of "ABCD" on the first line, the point cloud of the multiple lower endpoints of "ABCD" on the first line, the point cloud of the multiple upper endpoints of "1234-EF" on the second line, and the point cloud of the multiple lower endpoints of "1234-EF" on the second line. Ellipse E1 passes through the point cloud of the multiple upper endpoints of "ABCD" on the first line, ellipse E2 passes through the point cloud of the multiple lower endpoints of "ABCD" on the first line, ellipse E3 passes through the point cloud of the multiple upper endpoints of "1234-EF" on the second line, and ellipse E4 passes through the point cloud of the multiple lower endpoints of "1234-EF" on the second line. CPU 101 performs ellipse fitting on each point cloud and finds the optimal value of the ellipse using the steepest descent method.

[0029] 3(B), CPU 101 detects four ellipses E1 to E4 of the same size, aligns the center points of ellipses E1 to E4 vertically, and fixes the top, bottom, left, and right of ellipses E1 to E4. This ellipse detection makes it possible to grasp the tilt of the cylinder.

[0030] The ellipse used for ellipse fitting is given by the following equation (1). g(x,y)=ax 2 +bxy+cy 2+dx+ey+f (1) D=4ac-b 2 If g(x,y)=0 and g(x,y)=0, then x,y are points on the ellipse.

[0031] In ellipse fitting, Σ[g(x,y)] 2 By finding the coefficients a, b, c, d, and e that minimize this, the equation for the ellipse that passes closest to the point cloud can be obtained. In ellipse fitting, simultaneous fitting is performed so that the x-coordinate of the center and the major axis of multiple ellipses in the following equations (2) and (3) are the same. Note that if the major and minor axes of each ellipse are aligned with the x-axis and y-axis, respectively, the coefficient b = 0. Ellipse E1 a1x 2 +c1y 2 +d1x+e1y+f1=0 (2) Oval Em amx 2 +cmy 2 +dmx+emy+fm=0 (m=2 or greater integer) (3)

[0032] As an ellipse fitting method, for example, the following document 1 can be referred to. Reference 1: Andrew Fitzgibbon, Maurizio Pilu, “Direct linear least squares fitting of an ellipse”, November 2000, IEEE Transactions on Pattern Analysis and Machine Intelligence 21(5)

[0033] 3B, CPU 101 detects ellipses E1 to E4 from each of the four point groups, but if the character string consists of multiple lines of character strings, CPU 101 may detect ellipse E1 that passes through multiple upper endpoints of the topmost character string of the multiple lines of character strings, and ellipse E4 that passes through multiple lower endpoints of the bottommost character string of the multiple lines of character strings. In this case, the number of detected ellipses can be reduced, thereby reducing the amount of calculation by CPU 101 and shortening processing time.

[0034] Fig. 4 is an explanatory diagram of the image conversion processing of step S9 in Fig. 2. Fig. 5 is a diagram showing an example of a character string after the image conversion processing of step S9 in Fig. 2. First, it is assumed that CPU 101 has previously detected the coordinates (Cx1, Cy1) of the center point C1 of ellipse E1, the coordinates (Cx2, Cy2) of the center point C2 of ellipse E4, the radius A1 of the major axis of ellipse E1, the radius A2 of the major axis of ellipse E4, the radius B1 of the minor axis of ellipse E1, and the radius B2 of the minor axis of ellipse E4 in the ellipse fitting processing before the image conversion processing.

[0035] In the image conversion process, as shown in FIG. 4, the CPU 101 converts the image of the curved surface of the object (cylinder β) into a planar image α based on the coordinates of the center points C1 and C2 of the ellipses E1 and E4 detected in the ellipse fitting process, the radii A1 and A2 of the major axes of the ellipses E1 and E4, and the radii B1 and B2 of the minor axes of the ellipses E1 and E4, so that the character strings arranged in a curved manner on the curved surface of the object (cylinder β) are arranged in a straight line.

[0036] The CPU 101 calculates the difference (=Cy1-Cy2) between the Y coordinates of the center points C1 and C2 of the ellipses E1 and E4 as the length H in the Y direction of the planar image α, and calculates half the perimeter of either the ellipse E1 or the ellipse E4 as the length W in the X direction of the planar image α. The lengths H and W are rounded off to integers. For example, the half perimeters of the ellipses E1 and E4 can be calculated using the following equations (4) and (5), respectively. Half the perimeter of ellipse E1 = π[(3(A1+B1)-√(A1+3B1)(3A1+B1))] / 2 (4) Half the perimeter of ellipse E4 = π[(3(A2+B2)-√(A2+3B2)(3A2+B2))] / 2 (5)

[0037] The lower right corner of the planar image α is set as the origin O, the upward and leftward directions are set as positive directions, and the coordinates of an arbitrary point R on the planar image α are set as (x, y).

[0038] Next, CPU 101 obtains the color of point R(x, y) according to the following procedures (A) to (D). (A) When the angle at which the radii A1 and A2 of the major axes of the ellipse E1 and the ellipse E4 rotate clockwise from the horizontal direction is θ, the CPU 101 calculates the angle θ from the coordinate x and length W of the point R. Since the object is a cylinder β, the length W can be expressed as W = πr (r is the radius of the circle), and the coordinate x of point R can be expressed as x = 2πr × θ / 2π = rθ. Therefore, the angle θ can be expressed as θ = x / r = x / (W / π) = πx / W. Therefore, the angle θ can be calculated from the coordinate x of point R and the length W.

[0039] (B) If the upper and lower endpoints of a vertical line segment passing through point R are points P and Q, CPU 101 calculates the coordinates of points P' and Q' of ellipses E1 and E4 corresponding to the coordinates of point P and point Q based on the coordinates of center points C1 and C2 of ellipses E1 and E4, radii A1 and A2 of the major axes of ellipses E1 and E4, radii B1 and B2 of the minor axes of ellipses E1 and E4, and angle θ. The coordinates of point P' are (Cx1+A1cosθ, Cy1-B1sinθ), and the coordinates of point Q' are (Cx2+A2cosθ, Cy2-B2sinθ).

[0040] (C) The CPU 101 calculates the X, Y, and Z coordinates of point R' by setting the coordinates of the point that divides the Y coordinates of points P' and Q' internally in the ratio y:(Hy) as the Z coordinate of point R' on the cylinder β that corresponds to the coordinates of point R, and setting the X and Y coordinates of either point P' or point Q' as the X and Y coordinates of point R'. For example, the X, Y and Z coordinates of point R' can be expressed as (Cx2+A2cosθ, Cy2-B2sinθ, y).

[0041] (D) CPU 101 detects the color of point R' from the four surrounding pixels using bilinear interpolation, and sets the color of point R' as the color of point R. For example, the four pixels are the upper right, lower right, upper left, and lower left pixels adjacent to point R'. Bilinear interpolation is a well-known interpolation method using the four surrounding pixels, so a detailed description thereof will be omitted.

[0042] By repeating the above steps (A) to (D) for all points on the planar image α, the image of the curved surface of the object (cylinder β) can be converted into the planar image α. As a result, the character strings arranged in a curved shape on the curved surface in Figure 3(A) are arranged in a straight line on the planar image α as shown in Figure 5.

[0043] As described above, according to this embodiment, the information processing device 100 includes an IF 105 or a network IF 106 that receives an image of an object having a curved surface on which character strings arranged in a curved manner are written, and a CPU 101 that detects an ellipse E1 (first ellipse) that passes through multiple upper endpoints of the character string from the image and an ellipse E4 (second ellipse) that passes through multiple lower endpoints of the character string from the image, and converts the image of the curved surface of the object into a planar image α based on the center coordinates of the ellipse E1 and the ellipse E4, the radii A1 and A2 of the major axes of the ellipse E1 and the ellipse E4, and the radii B1 and B2 of the minor axes of the ellipse E1 and the ellipse E4. This allows the image of the curved surface of the object to be converted into the planar image α. Furthermore, the character strings arranged in a curved manner on the curved surface of the object in FIG. 3A are arranged linearly on the planar image α as shown in FIG. 5, thereby improving the accuracy of character recognition of character strings arranged in a curved manner.

[0044] The object is a cylinder, and the CPU 101 defines the difference between the Y coordinates of the centers of the ellipses E1 and E4 as the length H of the planar image α in the Y direction, defines half the perimeter of either the ellipse E1 or the ellipse E4 as the length W of the planar image α in the X direction, defines the lower right end of the planar image α as the origin, defines upward and leftward directions as positive, and defines the coordinates of an arbitrary point R on the planar image α as (x, y). When the angle at which the radius of the major axis of the planar image α rotates clockwise from the horizontal is defined as θ, the CPU 101 calculates the angle θ from the coordinate x of point R and the length W, and, assuming that the upper and lower endpoints of a vertical line segment passing through point R are points P and B2, the CPU 101 calculates the coordinates of points P' and Q' on the cylinder corresponding to the coordinates of points P and Q based on the center points C1 and C2 of the ellipses E1 and E4, the radii A1 and A2 of the major axes, the radii B1 and B2 of the minor axes, and the angle θ. Furthermore, CPU 101 sets the coordinates of the point that divides the Y coordinates of points P' and Q' internally in the equation y:(Hy) as the Z coordinate of point R' on the cylinder corresponding to the coordinates of point R, and sets the X coordinate and Y coordinate of either point P' or point Q' as the X coordinate and Y coordinate of point R' to calculate the X coordinate, Y coordinate and Z coordinate of point R', detects the color of point R' from surrounding pixels by bilinear interpolation, and sets the color of point R' as the color of point R. In this way, the image of the curved surface of the object can be converted into a planar image α.

[0045] Although the preferred embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0046] 100 Information processing device 101 CPU 102 RAM 103 ROM 104 HDD 105 Interface 106 Network Interface 107 Bus 110 Camera

Claims

1. an input means for inputting an image of an object having a curved surface on which character strings arranged in a curved shape are written; a detection means for detecting a first ellipse passing through a plurality of upper endpoints of the character string from the image and a second ellipse passing through a plurality of lower endpoints of the character string from the image; a conversion means for converting an image of the curved surface of the object into an image of a plane based on the central coordinates of the first ellipse and the second ellipse, the radii of the major axes of the first ellipse and the second ellipse, and the radii of the minor axes of the first ellipse and the second ellipse; An information processing device comprising:

2. the object is a cylinder, The conversion means a difference between the Y coordinates of the centers of the first ellipse and the second ellipse is defined as a length H in the Y direction of the image of the plane; a length W in the X direction of the image of the plane is set to half the perimeter of either the first ellipse or the second ellipse; The lower right corner of the image of the plane is set as the origin, and the upward and leftward directions are set as positive directions. The coordinates of an arbitrary point R in the image of the plane are set as (x, y), When the angle at which the radii of the major axes of the first ellipse and the second ellipse rotate clockwise from the horizontal direction is defined as θ, the angle θ is calculated from the coordinate x of the point R and the length W; When the upper and lower end points of a vertical line segment passing through the point R are defined as points P and Q, the coordinates of points P' and Q' on the cylinder corresponding to the coordinates of the point P and the point Q are calculated based on the central coordinates of the first ellipse and the second ellipse, the radius of the major axis, the radius of the minor axis, and the angle θ; The coordinates of the point that divides the Y coordinates of the points P' and Q' internally into y:(H-y) are set as the Z coordinate of the point R' on the cylinder that corresponds to the coordinates of the point R, and the X coordinate and Y coordinate of either the point P' or the point Q' are set as the X coordinate and Y coordinate of the point R', thereby calculating the X coordinate, Y coordinate and Z coordinate of the point R'; The color of the point R' is detected from the surrounding pixels by bilinear interpolation, and the color of the point R' is set as the color of the point R.

2. The information processing apparatus according to claim 1, wherein:

3. 3. The information processing device according to claim 1, wherein, when the character string is composed of multiple lines of character strings, the first ellipse is an ellipse that passes through multiple upper endpoints of the topmost character string of the multiple lines of character strings, and the second ellipse is an ellipse that passes through multiple lower endpoints of the bottommost character string of the multiple lines of character strings.

Images