Character detection device, learning device, and program
The character detection device optimizes neural networks using non-differentiable processes by integrating character detection errors and estimated final scores, enhancing detection accuracy and precision.
Patent Information
- Application Number
- JP2021009893
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-01-25
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-01-25
AI Technical Summary
Conventional character detection techniques using neural networks face challenges in direct optimization due to non-differentiable processing, such as NMS and final score calculation, leading to suboptimal learning outcomes.
A character detection device and learning device that integrates a character detection unit, an integration processing unit, a final score estimating unit, and a final score error calculation unit to directly optimize neural network parameters based on non-differentiable processing, using character detection errors and estimated final scores for learning.
Enables effective model learning and accurate character detection by optimizing neural networks through gradient methods, even with non-differentiable processes, improving detection accuracy and precision.
Smart Images

Figure 0007584312000001 
Figure 0007584312000002 
Figure 0007584312000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a character detection device, a learning device, and a program. [Background technology]
[0002] A technology for detecting characters contained in an image based on the image has been developed and is being used. For example, Non-Patent Document 1 describes a technology for detecting character regions in an image using a neural network.
[0003] In a process for detecting characters in an image using a neural network using conventional technology, first, a convolutional neural network (CNN) is used to determine whether each pixel in the image is included in a character region. In the next process, the shape of the character region is estimated to detect characters within the character region. The accuracy of character region detection is evaluated based on how many regions are detected where the IOU (Intersection over Union, the ratio of the common region to the union region) between the estimated character region shape and the correct character region shape exceeds a threshold.
[0004] In the technology described in Non-Patent Document 1, five values are calculated for each pixel in an image using CNN. The five values output from the CNN are the character region score value of the pixel and four values representing the rectangular shape. The character region score is a real value greater than or equal to 0 and less than or equal to 1. The four values representing the rectangular shape have information on the coordinate values of the four vertices for identifying the rectangle. Specifically, when the four sides of the rectangle are parallel to each coordinate axis in an XY orthogonal coordinate system, the four values are the difference between the X coordinate value of the pixel that outputs the value and the minimum and maximum values of the X coordinate of the rectangular region, and the difference between the Y coordinate of the pixel that outputs the value and the minimum and maximum values of the Y coordinate of the rectangular region. In other words, the four values are (1) the absolute value of the difference between the X coordinate of the target pixel and the maximum X coordinate of the rectangular region, (2) the absolute value of the difference between the X coordinate of the target pixel and the minimum X coordinate of the rectangular region, (3) the absolute value of the difference between the Y coordinate of the target pixel and the maximum Y coordinate of the rectangular region, and (4) the absolute value of the difference between the Y coordinate of the target pixel and the minimum Y coordinate of the rectangular region. The rectangular shape output (the above four values) of a pixel whose character region score exceeds a predetermined threshold represents the estimated character region. Then, the estimated character region is refined from a four-point rectangle to a polygonal polygon using a recurrent neural network (RNN), to output an accurate character region.
[0005] However, one character region contains many pixels. Therefore, when the above character region is output for each pixel, several overlapping character regions are actually output. In order to eliminate such overlaps, the conventional technology performs NMS processing. This allows the detected character regions to be organized and consolidated by excluding all but the region with the maximum score from the many overlapping estimated character regions. [Prior art documents] [Non-patent literature]
[0006] [Non-Patent Document 1] Xiaobing Wang, Yingying Jiang, Zhenbo Luo, Cheng-Lin Liu, Hyunsoo Choi, Sungjin Kim, Arbitrary Shape Scene Text Detection With Adaptive Text Region Representation, CVPR2019, 2019 Conference on Computer Vision and Pattern Recognition, arXiv:1905.05980v1, June 2019, [Downloaded November 11, 2020], Internet<URL:https: / / arxiv.org / pdf / 1905.05980.pdf> Summary of the Invention [Problem to be solved by the invention]
[0007] However, the conventional technology has the following problems. The conventional technology uses a neural network, and the learning of the neural network involves defining a loss function and adjusting the internal parameters to minimize the loss. The gradient descent method is generally used to minimize the loss when adjusting the internal parameters of a neural network. The gradient descent method is a method of searching for a set of parameter values that realizes the minimum value by utilizing the fact that the gradient (i.e., the differential value) of the function output in a multidimensional space formed by many parameters becomes 0 when the output of the loss function takes the minimum value. Therefore, the function that calculates the loss from the input to the neural network (i.e., the image) must be a differentiable function.
[0008] However, the NMS process and the process of calculating the final score used in the technology described in Non-Patent Document 1 are not differentiable functions. Therefore, in the technology of Non-Patent Document 1, the estimation of the parameter values of the CNN is performed based on the output of the stage before the NMS process (i.e., the output of a large number of rectangles including non-maximum scores). In other words, in the conventional technology, the final score (score of the area with a high IOU value) is only indirectly used for learning. In other words, direct learning based on the final score is not performed. For example, a situation may occur in which the loss worsens during learning but the IOU value improves, but the conventional technology has a problem in that optimal learning cannot be performed in such a case.
[0009] We have described above the problems with the technology described in Non-Patent Document 1. In addition, not only in Non-Patent Document 1 but also in other character region detection technologies, the calculation of IOU for calculating the final score is generally non-differentiable, so that direct optimization of the neural network based on the final score is also impossible.
[0010] The present invention has been made based on the recognition of the above problem, and aims to provide a character detection device, a learning device, and a program that can directly optimize a model based on training data, even in cases where non-differentiable processing (input / output relationships) is involved. [Means for solving the problem]
[0011] [1] In order to solve the above problem, a character detection device according to one embodiment of the present invention includes a character detection unit that inputs an image and outputs features related to character areas contained in the image; an integration processing unit that determines character areas by integrating overlapping character area candidates detected by the character detection unit and calculates a final score for all of the determined character areas based on ground-truth data; a final score estimation unit that calculates an estimated final score, which is an estimate of the final score, based on the features output by the character detection unit; and a final score error calculation unit that calculates a final score error, which is the error between the final score and the estimated final score, wherein the final score estimation unit adjusts internal parameters of a final score estimation model held by the final score estimation unit based on the final score error for learning, and the character detection unit adjusts internal parameters of the character detection model held by the character detection unit based on a character detection error, which is the error between the features and ground-truth data, and the estimated final score, for learning.
[0012] [2] Also, one aspect of the present invention is that in the above-mentioned character detection device, the features include, corresponding to a pixel contained in the image, a character area score indicating the degree to which the pixel is included in a character area, and position information indicating the position within the image of the character area including the pixel.
[0013] [3] Also, in one aspect of the present invention, in the above-mentioned character detection device, the integration processing unit integrates the overlapping character region candidates by performing non-maximum suppression processing based on the character region scores, and also performs processing to convert the rectangular character region after integration into a finer polygonal region.
[0014] [4] Also, one aspect of the present invention is that in the above-mentioned character detection device, the character detection unit adjusts internal parameters of the character detection model held by the character detection unit based only on the character detection error in a first stage of learning, and then adjusts the internal parameters of the character detection model held by the character detection unit based on both the character detection error and the estimated final score in a second stage of learning.
[0015] [5] Moreover, a learning device according to one embodiment of the present invention includes a character detection unit that inputs an image and outputs features related to character regions contained in the image; an integration processing unit that determines character regions by integrating overlapping character region candidates detected by the character detection unit and calculates a final score for all of the determined character regions based on ground-truth data; a final score estimation unit that calculates an estimated final score, which is an estimate of the final score, based on the features output by the character detection unit; and a final score error calculation unit that calculates a final score error, which is the error between the final score and the estimated final score, wherein the final score estimation unit adjusts internal parameters of a final score estimation model held by the final score estimation unit based on the final score error for learning, and the character detection unit adjusts internal parameters of the character detection model held by the character detection unit based on a character detection error, which is the error between the features and ground-truth data, and the estimated final score, for learning.
[0016] [6] Furthermore, a character detection device according to one embodiment of the present invention includes a character detection unit that inputs an image and outputs features related to character regions contained in the image, and an integration processing unit that determines character regions by integrating overlapping character region candidates detected by the character detection unit, and the character detection unit calculates the features based on a character detection model trained by the learning device described in [5] above.
[0017] [7] Moreover, one aspect of the present invention is a program for causing a computer to function as a character detection device, comprising: a character detection unit that inputs an image and outputs features related to character areas contained in the image; an integration processing unit that determines character areas by integrating overlapping character area candidates detected by the character detection unit and calculates a final score for all of the determined character areas based on ground-truth data; a final score estimation unit that calculates an estimated final score that is an estimate of the final score based on the features output by the character detection unit; and a final score error calculation unit that calculates a final score error that is the error between the final score and the estimated final score, wherein the final score estimation unit adjusts internal parameters of a final score estimation model held by the final score estimation unit based on the final score error for learning, and the character detection unit adjusts internal parameters of the character detection model held by the character detection unit based on a character detection error that is the error between the features and ground-truth data and the estimated final score for learning.
[0018] [8] Another aspect of the present invention is a program for causing a computer to function as a character detection device, comprising: a character detection unit that inputs an image and outputs features related to character regions contained in the image; and an integration processing unit that determines character regions by integrating overlapping character region candidates detected by the character detection unit, wherein the character detection unit calculates the features based on a character detection model trained by the learning device of [5]. Effect of the Invention
[0019] According to the present invention, it is possible to perform model learning using an effect gradient method or the like for character detection processing within an image that includes non-differentiable processing. [Brief description of the drawings]
[0020] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a character detection device according to an embodiment of the present invention. [Diagram 2]13 is a flowchart showing an operation procedure in a learning mode of the character detection device according to the embodiment. [Diagram 3] 13 is a flowchart showing an operation procedure in an estimation mode of the character detection device according to the embodiment. [Figure 4] 4 is an external force diagram showing an example of the configuration of a computer inside the character detection device according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0021] Next, an embodiment of the present invention will be described with reference to the drawings. The character detection device 1 of this embodiment is characterized by having a neural network (final score estimation unit 22 described below) for estimating a final score based on the output from a neural network for character detection (character detection unit 12 described below) and intermediate feature amounts. The character detection device 1 calculates the loss of the character detection neural network based on the output from the neural network for estimating the final score, and optimizes the neural network for character detection based on the loss.
[0022] FIG. 1 is a block diagram showing a schematic functional configuration of a character detection device according to the present embodiment. As shown in the figure, the character detection device 1 includes an image acquisition unit 11, a character detection unit 12, an output data storage unit 13, a character detection error calculation unit 14, an integration processing unit 21, a final score estimation unit 22, a final score error calculation unit 24, a control unit 31, and a learning data supply unit 32. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Also, non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used as necessary. Also, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.
[0023] The character detection device 1 is a device that detects character regions in an image using a machine learning technique. The character detection device 1 operates in two modes: a learning mode and an estimation mode. In the learning mode, the character detection device 1 learns an internal model based on learning data. That is, the character detection device 1 adjusts the values of internal parameters. In the estimation mode, the character detection device 1 executes a process of estimating the position of a character region included in an unknown image using a trained model. The control unit 31, which will be described later, controls whether the character detection device 1 operates in the learning mode or the estimation mode. Next, the functions of each unit constituting the character detection device 1 will be described.
[0024] The image acquisition unit 11 acquires an image to be processed. When the character detection device 1 operates in a learning mode, the image acquisition unit 11 acquires an image for learning provided by the learning data supply unit 32. When the character detection device 1 operates in an estimation mode, the image acquisition unit 11 acquires an image for character region detection, for example, from outside. The image acquisition unit 11 passes the acquired image to the character detection unit 12.
[0025] The character detection unit 12 obtains a feature amount related to character detection in an input image based on the input image. That is, the character detection unit 12 outputs a feature amount related to a character area included in the input image. The character detection unit 12 includes a neural network (character detection model) therein. The structure of this neural network may be the same as a general network for object detection in an image. For example, the network structure may be VGG (a network name derived from Visual Geometry Group). VGG has a structure in which a feature map of a general object recognition network is enlarged and superimposed and output as an image feature map. As the network structure, a network for object recognition such as ResNet (Residual Network) or Inception may be used.
[0026] The character detection unit 12 calculates a feature amount for each pixel by the above-mentioned neural network. This feature amount is a feature amount related to the detection of a character area. The input image can be expressed as a 3×H×W matrix. Here, 3 is the number of channels of pixel values (for example, channels corresponding to the three primary colors of RGB). H is the number of pixels in the height direction of the image. W is the number of pixels in the width direction of the image. The number of channels of pixel values may be other than 3. The feature amount calculated by the character detection unit 12 can be expressed as a C×H×W matrix. Here, C is the number of channels that is determined in advance. For example, when calculating the same feature amount as in the above-mentioned Non-Patent Document 1, C=5. When C=5, the feature amount for each pixel may be a character area score (1 channel) and a relative coordinate value (4 channels) representing a rectangle corresponding to the character area. If the rectangular area is a rectangle that does not have an inclination in the image, the relative coordinate values representing the rectangle may be the distance from each pixel's coordinate to the minimum and maximum X coordinates (coordinates in the width direction) and the minimum and maximum Y coordinates (coordinates in the height direction) of the rectangle. In other words, the above relative coordinate values are four values: (1) the absolute value of the difference between the X coordinate of the target pixel and the maximum X coordinate of the rectangular area, (2) the absolute value of the difference between the X coordinate of the target pixel and the minimum X coordinate of the rectangular area, (3) the absolute value of the difference between the Y coordinate of the target pixel and the maximum Y coordinate of the rectangular area, and (4) the absolute value of the difference between the Y coordinate of the target pixel and the minimum Y coordinate of the rectangular area.
[0027] In other words, the features output by the character detection unit 12 may include, for each pixel contained in the input image, a character area score indicating the degree to which that pixel is included in the character area, and position information indicating the position within the image of the character area containing that pixel.
[0028] The size (height direction, width direction) of the feature calculated by the character detection unit 12 does not necessarily have to be the same as the size of the input image. In order to reduce the calculation processing load, an image feature map / single character map with a size such as 1 / 2 or 1 / 4 of the input image in each of the height and width directions may be output. In this case, the feature calculated by the character detection unit 12 may be expressed as a matrix of C×H2×W2. Here, for example, H2=H / 2, W2=W / 2 may be used, or H2=H / 4, W2=W / 4 may be used. However, when the size of the feature is reduced, the accuracy of detecting small character strings in the image is deteriorated. The size of the feature is in a trade-off relationship with the character detection accuracy.
[0029] For learning purposes, the character detection unit 12 adjusts (updates) the internal parameters of the character detection model that the character detection unit 12 itself possesses based on the character detection error, which is the error between the output features and the correct data, and the estimated final score.
[0030] The character detection unit 12 may further perform a process of refining the obtained four-point rectangle into a polygonal polygon. This process can be performed using a pre-trained RNN (Recurrent Neural Network). The process of refining into a polygonal polygon using an RNN can be performed using existing technology. The data input to this RNN is the feature amount of the image. This image feature amount is output from a CNN (Convolutional Neural Network) based on pixel values. The output from the RNN is the coordinate values of the vertices that represent the polygonal polygon. The RNN outputs information on the vertices of the polygonal polygon, for example, two vertices at a time. The process of converting the four-point rectangle into a polygonal polygon allows for more accurate detection of the character area.
[0031] The output data storage unit 13 at least temporarily stores data output by the character detection unit 12. Specifically, the output data storage unit 13 stores data of the above-mentioned C×H×W matrix corresponding to an image of H pixels in the height direction and W pixels in the width direction.
[0032] The character detection error calculation unit 14 calculates an error based on the data written in the output data storage unit 13 and the correct answer data supplied from the learning data supply unit 32. That is, the character detection error calculation unit 14 calculates a character detection error for updating the parameters of the character detection unit 12. The character detection error calculation unit 14 calculates the character detection error as the sum of the first error and the second error described below. However, depending on the stage of the learning process, the character detection error calculation unit 14 may not use the second error and may use only the first error as the character detection error. For example, at the beginning of learning, the accuracy of the final score estimated by the final score estimation unit 22 is low. Therefore, until a predetermined number of learnings are completed, the character detection error calculation unit 14 may not use the second error in order to stabilize the learning.
[0033] That is, in the first stage of learning, character detection unit 12 adjusts the internal parameters of the character detection model held by character detection unit 12 based only on the character detection error (the error calculated by character detection error calculation unit 14). Next, in the second stage of learning, character detection unit 12 adjusts the internal parameters of the character detection model based on both the character detection error and the estimated final score (the estimated final score calculated by final score estimation unit 22).
[0034] The above-mentioned first error is a general estimated error obtained from the output of character detection unit 12. When the neural network of character detection unit 12 outputs a total of five values for each pixel, including a character area score value and four values for rectangular coordinates, character detection error calculation unit 14 calculates the first error in the following manner. For the character area score, character detection error calculation unit 14 calculates the cross entropy error between the calculated character area score and the correct score. For the rectangular coordinates, character detection error calculation unit 14 calculates the mean square error from the correct coordinates.
[0035] The second error is an error obtained from the estimated final score calculated by the final score estimation unit 22. Specifically, the second error is the difference between the estimated final score and the maximum score value (i.e., "1"). The character detection error calculation unit 14 calculates the second error using a general error function such as Mean Square Error (MSE) or Binary Cross-entropy.
[0036] The integration processing unit 21 integrates the detected overlapping estimated character regions. Specifically, the integration processing unit 21 integrates the detected character regions by excluding all but the region with the maximum score from among the many overlapping estimated character regions found by the character detection unit 12. The integration processing unit 21 eliminates all but the character region with the maximum score from among the overlapping character regions by NMS (Non-maximum-suppression) processing. The NMS processing itself can be performed by existing technology.
[0037] That is, the integration processing unit 21 integrates overlapping character region candidates by performing non-maximum suppression (NMS) processing based on the character region scores.
[0038] When operating in the learning mode, the integration processing unit 21 further calculates a final score for the entire character region (which may be a plurality of character regions) output. In the process of detecting a character region in an image, generally, when an IOU (above-mentioned Intersection over Union) value, which indicates a matching rate with an accurate region, exceeds a predetermined threshold, the detection is deemed successful. The threshold here may be, for example, 0.5. The integration processing unit 21 calculates the final score based on the recall rate and the precision rate of the region detection. The recall rate is a numerical value indicating how much of the actual character region was detected. The precision rate is a numerical value indicating how much of a region that is not actually a character region was erroneously detected. The integration processing unit 21 calculates, for example, an F value, which is the harmonic mean of the recall rate and the precision rate, as the final score. The final score calculated by the integration processing unit 21 is a numerical value indicating the accuracy of character detection. In other words, the final score is numerical data indicating the detection accuracy of the character detection unit 12.
[0039] When the processing of the integration processing unit 21 is regarded as a function, the function is a non-differentiable function. In other words, the processing of the integration processing unit 21 is a non-differentiable processing. The final score calculated by the integration processing unit 21 is used as teacher data for optimizing the neural network of the final score estimation unit 22. In other words, the final score estimation unit 22 is optimized by learning so as to be able to estimate a score close to the final score calculated by the integration processing unit 21. Even if the processing of the integration processing unit 21 itself is a non-differentiable processing, it is possible to optimize the neural network of the final score estimation unit 22 and the character detection unit 12 so as to reduce the error between the final score calculated by the integration processing unit 21 and the estimated final score calculated by the final score estimation unit 22.
[0040] That is, integration processing unit 21 determines character regions by integrating overlapping character region candidates detected by character detection unit 12. Furthermore, integration processing unit 21 calculates a final score for all (the whole) of the determined character regions, based on the correct answer data supplied by learning data supply unit 32.
[0041] The integration processing unit 21 may obtain the final score by a procedure other than the above-mentioned method. In this embodiment, it is not necessary to realize the calculation of the final score as a differentiable calculation. Therefore, the integration processing unit 21 may calculate any score as the final score, regardless of whether it is differentiable or not.
[0042] The final score estimation unit 22 calculates an estimated final score, which is an estimated value of the above-mentioned final score, based on the feature amount output by the character detection unit 12. Specifically, when the character detection device 1 operates in the learning mode, the final score estimation unit 22 estimates the final score based on the data output by the character detection unit 12 and the correct answer data for character detection supplied by the learning data supply unit 32. The final score estimation unit 22 reads the data output by the character detection unit 12 (the aforementioned matrix having a size of C×H×W, etc.) from the output data storage unit 13. The final score estimation unit 22 is realized using a neural network (final score estimation model).
[0043] The correct answer data for character detection may be represented as a list of coordinate values of one or more character regions included in the input image, or equivalent data. In other words, the correct answer data for character detection may also be represented as binary data (e.g., 1 if a pixel is included in the correct character region, and 0 if the pixel is not included in the correct character region) indicating whether or not the pixel is a character region in an image of height H pixels by width W pixels. The final score estimation unit 22 receives as input the correct answer data indicating whether or not each pixel is a character region. In this case, the data input to the final score estimation unit 22 is a matrix of size (C+1)×H×W.
[0044] Instead of expressing the correct answer data for character detection as a binary value of 0 and 1 as described above, the following modified example may be used to distinguish between each character area. That is, if a pixel is not included in a character area, the value is 0, if the pixel is included in a first character area, the value is 1, if the pixel is included in a second character area, the value is 2, and so on. That is, in this modified example, the map representing the correct answer for character detection has information that identifies each character area. In other words, a label is assigned to each correct answer character area, and the map is filled with a value corresponding to the label.
[0045] Furthermore, attributes of each text region may be expressed on the map. For example, the value corresponding to a pixel may be changed depending on whether the character string included in the text region is Japanese or English (difference in language).
[0046] The output from the final score estimation unit 22 is a final score value. When the above-mentioned F value is used as the final score, the range of the value output by the final score estimation unit 22 can be limited to 0 or more and 1 or less.
[0047] The final score estimation unit 22 can update its own internal parameters based on the final score error calculated by the final score error calculation unit 24. In other words, the final score estimation unit 22 adjusts the internal parameters of the final score estimation model held by the final score estimation unit based on the final score error.
[0048] The final score error calculation unit 24 calculates a final score error. Specifically, the final score error calculation unit 24 calculates the error between the final score calculated by the integration processing unit 21 and the estimated final score calculated by the final score estimation unit 22. The error calculated by the final score error calculation unit 24 is used to update the parameter values of the neural network by the error backpropagation method. Note that the final score error calculation unit 24 may calculate the error using a general error function such as Mean Square Error (MSE) or Binary Cross-entropy (cross entropy for binary classification).
[0049] The control unit 31 controls the operation of the character detection device 1. The control unit 31 controls the mode in which the character detection device 1 operates. As described above, there are two types of modes: a learning mode and an estimation mode. When operating in the learning mode, the control unit 31 controls the character detection device 1 to learn a model (machine learning) based on learning data. When operating in the estimation mode, the control unit 31 controls the character detection device 1 to perform processing to detect a character region of an unknown image as a processing target based on a trained model. Note that, as will be described later as a modified example, the character detection device 1 may operate in only one of the learning mode and the estimation mode.
[0050] The learning data supply unit 32 supplies learning data when the character detection device 1 operates in the learning mode. The learning data is a set of pairs of an image and correct answer data corresponding to the image. The learning data includes many of the above pairs in order to learn the model. The correct answer data represents correct answers for information on the position (coordinates) of a character region in an image. One image may include multiple character regions. A pair of an image that does not include a character region and its correct answer data may be included as part of the learning data. The information on the position of the character region may be, for example, a set of rectangular coordinate values. The information on the position of the character region may be map information (binary information) that corresponds to each pixel of the image and indicates whether the pixel is included in the character region. The information on the position of the character region may include information that identifies the character region that includes the pixel (for example, a serial number assigned to each character region) that corresponds to each pixel of the image. The information on the character region may include information that corresponds to each pixel of the image and indicates an attribute of the character region that includes the pixel (for example, an attribute such as whether the character region includes a Japanese character string or an English character string).
[0051] When operating in the learning mode, the learning data supplying unit 32 passes the image to the image acquiring unit 11. The learning data supplying unit 32 also passes the correct answer data to necessary functional units. The learning data supplying unit 32 passes the correct answer data to, for example, the character detection error calculating unit 14 and the integration processing unit 21.
[0052] Next, the operation procedure of the character detection device 1 will be described with reference to respective flow charts for the learning mode and the estimation mode.
[0053] 2 is a flowchart showing the procedure of processing in the learning mode of the character detection device 1 of this embodiment. The following will be described with reference to this flowchart.
[0054] In step S1, the image acquisition unit 11 acquires an image for learning. This learning image is supplied by the learning data supply unit 32. The learning data supply unit 32 also supplies correct answer data corresponding to the learning image. The correct answer data is rectangular coordinate data indicating a character area in the image. The learning data supply unit 32 passes the correct answer data to the character detection error calculation unit 14.
[0055] In step S2, the character detection unit 12 calculates features related to character detection based on the image acquired by the image acquisition unit 11. This step is a step during the learning process, and the parameter values of the neural network used here may be the parameter values at that time. The input and output data in the character detection unit 12 are as already explained. The character detection unit 12 writes the output data to the output data storage unit 13.
[0056] In step S3, the integration processing unit 21 reads the output data from the character detection unit 12 from the output data storage unit 13. The integration processing unit 21 calculates a final score based on the read data. This final score is the target for optimization by the character detection device 1. An example of the method for calculating the final score has already been described.
[0057] In step S3, integration processing unit 21 performs NMS processing in the process up to calculating the final score. As a result, integration processing unit 21 organizes and integrates the detected character regions by excluding all regions other than the region with the maximum score from among the many overlapping estimated character regions obtained by character detection unit 12.
[0058] In step S4, the final score error calculation unit 24 calculates a final score error. Specifically, the final score error calculation unit 24 obtains an error between the true final score calculated by the integration processing unit 21 and the estimated final score estimated by the final score estimation unit 22.
[0059] In step S5, character detection error calculation unit 14 obtains a character detection error for updating the parameters of character detection unit 12. Character detection error calculation unit 14 calculates the character detection error as the sum of the first error and the second error described above. However, character detection error calculation unit 14 may use only the first error as the character detection error without using the second error depending on the stage of the learning process. For example, at the beginning of learning, the accuracy of the final score estimated by final score estimation unit 22 is low. Therefore, until a predetermined number of learnings are completed, character detection error calculation unit 14 may not use the second error in order to stabilize the learning.
[0060] In step S6, the final score estimation unit 22 updates its own internal parameters based on the final score error calculated in step S4 by the final score error calculation unit 24. For this parameter update, the final score estimation unit 22 can use a general neural network optimization method such as SGD (Stochastic Gradient Descent) or Adam.
[0061] In step S7, character detection unit 12 updates its own internal parameters based on the character detection error calculated in step S5 by character detection error calculation unit 14. For this parameter update, character detection unit 12 can use a general neural network optimization method such as the above-mentioned SGD (Stochastic Gradient Descent) or Adam.
[0062] In step S8, the control unit 31 determines a learning termination condition. The termination condition here is, for example, that "parameter updates have been performed a predetermined number of times using learning data" or "sufficient convergence has occurred, and the error no longer decreases even if the set of parameter values is updated." If the predetermined termination condition is met (step S8: YES), the character detection device 1 terminates the entire process of this flowchart. If the termination condition is not met (step S8: NO), the control unit 31 controls the character detection device 1 to return to step S1 so as to continue processing based on the next learning data.
[0063] The above is the processing procedure when character detection device 1 operates in the learning mode. The processing in the learning mode described above produces the following two effects. First, the internal parameters of the neural network of final score estimation unit 22 are optimized so that the error between the final score calculated by integration processing unit 21 and the estimated final score calculated by final score estimation unit 22 is minimized. Second, the internal parameters of the neural network of character detection unit 12 are optimized so that the loss in character detection is minimized and the estimated final score calculated by final score estimation unit 22 is maximized.
[0064] Through this learning, it is expected that final score estimation unit 22 will be able to estimate a final score close to the final score calculated by integration processing unit 21 based on the input data. In other words, final score estimation unit 22 outputs a good final score close to the actual final score. Then, based on the estimated final score output by such final score estimation unit 22, the neural network of character detection unit 12 is optimized. As a result, character detection unit 12 will be able to output a good character region detection result, in other words, a character region detection result that will obtain a high final score, even for an unknown input image.
[0065] 3 is a flowchart showing the procedure of processing in the estimation mode of the character detection device 1 of this embodiment. Note that, when the character detection device 1 operates in the estimation mode, it is assumed that optimization of the internal parameters (parameters of the neural network, etc.) has been completed. The following description will be given with reference to this flowchart.
[0066] First, in step S11, the image acquisition unit 11 acquires an image to be estimated. This image may include one or more character regions, but the positions (coordinates) of the character regions in the image are unknown. The character detection device 1 estimates the character regions in this image.
[0067] In step S12, the character detection unit 12 calculates features related to character detection based on the image acquired by the image acquisition unit 11. The neural network of the character detection unit 12 calculates the features based on the optimized parameter values. The character detection unit 12 writes the calculated feature data to the output data storage unit 13.
[0068] In step S13, integration processing unit 21 reads output data from character detection unit 12 from output data storage unit 13. Integration processing unit 21 performs NMS processing based on the read data. As a result, integration processing unit 21 organizes and integrates the detected character regions by excluding all but the region with the maximum score from among the many overlapping estimated character regions found by character detection unit 12. In this way, integration processing unit 21 outputs information on any number of detected character regions (0 or more).
[0069] FIG. 4 is a block diagram showing an example of the internal configuration of the character detection device 1. The character detection device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. The RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to a RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.
[0070] At least a part of the functions of the character detection device 1 can be realized as a program that runs on a computer. In that case, the program may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. The term "computer-readable recording medium" may also include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the functions described above, or may be a program that can realize the functions described above in combination with a program already recorded in the computer system.
[0071] Although the embodiment has been described above, the present invention can also be embodied in the following modified examples.
[0072] In the above embodiment, the character detection device 1 operates in both the learning mode and the estimation mode. The character detection device 1 first operates in the learning mode to learn a model, and after the learning is completed, the character detection device 1 can operate in the estimation mode based on the learned model. As one aspect of the modified example, the character detection device 1 may operate only in the learning mode. In that case, the character detection device 1 may be called a "learning device". In that case, the character detection device 1 only learns a model (neural network). The model learned in the character detection device 1 can be transplanted to another device to allow the other device to function as a character detection device. As another aspect of the modified example, the character detection device 1 may operate only in the estimation mode. In this case, the model has already been learned. For example, a model learned in another device may be transplanted to the character detection device 1 to be configured to operate only in the estimation mode. When the character detection device 1 operates only in the learning mode or only in the estimation mode, it is sufficient that the character detection device 1 has only the functions necessary for operation in each mode among the functions described with reference to FIG. 1.
[0073] In the above embodiment, the process in which the character detection unit 12 refines a four-point rectangle of a detected character region into a polygonal polygon has also been described. In this case, the integration processing unit 21 performs non-maximum suppression (NMS) processing based on the character region score on the polygonal character region after refinement. As a modified example, the order of this processing may be reversed. That is, in this modified example, the integration processing unit 21 performs NMS processing on the four-point rectangular character region detected by the character detection unit 12. Then, the character detection device 1 refines the four-point rectangular character region remaining as a result of NMS into a polygonal polygon.
[0074] As described above, in this embodiment (including the modified example), a character area is detected using the neural network of the character detection unit 12. Furthermore, the integration processing unit 21 integrates overlapping character areas. When the character detection device 1 operates in the learning mode, the integration processing unit 21 calculates a final score based on the correct answer data. The processing of the integration processing unit 21 includes non-differentiable processing. When the character detection device 1 operates in the learning mode, the final score estimation unit 22 estimates the final score calculated by the integration processing unit 21. The neural network of the final score estimation unit 22 itself is also optimized by learning. Since all the processing performed by the final score estimation unit 22 is differentiable processing, it is possible to optimize the neural network of the character detection unit 12 based on the estimated final score output by the final score estimation unit 22.
[0075] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]
[0076] The present invention can be used, for example, to attach tag information to images (including videos), to enable image searches by text, and to add other added value to images, but the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]
[0077] 1. Character detection device (learning device) 11 Image acquisition section 12 Character detection section 13 Output data storage unit 14 Character detection error calculation unit 21 Integrated Processing Unit 22 Final Score Estimation Section 24 Final score error calculation section 31 Control Unit 32 Learning Data Supply Unit 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus
Claims
1. a character detection unit that inputs an image and outputs a feature amount relating to a character region included in the image; an integration processing unit that determines character regions by integrating overlapping character region candidates detected by the character detection unit, and calculates a final score for all of the determined character regions based on ground truth data; a final score estimation unit that calculates an estimated final score, which is an estimated value of the final score, based on the feature amount output by the character detection unit; a final score error calculation unit that calculates a final score error, which is an error between the final score and the estimated final score; Equipped with the final score estimation unit adjusts an internal parameter of a final score estimation model held by the final score estimation unit based on the final score error for learning; the character detection unit adjusts internal parameters of a character detection model held by the character detection unit, for learning, based on a character detection error, which is an error between the feature amount and correct answer data, and the estimated final score. Character detection device.
2. The feature amount includes, for each pixel included in the image, a character region score that indicates a degree to which the pixel is included in a character region, and position information that indicates a position in the image of the character region that includes the pixel. The character detection device according to claim 1 .
3. the integration processing unit integrates the overlapping character region candidates by performing a non-maximum suppression process based on the character region score, and also performs a process of converting the rectangular character region after integration into a finer polygonal region. The character detection device according to claim 2 .
4. the character detection unit adjusts internal parameters of the character detection model held by the character detection unit based only on the character detection error in a first stage of learning, and then adjusts internal parameters of the character detection model held by the character detection unit based on both the character detection error and the estimated final score in a second stage of learning; The character detection device according to any one of claims 1 to 3.
5. a character detection unit that inputs an image and outputs a feature amount relating to a character region included in the image; an integration processing unit that determines character regions by integrating overlapping character region candidates detected by the character detection unit, and calculates a final score for all of the determined character regions based on ground truth data; a final score estimation unit that calculates an estimated final score, which is an estimated value of the final score, based on the feature amount output by the character detection unit; a final score error calculation unit that calculates a final score error, which is an error between the final score and the estimated final score; Equipped with the final score estimation unit adjusts an internal parameter of a final score estimation model held by the final score estimation unit based on the final score error for learning; the character detection unit adjusts internal parameters of a character detection model held by the character detection unit, for learning, based on a character detection error, which is an error between the feature amount and correct answer data, and the estimated final score. Learning device.
6. a character detection unit that inputs an image and outputs a feature amount relating to a character region included in the image; an integration processing unit that determines a character region by integrating overlapping character region candidates detected by the character detection unit; Equipped with The character detection unit calculates the feature amount based on a character detection model trained by the learning device according to claim 5. Character detection device.
7. a character detection unit that inputs an image and outputs a feature amount relating to a character region included in the image; an integration processing unit that determines character regions by integrating overlapping character region candidates detected by the character detection unit, and calculates a final score for all of the determined character regions based on ground truth data; a final score estimation unit that calculates an estimated final score, which is an estimated value of the final score, based on the feature amount output by the character detection unit; a final score error calculation unit that calculates a final score error, which is an error between the final score and the estimated final score; Equipped with the final score estimation unit adjusts an internal parameter of a final score estimation model held by the final score estimation unit based on the final score error for learning; the character detection unit adjusts internal parameters of a character detection model held by the character detection unit, for learning, based on a character detection error, which is an error between the feature amount and correct answer data, and the estimated final score. A program that causes a computer to function as a character detection device.
8. a character detection unit that inputs an image and outputs a feature amount relating to a character region included in the image; an integration processing unit that determines a character region by integrating overlapping character region candidates detected by the character detection unit; Equipped with The character detection unit calculates the feature amount based on a character detection model trained by the learning device according to claim 5. A program that causes a computer to function as a character detection device.
Citation Information
Patent Citations
Error calculator and its program
JP2019029938A
Character recognition device, character recognition program, and character recognition method
JP2020017156A
