Image processing device and image processing method

Abstract

To create a model in which an area of an object can be appropriately extracted from an image.SOLUTION: An image processing apparatus outputs a contour of an object that is included in an image when the image is input as an estimation function using a parametric function; holds a model that includes a parameter of the parametric function as a model parameter, an input image, and teacher data that indicates a correct answer function indicating a correct answer contour which is a correct answer of a contour of the object included in the input image; acquires an estimation function of the contour of the object included in the input image by inputting the input image into the model; calculates an error between the correct answer function and the acquired estimation function; and when the calculated error is larger than a predetermined value, updates the parameter of the parametric function such that the error becomes small.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] As a background technique of this technical field, there is JP 2018-180660 A (Patent Document 1). This publication states that "The biometric authentication device includes an image input unit that captures a finger under ambient light to acquire at least one biometric image, an authentication processing unit that processes the biometric image, and a storage unit that stores registration information related to at least one of a plurality of biometric features obtained from the finger. The authentication processing unit uses the registration information in the storage unit to generate a color transformation matrix that minimizes the color difference between the reference biometric image and the representative color of the biometric image at the time of authentication, generates a color-converted biometric image using the generated color transformation matrix (S111), extracts features (S112), calculates the similarity with the registration information, and performs matching (S113)" (see abstract).

[0003] Furthermore, Non-Patent Document 1 discloses a method for modeling a curved closed region such as a signboard using a spline curve or the like and estimating the curve parameters when extracting the region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-180660 A [Non-patent literature]

[0005] [Non-Patent Document 1] Yuliang Liu, et. al. “ABCNet: Real-time Scene Text Spotting with Adaptive Bezier-Curve Network”, February 2020 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Patent Document 1 needs to estimate which class each pixel of the input image belongs to, and does not consider the validity of the region shape of the estimated result. In addition, the technology described in Non-Patent Document 1 does not estimate on a pixel-by-pixel basis, but directly represents and estimates the region boundary, but cannot reflect differences in the shape of the region in the estimation score. Therefore, one aspect of the present invention generates a model that makes it possible to appropriately extract the region of an object from an image. [Means for solving the problem]

[0007] In order to solve the above problem, one aspect of the present invention employs the following configuration: An image processing device includes a processor and a memory, and when an image is input, the memory stores a model that outputs the contour of an object included in the image as an estimated function using a parametric function and includes parameters of the parametric function as model parameters, an input image, and teacher data indicating a correct function that indicates a correct contour that is a correct solution for the contour of the object included in the input image, and the processor inputs the input image to the model to obtain an estimated function of the contour of the object included in the input image, calculates an error between the correct function indicated by the teacher data and the obtained estimated function, and if the calculated error is greater than a predetermined value, updates the parameters of the parametric function so as to reduce the calculated error. Effect of the Invention

[0008] According to one aspect of the present invention, it is possible to generate a model that enables appropriate extraction of an object region from an image.

[0009] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment. [Diagram 2] 11 is a flowchart showing an example of a process in which the image processing system in the first embodiment converts a background separation image into a parametric curve. [Diagram 3] 11 is a flowchart showing an example of a training phase by the image processing system in the first embodiment. [Figure 4] 4 is an example of a parametric curve that expresses a part of a contour curve in the first embodiment. [Diagram 5] 4A to 4C are explanatory diagrams showing an example of a contour tracing process, a connected component extraction process, and a curve division process in the first embodiment. [Figure 6] 10A to 10C are explanatory diagrams showing another example of the contour tracing process, the connected component extraction process, and the curve division process in the first embodiment. [Figure 7] FIG. 4 is an explanatory diagram showing an example of a method for calculating an error between an estimated contour curve and a correct contour curve in the first embodiment. [Figure 8] 11 is a flowchart showing an example of a curve error calculation process in the first embodiment. [Figure 9] 10 is an explanatory diagram showing another example of a method for calculating an error between an estimated contour curve and a correct contour curve in the first embodiment. FIG. [Figure 10] 13 is a flowchart showing another example of the curve error calculation process in the first embodiment. [Figure 11] 4 is an example of an attribute parameter input screen displayed on a display device of the information processing device in the first embodiment. [Figure 12] 10 is an explanatory diagram showing another example of a method for calculating an error between an estimated contour curve and a correct contour curve in the first embodiment. FIG. [Figure 13] 11 is a flowchart showing an example of a curve error calculation process in the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. It should be noted that the embodiment is merely an example for implementing the present invention and does not limit the technical scope of the present invention. EXAMPLES

[0012] Fig. 1 is a block diagram showing a configuration example of an image processing system 10. The image processing system 10 extracts a detected object from an image in which an object such as a car, a person, or a signboard, as well as a face, a finger, or a palm, is captured. The image processing system 10 includes an imaging device 2, an information processing device 3, and a server 5, which are connected to each other via a network 4 such as the Internet. The imaging device 2 may be attached to the information processing device 3, for example. Note that, although the image processing system 10 includes one information processing device 3 in the example of Fig. 1, the image processing system 10 may include multiple information processing devices 3.

[0013] In the image processing system 10, a still image or a video captured by an imaging device 2 is input to an information processing device 3, which performs a series of processes including video processing, curve parameter extraction from the captured image, and contour reproduction. The information processing device 3 and a server 5 perform processes related to transmission and reception of training data, a post-training model, and parameters during training via a network 4. The information processing device 3 may display, for example, on a display device 36 provided in the information processing device 3, whether the image capture was successful and a guide on a method for presenting an object.

[0014] The information processing device 3 is configured by a computer including, for example, a CPU 31, a memory 32, an auxiliary storage device 33, a communication device 34, an input device 35, and a display device 36. The CPU 31 includes a processor and executes a program stored in the memory 32.

[0015] The memory 32 includes a ROM (Read Only Memory), which is a non-volatile storage element, and a RAM (Random Access Memory), which is a volatile storage element. The ROM stores immutable programs (e.g., a Basic Input / Output System (BIOS)). The RAM is a high-speed, volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the CPU 31 and data used when the programs are executed.

[0016] The auxiliary storage device 33 is a large-capacity non-volatile storage device such as a magnetic storage device (HDD (Hard Disk Drive)) or a flash memory (SSD (Solid State Drive)), and stores the programs executed by the CPU 31 and data used when the programs are executed. That is, the programs are read from the auxiliary storage device 33, loaded into the memory 32, and executed by the CPU 31.

[0017] In this embodiment, the information used by the image processing system 10 does not depend on the data structure and may be expressed in any data structure. For example, the information may be stored in a data structure appropriately selected from a table, a list, a database, or a queue.

[0018] The input device 35 is a device such as a keyboard or a mouse that receives input from an operator. The display device 36 is a device such as a display device or a printer that outputs the execution result of a program in a format that can be viewed by an operator. The information processing device 3 may also have an input interface for receiving input of moving images from the imaging device 2, or may receive moving images from the imaging device 2 via the network 4. The communication device 34 is a network interface device that controls communication with other devices according to a predetermined protocol.

[0019] The program executed by the CPU 31 may be provided to the information processing device 3 from another information processing device equipped with a non-transitory storage device via a removable medium (such as a CD-ROM or a flash memory) which is a non-transitory storage medium or via the network 4, and may be stored in a non-volatile auxiliary storage device 33 which is a non-transitory storage medium. For this reason, the information processing device 3 may have an interface for reading data from the removable medium. The same applies to the server 5.

[0020] The information processing device 3 is a computer system configured on one physical computer or on multiple logically or physically configured computers, and may operate in separate threads on the same computer, or may operate on a virtual computer constructed on multiple physical computer resources. The same applies to the server 5.

[0021] The CPU 31 includes, for example, a moving image input unit 320, a moving image processing unit 321, and an object detection unit 322. The moving image input unit 320 receives input of a moving image from the imaging device 2. The moving image input unit 320 may be a functional unit pre-installed in a mobile terminal or a notebook PC (Personal Computer), which is an example of the information processing device 3.

[0022] The video processing unit 321 converts the video inputted by the video input unit 320 into a plurality of still images. The object detection unit 322 includes an object detection unit 323 and a contour reproduction unit 324. The object detection unit 323 detects and extracts a detected object from each still image. The contour reproduction unit 324 reproduces the contour of the target region based on the parameters of the curve.

[0023] For example, CPU 31 functions as video input unit 320 by operating according to a video input program loaded in memory 32, and functions as video processing unit 321 by operating according to a video processing program loaded in memory 32. The same applies to the relationships between programs and functional units for other functional units included in CPU 31. The same applies to the relationships between programs and functional units for functional units described later included in CPU 51 of server 5.

[0024] Some or all of the functions of the functional units included in the CPU 31 and the CPU 51 may be realized by hardware such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA).

[0025] The auxiliary storage device 33 holds a post-training model 328 and training data 330. The post-training model 328 is a model that outputs the contour of a detection target object when an image including the detection target object is input, and is a model after training has been performed. The training data 330 is data used for training the model, and includes information linking an image including the detection target object with information indicating the area of ​​the detection target object.

[0026] The server 5 is configured by a computer including, for example, a CPU 51, a memory 52, an auxiliary storage device 53, a communication device 54, an input device 55, and a display device 56. Descriptions of the hardware of the CPU 51, the memory 52, the auxiliary storage device 53, the communication device 54, the input device 55, and the display device 56 are omitted because they are similar to the descriptions of the hardware of the CPU 31, the memory 32, the auxiliary storage device 33, the communication device 34, the input device 35, and the display device 36.

[0027] The CPU 51 includes a training unit 151 and a data distribution unit 155. The training unit 151 generates a trained model 162 by training a model using training data 161. The training unit 151 also includes a contour curve parameter extraction unit 152, a regression estimation training unit 153, and a curve error calculation unit 154.

[0028] The contour curve parameter extraction unit 152 generates teacher data for the training data 161. The teacher data generated from one input image is compressed and expressed as a finite number of vector values ​​(control point coordinates) indicating a parametric curve expressing the contour of a two-dimensional background separation image.

[0029] The background separated image includes, for example, an object region extracted from the input image and a background image separated from the object region. For example, the background separated image is a binary image, and the object region and the background region are separated by a binary value.

[0030] For example, each pixel in the object region is assigned a value of 1, and each pixel in the background region is assigned a value of 0. When the training data generated from this background separated image is expressed, for example, by a Bezier curve, a spline curve, linear interpolation, or a polynomial expression, it can be compressed and expressed by a finite number of vector values ​​representing a parametric curve. The contour curve parameter extraction unit 152 converts the background separated image into a finite number of vector values ​​representing a parametric curve.

[0031] In a model training phase, the regression estimation training unit 153 trains the model using training data 161. The regression estimation training unit 153 inputs an object region and a background region of an input image as teacher data, and trains a model that generates a finite number of vector values ​​that represent a parametric curve.

[0032] In addition, in general, in a method for estimating an object region from an image, the authentication rate may deteriorate depending on the illuminance, lighting environment, and the default color of the object (for example, color differences due to race in the case of biometric information.) For this reason, the input data for the model may include illuminance, lighting color, and the default color of the object as attribute parameters in addition to the image.

[0033] When an attribute parameter is added during training, the contour curve parameter extraction unit 152 inputs the attribute parameter and performs estimation, or the contour curve parameter extraction unit 152 may learn a different training object model for each value of the attribute parameter (for each combination of values ​​of the multiple types of attribute parameters, in the case where multiple types of attribute parameters are given), and switch the training object model according to the value of the attribute parameter input and selected by the user.

[0034] In the training phase, the curve error calculation unit 154 digitizes the error between the estimated curve and the correct curve, and feeds it back as error information to the regression estimation training unit 153. The data distribution unit 155 distributes the trained model 162 to the information processing device 3. When the data distribution unit 155 receives a distribution request from the information processing device 3, it may distribute the trained model 162 to the information processing device 3.

[0035] 1, the processing by the functional units included in either the CPU 31 or the CPU 51 may be executed by the other, or may be distributed between the two. Also, the information processing device 3 and the server 5 may be integrated.

[0036] The auxiliary storage device 53 holds training data 161 and a post-training model 162. The training data 161 is data used for training the post-training model 162, and includes information linking an input image with a set of teacher data generated by the contour curve parameter extraction unit 152. The training data 161 includes data collected from an information processing device 3. The information processing device 3 from which the training data 161 is collected is not limited to one, and may be multiple information processing devices 3. The post-training model 162 is a model that outputs a contour of a detection target when an image including the target is input, and is a model after training has been performed.

[0037] FIG. 2 is a flowchart showing an example of a process in which the image processing system 10 converts a background separation image into a parametric curve (e.g., a Bezier curve, a spline curve, a linear interpolation, or a polynomial expression) having a parameter t as a parameter and defined by a finite number of vector values.

[0038] The background separation image includes an object region extracted from the input image and a background image separated from the object region. For example, the background separation image is a binary image, and the object region and background region of the background separation image are separated by a binary value (for example, each pixel of the object region is assigned a value of 1, and each pixel of the background region is assigned a value of 0). The target image of the background separation process in this embodiment is a visible light image in which a detection target object is captured. Below, an example will be described in which the detection target object included in the target image of the background separation process is a plurality of fingers.

[0039] Before the processing in Figure 2 starts, the video input unit 320 receives a video of a hand or fingers from the imaging device 2, the video processing unit 321 cuts the video into a plurality of still images, the object detection unit 323 generates a background-separated image by separating the background from the detected object using the plurality of still images, and transmits the generated background-separated image to the server 5.

[0040] In step S201, the contour curve parameter extraction unit 152 performs contour tracing on the background separated image. In general, the result of contour tracing is expressed as a series of continuous coordinate points on a curve. In step S202, the contour curve parameter extraction unit 152 extracts one or more connected components by dividing the contour. However, there are cases where the contour is divided into connected components at the stage when the contour is extracted, in which case the process of step S202 does not need to be executed.

[0041] Since the contour is a closed curve, in step S203, the contour curve parameter extraction unit 152 selects one unselected connected component from the extracted connected components, and divides the selected connected component into partial curves that can be approximated by curves.

[0042] In step S204, the contour curve parameter extraction unit 152 approximates each of the divided partial curves with a parametric curve. An example of the approximation process in step S204 will be described later with reference to FIG.

[0043] In step S205, the contour curve parameter extraction unit 152 extracts parameters of the curve approximated in step S204. In step S206, the contour curve parameter extraction unit 152 determines whether curve parameters have been extracted for all connected components.

[0044] When the contour curve parameter extraction unit 152 determines that there is a connected component for which curve parameters have not been extracted (S206: NO), the process returns to step S203. When the contour curve parameter extraction unit 152 determines that curve parameters have been extracted for all connected components (S206: YES), the contour curve parameter extraction unit 152 stores, in step S207, teacher data including a set of extracted curve parameters and a pair of an input image as one element of the training data 161. In addition, the contour curve parameter extraction unit 152 may transmit the teacher data to the information processing device 3 and store it in the training data 330.

[0045] As a result, the image processing system 10 can generate a trained model 162 that enables finger region estimation that reflects prior knowledge of finger shapes. The image processing system 10 can also generate a trained model 162 that enables finger regions to be correctly separated and extracted when the finger regions are in contact with each other.

[0046] 3 is a flowchart showing an example of a training phase performed by the image processing system 10. In the training phase, the image processing system 10 separates object regions and background regions of video frame images based on the range of values ​​of color space information such as RGB and YUV, and generates training data 161 for use in supervised learning. Furthermore, the image processing system 10 trains (generates) a machine learning model (trained model 162) using the generated training data 161. In the estimation process, the image processing system 10 extracts object regions from video frame images using the generated machine learning model.

[0047] In step S300, the information processing device 3 accepts input of a processing target image in which a detection target object is captured (an original image used to generate a background separated image acquired before the processing of FIG. 2 starts) and attribute parameters used by the training unit 151 via, for example, the input device 35, and transmits them to the server 5. In general, in a method for estimating an object region from an image, degradation of the authentication rate may occur depending on illuminance, lighting environment, and default color of the object (for example, color differences due to race when biometric information is the object). For this reason, in addition to the processing target image, illuminance, lighting color, default color of the object, and the like may be included in the attribute parameters as input data.

[0048] In step S301, the object detection unit 323 cuts out a partial area of ​​a certain size in which the extraction target is captured from the image to be processed. The object detection unit 323 may, for example, display a guide or the like indicating the position to place a hand on the screen on the display device 36 of the information processing device 3 to set in advance the position of the partial area to be cut out according to an input via the input device 35, or may detect a circumscribed rectangular area of ​​the detection target using an input device object detection technique. The object detection unit 323 transmits the cut-out image to the server 5 and stores it in the training data 161.

[0049] In step S302, the training unit 151 acquires a background-separated image (a background-separated image acquired before the processing in FIG. 2 starts) corresponding to the detected object in step S301. The training unit 151 may acquire the background-separated image manually in advance via the input device 55, or may automatically extract the background-separated image using an existing method. For example, the training unit 151 separates the object region and the background region from the cut-out partial region based on color space information such as RGB or YUV (for example, attribute parameters such as skin color), and acquires the background-separated image.

[0050] In step S303, the contour curve parameter extraction unit 152 extracts curve parameters for each extraction target region according to the process described in Fig. 2, and stores them as part of the training data 161. In step S304, the training unit 151 determines whether the number of samples of the training data has reached a predetermined number n. If the training unit 151 determines that the number of samples of the training data has not reached the predetermined number n (S304: NO), the process returns to step S301. If the training unit 151 determines that the number of samples of the training data has reached the predetermined number n (S304: YES), the process proceeds to step S305.

[0051] Next, in steps S305 to S311, regression estimation training unit 153 uses accumulated training data 161 to train an estimation model that receives an image including a detection target and outputs curve parameters.

[0052] First, in step S305, the image processing system 10 acquires, as input data, the illuminance, lighting color, default color of the object, and the like as attribute parameters in addition to the image to be processed, in the same manner as in step S300.

[0053] In step S306, the regression estimation training unit 153 estimates curve parameters using the currently-current model under training 163 (at the first training, model parameters of the estimation model are initialized with random values. Also, the model under training 163 is temporarily stored in the memory 52). In step S307, the regression estimation training unit 153 estimates the curve parameters using the estimated parameters. The contour reproduction unit 156 reproduces the contour curve using the estimated parameters.

[0054] In step S308, the curve error calculation unit 154 calculates the error (hereinafter referred to as curve error) between the estimated contour curve, which is the reproduced contour curve, and the correct contour curve (contour curve given by curve parameters included in the teacher data) stored in the training data 161. A specific example of the curve error calculation will be described later with reference to Figs. 7 and 9.

[0055] In step S309, the regression estimation training unit 153 determines whether the curve error is less than a predetermined threshold δ. When the regression estimation training unit 153 determines that the curve error is equal to or greater than the predetermined threshold δ (S309: NO), in step S310, the regression estimation training unit 153 updates the model parameters in the model under training 163 in a direction that reduces the error, and returns to step S306. The method of correcting the model parameters by the regression estimation training unit 153 depends on the machine learning method of the model. For example, when a deep learning model is used, the model parameters can be corrected using stochastic gradient descent (SGD).

[0056] If regression estimation training unit 153 determines that the curve error is less than the predetermined threshold δ (S309: YES), it ends the training phase and saves model in training 163 as post-training model 162 in step S311.

[0057] 3, the image processing system 10 can generate a trained model 162 that is capable of estimating a finger region reflecting prior knowledge of finger shapes and is capable of correctly separating and extracting finger regions when the finger regions are in contact, etc. Furthermore, when extracting a finger region from an input image, for example, the image processing system 10 extracts a contour line region but does not estimate whether the pixels inside the contour line region are background or not, and can efficiently extract an object region with a small amount of calculation.

[0058] FIG. 4 shows an example of a parametric curve that expresses a part of a contour curve. A parametric curve using a parameter t as a parameter is defined by (Equation 1) that draws a curve based on n control points. In (Equation 1), when the value of the parameter t changes from 0 to 1, the control point P 0 and P n-1 A parametric curve with both ends at P 0 ~P n-1 are expressed as coordinate points with x and y coordinates, respectively.

[0059]

number

[0060] If the number of samples of parameter t is k, then (Equation 1) can be expressed as a system of k simultaneous linear equations with n variables, or as a product of k×n-dimensional coefficient matrices (see (Equation 2) and (Equation 3) below). In (Equation 2) and (Equation 3), x 1 ~x k and y 1 ~y k represents the x-coordinate or y-coordinate of a point sampled from the curve to be approximated. The contour curve parameter extraction unit 152 calculates the pseudo inverse matrix of the right-hand side of (Equation 2) and (Equation 3) to obtain the control point P 0 ~P n-1 can be calculated (step S205).

[0061]

number

[0062] The control point P calculated by the above procedure 0 ~P n-1 are stored in training data 161 and used in the training phase in FIG.

[0063] The curve 401 is formed by the control points P 0 ~P 3 The control points calculated by the above procedure are called initial control points. Below, an example with four control points will be used for explanation.

[0064] In the recursive calculation process of the control points, the contour curve parameter extraction unit 152 extracts all adjacent control points P i P i+1 A straight line P consisting of i P i+1 Divide into t:1-t using the parameter t in (Equation 1) and create a new control point P k Calculate.

[0065] In the first recursive calculation of new control points in the curve 401, the contour curve parameter extraction unit 152 extracts a line P 0 P 1 A new control point P is used as the division point of 4 Calculate P 1 P 2 The control point P is used as the division point of 5 Calculate the line P 2 P 3 The control point P is used as the division point of 5 In the second recursive calculation of the new control points, the contour curve parameter extraction unit 152 calculates the straight line P 4 P 5 The control point P is used as the division point of 7 Calculate the line P 5 P 6 The control point P is used as the division point of 8 Finally, in the third recursive calculation of the new control points, the contour curve parameter extraction unit 152 calculates the straight line P 7 P 8 Point P is the division point of 9 The number of recursive calculations is, for example, the number of initial control points minus 1. The point P calculated in the last recursive calculation is 9 represents the point f(t) on the curve.

[0066] That is, the contour curve parameter extraction unit 152 extracts the control point P 0 ~P n By changing the parameter t in (Equation 1) from 0 to 1 using the above formula, the coordinate points on the parametric curve f(t) can be calculated, and the curve 401 can be reproduced. Note that the curve 402 is reproduced by a different control point P 0 ~P 3 1 is an example of a curve calculated using the same process as described above.

[0067] Fig. 5 is an explanatory diagram showing an example of the contour tracing process, connected component extraction process, and curve division process in steps S201 to S203 in Fig. 2. In the example of Fig. 5, an image of a hand is input, and curve parameters of four finger regions are extracted, but the target object is not limited to four fingers.

[0068] Background separated image 500, which is a background separated image of the hand region image, includes object region 501, object region 502, object region 503, and object region 504. Object region 501, object region 502, object region 503, and object region 504 are object regions of the index finger, middle finger, ring finger, and little finger, respectively.

[0069] Image 520 is an image in which the contour of an object is extracted from background separated image 500. Contour 505 is the contour line of object region 501. In step S201, contour curve parameter extraction unit 152 can extract contour 505 by a general image processing method. Alternatively, contour curve parameter extraction unit 152 may acquire contour 505 by input via input device 55.

[0070] In step S202, the contour curve parameter extraction unit 152 generally extracts a contour from an object region as a series of continuous coordinate points. However, for unconnected contours such as object region 501, object region 502, object region 503, and object region 504, the contour is extracted as separate continuous coordinate point data from each of these object regions.

[0071] The contour curve parameter extraction unit 152 determines a start point and an end point for one of the extracted contours. In the example of the object region 501, the contour curve parameter extraction unit 152 extracts a start point 506 and an end point 507 that correspond to the positions of the fingertips. Specifically, for example, the contour curve parameter extraction unit 152 can extract, among the circumscribed rectangle coordinates of the object region 501, a point closest to the lower left coordinate as the start point and a point closest to the lower right coordinate as the end point.

[0072] Also, for example, the contour curve parameter extraction unit 152 may search the contour coordinates counterclockwise or clockwise, and set the coordinates where the change in the inclination of the traveling direction is large (for example, 30 degrees or more) as the start point and end point, respectively, in the lower 1 / 3 region of the circumscribing rectangle coordinates of the object region 501. A curve 508 is an example of a curve extracted by the contour curve parameter extraction unit 152 from the contour 505, the start point 506, and the end point 507.

[0073] In order to facilitate approximation with a parametric curve, the contour curve parameter extraction unit 152 divides the extracted curve into partial curves in step 203. In the example of Fig. 5, the contour curve parameter extraction unit 152 divides the curve 508 into partial curves 509 and 510.

[0074] The contour curve parameter extraction unit 152 extracts control points P1 to P8 of the parametric curve from the partial curves 509 and 510 in accordance with the procedure described above using Fig. 4, (Equation 2), and (Equation 3). Note that in the example of Fig. 5, the contour curve parameter extraction unit 152 extracts four control points from each partial curve, but the number of control points is not limited to this.

[0075] Fig. 6 is an explanatory diagram showing another example of the contour tracing process, connected component extraction process, and curve division process in steps S201 to S203 in Fig. 2. In the example of Fig. 6, an image capturing the periphery of the pupil is input, and curve parameters of the iris region are extracted.

[0076] The background separated image 62 obtained from the input iris image 61 of the eye region image includes an object region 601 which is an object region of the iris region. Contours 602 and 603 are contour lines of the object region 601. In step S201, the contour curve parameter extraction unit 152 can extract the contours 602 and 603 by a general image processing method. Alternatively, the contour curve parameter extraction unit 152 may acquire the contours 602 and 603 by input via the input device 55.

[0077] In step S202, the contour curve parameter extraction unit 152 generally extracts contours 602 and 603 from the object region as a series of continuous coordinate points. In step S203, the contour curve parameter extraction unit 152 divides each of the extracted contours 602 and 603 into partial curves to facilitate approximation with parametric curves. In the example of Fig. 6, the contour 602, which is the outer contour, is divided into two partial curves 609 and 610, and the contour 603, which is the inner contour, is divided into two partial curves 611 and 612.

[0078] 4, (Equation 2), and (Equation 3), the contour curve parameter extraction unit 152 extracts control points P1 to P4 of the parametric curve from each of the partial curves 609, 610, 611, and 612. Note that in the example of Fig. 5, the contour curve parameter extraction unit 152 extracts four control points from each partial curve, but the number of control points is not limited to this.

[0079] 7 is an explanatory diagram showing an example of a method for calculating the error between the estimated contour curve and the correct contour curve in step S308. Here, the partial curve 509 after division in FIG. 5 will be explained as an example.

[0080] The estimated contour curve 70 is a curve reproduced using the control points P1 to P4 estimated by the model in training 163. The correct contour curve 71 is a curve reproduced using the correct control points P1 to P4. For example, the error between the estimated contour curve 70 and the correct contour curve 71 is defined by the area between the two curves, the estimated contour curve 70 and the correct contour curve 71.

[0081] For sections that are common to both curves, the curve error calculation unit 154 calculates the difference between the integral values ​​of each curve, and for sections where only one curve exists, it calculates the difference between the integral values ​​by extending the Y coordinate of the other curve as a fixed value of the endpoint coordinate.

[0082] In the example of Fig. 7, the curve error calculation unit 154 divides the section at the X coordinate where the two curves intersect, with the long axis direction of the finger as the X axis and the direction perpendicular to the long axis direction of the finger as the Y axis, and calculates the area of ​​the region in each section. In the example of Fig. 7, the estimated contour curve 70 and the correct contour curve 71 intersect at the intersection 701, so the curve error calculation unit 154 calculates the area of ​​the region x 1 ~x i and the interval x i ~x n The difference between the integrals is calculated by dividing the curve into and . The following (Equation 4) is a formula for integrating the curve of (Equation 1) over the section a to b with respect to the parameter t.

[0083]

number

[0084] Moreover, the following (Equation 5) is a formula for calculating (a) in (Equation 4).

[0085]

number

[0086] The final equation in (Equation 5) uses the integral formula for the beta function. Using the above formula, the curve error calculation unit 154 calculates the area between the two curves by subtracting the integral of the curve with the smaller Y value from the integral of the curve with the larger Y value, between the estimated contour curve 70 and the correct contour curve 71, at the control point P i The vertical relationship of the two curves on the Y-axis can be determined for each section that is bounded by the intersection of the two curves.

[0087] The curve error calculation unit 154 can calculate the intersection point between the estimated contour curve 70 and the correct contour curve 71 by using the Bezier Clipping method or the like. Furthermore, the curve error calculation unit 154 may add the integral amount of the protruding portion of the estimated contour curve 70 or the correct contour curve 71 to the error term as a correction term.

[0088] In the example of FIG. 7, the curve error calculation unit 154 calculates x 0 ~x 1 For the section, the integration of the estimated contour curve 70 using (Equation 4) and the endpoint x of the correct contour curve 71 are 0 The area of ​​the rectangular area with the Y coordinate fixed in the section x n ~x n+1 Regarding the above, the integration of the correct contour curve 71 using (Equation 4) and the end point x of the estimated contour curve 70 are n+1 The area of ​​the rectangular area with the Y coordinate of the point 1 fixed is calculated, and the difference is added to the curve error.

[0089] As described above, the image processing system 10 of this embodiment can generate a trained model 162 that enables finger region estimation that reflects prior knowledge of finger shapes. The image processing system 10 can also generate a trained model 162 that enables finger regions to be correctly separated and extracted when the finger regions are in contact with each other. The image processing system 10 can also directly reflect errors in the contour shapes in the training model 163.

[0090] 8 is a flowchart showing an example of the curve error calculation process in step S308. An example of curve error calculation for the estimated contour curve 70 and correct contour curve 71 in FIG.

[0091] In step S801, the curve error calculation unit 154 calculates the x-coordinate (x 1 Specifically, for example, the curve error calculation unit 154 determines the larger of the X coordinates of the left ends of the estimated contour curve 70 and the correct contour curve 71 as the starting point.

[0092] In step S802, the curve error calculation unit 154 calculates the x-coordinate (x n Specifically, for example, the curve error calculation unit 154 determines the smaller of the X coordinates of the right ends of the estimated contour curve 70 and the correct contour curve 71 as the end point.

[0093] In step S803, the curve error calculation unit 154 calculates the smaller of the X coordinates of the left ends of the estimated contour curve 70 and the correct contour curve 71 (x 0 ) is calculated. The curve error calculation unit 154 also calculates the curve error 0 ~x 1 For either the estimated contour curve 70 or the correct contour curve 71 (section x 0 ~x 1 The integral of the left end point of the curve (the Y coordinate value up to the curve where the Y coordinate value was defined) and the Y coordinate of the left end point of the other curve and the interval x 0 ~x 1 The difference between the rectangular area formed by and is calculated as the curve error.

[0094] In step S804, the curve error calculation unit 154 calculates the larger of the X coordinates of the right ends of the estimated contour curve 70 and the correct contour curve 71 (x n+1 The curve error calculation unit 154 calculates the curve error n ~x n+1 For either the estimated contour curve 70 or the correct contour curve 71 (section x n ~x n+1 The integral of the right end point of the other curve and the interval x n ~x n+1 The difference between the rectangular area formed by and is calculated as the curve error.

[0095] In step S805, the curve error calculation unit 154 calculates the intersection point between the estimated contour curve 70 and the correct contour curve 71, and divides the x-coordinate of the intersection point into two parts on the x-axis. 0 ~x n The curve error calculation unit 154 calculates the intersection points of the two curves using, for example, the Beziel Clipping method.

[0096] In step S806, the curve error calculation unit 154 selects an unselected divided partial section, and calculates the area between the curves for the selected partial section according to the procedure described with reference to Fig. 7. In step S807, the curve error calculation unit 154 determines whether the area between the curves for all partial sections has been calculated. If the curve error calculation unit 154 determines that there is a partial section for which the area between the curves has not been calculated (S807: NO), the process returns to step S806.

[0097] When it is determined that the area between the curves of all the partial sections has been calculated (S807: YES), the curve error calculation unit 154 calculates the sum of the area between the curves of all the sections in step S808. In this manner, the curve error calculation unit 154 can calculate the curve error and feed back the curve error to the model in training 163.

[0098] As described above, the image processing system 10 of this embodiment can generate a trained model 162 that enables finger region estimation that reflects prior knowledge of finger shapes. The image processing system 10 can also generate a trained model 162 that enables finger regions to be correctly separated and extracted when the finger regions are in contact with each other. The image processing system 10 can also directly reflect errors in the contour shapes in the training model 163.

[0099] 9 is an explanatory diagram showing another example of the method of calculating the error between the estimated contour curve and the correct contour curve in step S308. Here, the partial curve 509 after division in FIG. 5 will be taken as an example.

[0100] The estimated contour curve 90 is a curve reproduced using the control points P1 to P4 estimated by the model in training 163. The correct contour curve 91 is a curve reproduced using the correct control points P1 to P4. For example, the error between the two parametric curves, the estimated contour curve 90 and the correct contour curve 91, is defined by the square error based on a finite number of sampling points of the two parametric curves.

[0101] Points 901 to 909 on the estimated contour curve 90 indicate the values ​​of f(t) (the coordinates of each point) when the parameter t in (Equation 1) is sampled at equal intervals in the range from 0 to 1 (divided into 8 equal parts using 9 coordinates in the example of Figure 9).

[0102] Points 911 to 919 on the correct contour curve 91 indicate the values ​​of f(t) (the coordinates of each point) when the parameter t in (Equation 1) is sampled at equal intervals in the range from 0 to 1 (8 equal divisions using 9 coordinates in the example of Figure 9).

[0103] The curve error calculation unit 154 can calculate an error function required when updating the model under training 163 by using the following error D (square error) between points 901 to 909 and points 911 to 919. p (t) represents the estimated contour curve 90. The following function f g(t) represents the correct contour curve 91. The definition of the function J below is the same as the function J in (Equation 1). t and P S represent the control point coordinates of the estimated contour curve 90 and the control point coordinates of the correct contour curve 91, respectively.

[0104]

number

[0105] As described above, the image processing system 10 of this embodiment can generate a trained model 162 that enables finger region estimation that reflects prior knowledge of finger shapes. The image processing system 10 can also generate a trained model 162 that enables finger regions to be correctly separated and extracted when the finger regions are in contact with each other. The image processing system 10 can also directly reflect errors in the contour shapes in the training model 163.

[0106] 10 is a flowchart showing another example of the curve error calculation process in step S308. An example of curve error calculation for the estimated contour curve 90 and correct contour curve 91 in FIG.

[0107] In step S1001, the curve error calculation unit 154 sets the number of samples m of sampling points for error calculation. The curve error calculation unit 154 may set the number of samples m according to, for example, an input via the input device 55, or may set the number of samples m according to the width of the domain of the parameter t, the width of the value range, or the total length of the correct curve (for example, so that the total length of the correct curve / the number of samples m is a value included in a predetermined range).

[0108] In step S1002, the curve error calculation unit 154 samples the parameter t of the estimated contour curve 90 at m points at equal intervals in the range from 0 to 1. In step S1003, the curve error calculation unit 154 samples the parameter t of the correct contour curve 91 at m points at equal intervals in the range from 0 to 1. In step S1004, the curve error calculation unit 154 calculates, as the curve error, the mean square error between sampling points having the same value of the parameter t, among the sampling points of the estimated contour curve 90 and the correct contour curve 91.

[0109] As described above, the image processing system 10 of this embodiment can generate a trained model 162 that enables finger region estimation that reflects prior knowledge of finger shapes. The image processing system 10 can also generate a trained model 162 that enables finger regions to be correctly separated and extracted when the finger regions are in contact with each other. The image processing system 10 can also directly reflect errors in the contour shapes in the training model 163.

[0110] 11 is an example of an attribute parameter input screen displayed on the display device 36 of the information processing device 3 in step S300. In the example of FIG. 11, the information processing device 3 is a smartphone, and the display device 36 is a touch panel that also serves as the input device 35.

[0111] For example, a usage environment selection box 1101, a target color selection box 1102, and an illuminance selection box 1103 are displayed on the display device 36. In the usage environment selection box 1101, it is possible to select the usage environment in which the input image is to be captured. The usage environment is an example of an attribute parameter, and for example, environmental attributes such as home, outdoors, or office can be selected.

[0112] In the object color selection box 1102, an object color, which is the color of an object included in the input image, can be selected. The object color is an example of an attribute parameter. In the illuminance selection box 1103, the illuminance of the usage environment in which the input image is captured can be selected.

[0113] The attribute parameter input screen in FIG. 11 enables the image processing system 10 to accept input of not only an image but also attribute parameters such as illuminance, lighting color, and the default color of an object, thereby enabling finger area estimation taking the attribute parameters into account.

[0114] 12 is an explanatory diagram showing another example of the method of calculating the error between the estimated contour curve and the correct contour curve in step S308. Here, the finger estimated contour 505 in FIG. 5 will be taken as an example for explanation.

[0115] The correct central contour curve 1201 is an example of a correct contour curve (a contour curve given by curve parameters included in teacher data) stored in the training data 161. The correct central contour curve 1201 may be a correct contour curve for any of a plurality of finger types (index finger, middle finger, ring finger, little finger, etc.), but here it is assumed to be a correct contour curve for the middle finger.

[0116] A contour curve 1202 is an example of an estimated contour curve after a coordinate transformation in which the estimated contour 505 of the finger is transformed using a coordinate transformation determined by a parameter θ. For example, an affine transformation or a projective transformation can be used as the coordinate transformation determined by the parameter θ. The affine transformation and the projective transformation can be expressed by, for example, a 3×3 matrix.

[0117] Fig. 13 is a flowchart showing an example of the curve error calculation process in step S308. An example of curve error calculation for the contour curve 1202 after coordinate conversion and the correct central contour curve 1201 in Fig. 12 will be described below. In the example of Fig. 13, the above-mentioned θ is a parameter in the parametric curve function (i.e., a model parameter of the trained model 162 and the training model 163). In the example of Fig. 13, the above-mentioned θ is learned independently in advance using a correct contour curve of any one of a plurality of finger types (index finger, middle finger, ring finger, little finger, etc.), and may be set as a fixed parameter when learning the parameter in the parametric curve function (i.e., a model parameter of the trained model 162 and the training model 163).

[0118] In step S1301, the curve error calculation unit 154 selects the correct central contour curve 1201. In step S1302, the curve error calculation unit 154 converts the coordinates of the estimated contour 505 into the contour curve 1202 using the parameter θ in the model under training 163.

[0119] In step 1303, the curve error calculation unit 154 finds an error between the contour curve 1202 after coordinate transformation and the correct central contour curve 1201. The curve error calculation unit 154 may calculate the error using the procedure shown in Fig. 8 or 10, or may represent the contour curve as a two-dimensional vector (where a contour position is set to 1 and other positions are set to 0) and calculate the error using the cross entropy or IOU (Intersection over union) between the two-dimensional vector of the contour curve 1202 after coordinate transformation and the two-dimensional vector of the correct central contour curve 1201.

[0120] In step 1304, the regression estimation training unit 153 determines whether the curve error is less than a predetermined threshold δ. When the regression estimation training unit 153 determines that the curve error is equal to or greater than the predetermined threshold δ (S1304: NO), in step S1306, the regression estimation training unit 153 updates the model parameters in the model under training 163 in a direction that reduces the error, and returns to step S306. The method of correcting the model parameters by the regression estimation training unit 153 depends on the machine learning method of the model. For example, when a deep learning model is used, the model parameters can be corrected using stochastic gradient descent (SGD).

[0121] In step S1302, the input to be subjected to coordinate conversion is not limited to a contour line, but may be only sampled feature points such as end points and branch points on a curve. Also, the input data is not limited to an image (two-dimensional data), but may be three-dimensional or higher dimensional data such as (x, y, z).

[0122] If regression estimation training unit 153 determines that the curve error is less than the predetermined threshold δ (S1304: YES), it ends the training phase and saves model in training 163 as post-training model 162 in step S1305.

[0123] 13, the image processing system 10 can generate a trained model 162 that takes into account the relative positional relationships of multiple contour regions. Furthermore, when extracting a finger region from an input image, for example, the image processing system 10 extracts a contour region but does not estimate whether the pixels inside the contour region are background or not, and can efficiently extract an object region with a small amount of calculation.

[0124] The present invention is not limited to the above-described embodiment, but includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0125] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0126] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0127] 3 Information processing device, 5 Server, 10 Image processing system, 31 CPU, 32 Memory, 33 Auxiliary storage device, 34 Communication device, 35 Input device, 36 Display device, 51 CPU, 52 Memory, 53 Auxiliary storage device, 54 Communication device, 55 Input device, 56 Display device, 151 Training unit, 152 Contour curve parameter extraction unit, 153 Regression estimation training unit, 154 Curve error calculation unit, 156 Contour reproduction unit, 161 Training data, 162 Post-training model

Claims

1. An image processing device, A processor and a memory are included. The memory includes: a model that, when an image is input, outputs an outline of an object included in the image as an estimated function using a parametric function, and includes parameters of the parametric function as model parameters; An input image; and teacher data indicating a correct function indicating a correct contour which is a correct solution for the contour of the object included in the input image; The processor, The input image is input to the model to obtain an estimation function for a contour of an object included in the input image; Calculating an error between a correct function indicated by the teaching data and the acquired estimated function; If the calculated error is greater than a predetermined value, the image processing device updates a parameter of the parametric function so as to reduce the calculated error.

2. 2. The image processing device according to claim 1, The processor generates a background separated image from the input image, generates the correct function based on the generated background separated image, and stores the correct function in the memory.

3. 3. The image processing device according to claim 2, The processor generates the correct function by converting the contour of the background separation image into a function defined using parameters and control points.

4. 4. The image processing device according to claim 3, The processor, Dividing the contour of the background separated image into a plurality of partial contours by a predetermined method; The image processing device generates the correct function by converting each of the divided partial contours into a function defined by parameters and control points.

5. 2. The image processing device according to claim 1, The image processing device, wherein the processor calculates the error based on an area of ​​a region surrounded by the correct function and the estimated function.

6. 2. The image processing device according to claim 1, The solution function is a parametric function, The processor, Sampling parameters of the ground truth function and the estimation function; The image processing device calculates the error based on the square of the distance between coordinate points having the same value of the sampled parameters in the correct function and the estimated function.

7. 2. The image processing device according to claim 1, The image processing device, wherein the processor calculates the error based on an area of ​​a region surrounded by the correct function and a transformed estimated function obtained by performing coordinate transformation on the estimated function.

8. 2. The image processing device according to claim 1, The processor calculates the error based on the square of the distance between a coordinate point obtained by performing coordinate transformation on the coordinate point calculated by sampling the parameters of the estimation function and a coordinate point calculated by sampling the parameters of the correct function.

9. 2. The image processing device according to claim 1, The memory includes: A plurality of the models are held, each of which is defined for each attribute parameter; The processor, Obtaining attribute parameters of the input image; An image processing apparatus that acquires an estimation function for a contour of an object included in the input image by inputting the input image into a model corresponding to attribute parameters of the acquired input image.

10. 10. The image processing device according to claim 9, The attribute parameters include at least one of a parameter indicating a shooting environment of the input image, a parameter indicating a color of an object included in the input image, and a parameter indicating an illuminance at the time of shooting the input image.

11. 10. The image processing device according to claim 9, connected to an input device and a display device; The processor, Displaying the types of the attribute parameters on the display device; An image processing device that obtains, via the input device, values ​​of attribute parameters of the displayed type.

12. An image processing method by an image processing device, comprising: The image processing device includes a processor and a memory. The memory includes: a model that, when an image is input, outputs an outline of an object included in the image as an estimated function using a parametric function, and includes parameters of the parametric function as model parameters; An input image; and teacher data indicating a correct function indicating a correct contour which is a correct solution for the contour of the object included in the input image; The image processing method includes: the processor inputs the input image into the model to obtain an estimation function for a contour of an object included in the input image; The processor calculates an error between a correct function indicated by the teaching data and the obtained estimated function; The image processing method, wherein, when the calculated error is greater than a predetermined value, the processor updates parameters of the parametric function so as to reduce the calculated error.

Images