Learning device, image segmentation device, learning method, and program

Abstract

A learning device that learns a model, which is a model in an image segmentation device that segments an object to be recognized in an image using the model, includes: a first evaluation value acquisition unit that acquires a first evaluation value that is calculated using a first loss function and that is used to acquire a second evaluation value that is calculated using a second loss function and that is used to segment the object to be recognized in the image; the first loss function evaluates a feature quantity for recognizing the recognition object in the image; a second evaluation value acquisition unit that acquires a second evaluation value that is calculated using a second loss function that evaluates the continuity of a recognition target recognized in the image; and a learning execution unit that performs learning of the optimization model by performing error back propagation on the basis of the first evaluation value and the second evaluation value.

Description

[0001] Related Applications

[0002] This application is based on Japanese Patent Application No. 2022-139818 filed on September 2, 2022, and the contents thereof are incorporated herein by reference. Technical Field

[0003] The present invention relates to a learning device, an image segmentation device, a learning method and a program. Background Art

[0004] In daily life and industrial situations, linear objects are often recognized. Therefore, even in the field of image recognition technology that has been used in recent years, a technology for accurately recognizing linear objects is being sought.

[0005] For example, Patent Document 1 describes a method for improving the recognition resolution of linear cracks and the like by using semantic segmentation using deep learning that assigns labels to all pixels in an image.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application Publication No. 2021-184163 Summary of the invention

[0009] Technical problem to be solved by the invention

[0010] However, conventional methods cannot accurately and effectively evaluate characteristics of linear objects, such as continuity and indefinite shape of contours.

[0011] An object of the present invention is to provide a technology for accurately and effectively recognizing a linear object.

[0012] Solutions for solving technical problems

[0013] The present invention adopts the following structure to solve the above-mentioned problems.

[0014] A learning device according to one aspect of the present invention learns a model, wherein the model is a model in an image segmentation device, and the image segmentation device uses the model to segment an identification object in an image. The learning device comprises: a first evaluation value acquisition unit, which acquires a first evaluation value, wherein the first evaluation value is calculated using a first loss function, and the first loss function evaluates a feature value for identifying the identification object in the image; a second evaluation value acquisition unit, which acquires a second evaluation value, wherein the second evaluation value is calculated using a second loss function, and the second loss function evaluates the continuity of the identification object identified in the image; and a learning execution unit, which optimizes the learning of the model by performing error back propagation based on the first evaluation value and the second evaluation value.

[0015] According to the above configuration, the accuracy of the evaluation of the continuity feature of the linear object is improved, so that a model that accurately recognizes the linear object can be generated.

[0016] Furthermore, the learning device may include a third evaluation value acquisition unit, the third evaluation value acquisition unit acquires a third evaluation value, the third evaluation value is calculated using a third loss function, the third loss function evaluates the feature quantity that emphasizes the contour of the recognition object, and the learning execution unit optimizes the learning of the model by performing error back propagation based on the first evaluation value, the second evaluation value, and the third evaluation value. As a result, the accuracy of the evaluation of the feature of the indefinite shape contour of the linear object is improved, so that a model that accurately recognizes the linear object can be generated.

[0017] Furthermore, the second evaluation value acquisition unit may include: a first element product calculation unit that calculates a first element product matrix obtained by multiplying the correct matrix data showing the position of the correct segmentation mask of the recognition object and the matrix data after the position of the correct segmentation mask is shifted; a second element product calculation unit that calculates a second element product matrix obtained by multiplying the predicted matrix data showing the predicted position of the segmentation mask of the recognition object and the matrix data after the predicted position of the segmentation mask is shifted in the same manner as the correct segmentation mask; and an evaluation value calculation unit that calculates the second evaluation value using the difference between the first element product matrix and the second element product matrix as the second loss function. Thus, it is possible to learn a model suitable for evaluating the continuity feature of a linear object.

[0018] Furthermore, the recognition object may be a linear object, and the first element product calculation unit may calculate a first element product matrix obtained by multiplying the elements of the matrix data after the position of the correct segmentation mask is moved along the length direction of the linear shape. Thus, it is possible to learn a model suitable for evaluating the continuity feature of the linear object.

[0019] An image segmentation device according to one aspect of the present invention uses a model to segment an object to be identified in an image, the model being a model learned by the above-mentioned learning device, and the image segmentation device includes a segmentation execution unit, the segmentation execution unit using the learned model to segment the object to be identified in the image. As a result, the accuracy of the evaluation of the continuity feature of a linear object is improved, so that the linear object can be accurately identified.

[0020] A learning method according to one aspect of the present invention learns a model, wherein the model is a model in an image segmentation device, and the image segmentation device uses the model to segment an object to be identified in an image, and the learning method includes the following steps: obtaining a first evaluation value, wherein the first evaluation value is calculated using a first loss function, wherein the first loss function evaluates a feature quantity for identifying the object to be identified in the image; obtaining a second evaluation value, wherein the second evaluation value is calculated using a second loss function, wherein the second loss function evaluates the continuity of the object to be identified in the image; and optimizing the learning of the model by performing error back propagation based on the first evaluation value and the second evaluation value. According to the above structure, the accuracy of the evaluation of the continuity feature of a linear object is improved, so that a model for accurately identifying a linear object can be generated.

[0021] A program according to one aspect of the present invention causes a computer to function as a first evaluation value acquisition unit, a second evaluation value acquisition unit, and a learning execution unit, the computer executing learning of a model, the model being a model in an image segmentation device, the image segmentation device using the model to segment an object to be identified in an image, the first evaluation value acquisition unit acquiring a first evaluation value, the first evaluation value being calculated using a first loss function, the first loss function evaluating a feature quantity for identifying the object to be identified in the image, the second evaluation value acquisition unit acquiring a second evaluation value, the second evaluation value being calculated using a second loss function, the second loss function evaluating the continuity of the object to be identified in the image, the learning execution unit optimizing the learning of the model by performing error back propagation based on the first evaluation value and the second evaluation value. According to the above structure, the accuracy of the evaluation of the continuity feature of a linear object is improved, so a model for accurately identifying a linear object can be generated.

[0022] Effects of the Invention

[0023] According to the present invention, it is possible to provide a technology for accurately and efficiently recognizing a linear object. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a diagram schematically showing a learning device 1 and an image segmentation device 2 according to an embodiment of the present invention.

[0025] Figure 2 It is a diagram showing an example of the hardware configuration of the learning device 1 and the image segmentation device 2 according to the embodiment of the present invention.

[0026] Figure 3 1 is a diagram showing an example of the functional configuration of the learning device 1 and the image segmentation device 2 according to the embodiment of the present invention.

[0027] Figure 4 This is a flowchart showing the steps of learning a model performed by the learning device 1 according to the embodiment of the present invention.

[0028] Figure 5 It is a diagram for explaining a method of calculating a second evaluation value by the learning device 1 according to the embodiment of the present invention.

[0029] Figure 6 1 is a flowchart showing the steps of segmentation performed by the image segmentation device 2 according to the embodiment of the present invention.

[0030] Figure 7 It is a diagram for explaining the segmentation of a linear object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Hereinafter, an embodiment (hereinafter, also referred to as "the present embodiment") according to one aspect of the present invention will be described with reference to the accompanying drawings. However, the embodiment described below is merely an example of the present invention in any key point. Of course, various improvements and deformations can be made without departing from the scope of the present invention. That is, when implementing the present invention, a specific structure corresponding to the embodiment may also be appropriately adopted. In addition, although the data appearing in the present embodiment is described in natural language, more specifically, it is specified in a simulation language, command, parameter, machine language, etc. that can be recognized by a computer.

[0032] §1 Application Examples

[0033] use Figure 1 An example of a case where the present invention is applied will be described. Figure 1 1 is a diagram showing an overview of the learning device 1 and the image segmentation device 2 of the present invention. The image segmentation device 2 uses a learned model learned by the learning device 1 in a segmentation execution unit 201 to perform instance segmentation of an identification target in an image.

[0034] The learning model that has been learned by the learning device 1 is suitable for segmenting linear objects such as metal wires and electric wires. The learning device 1 obtains a first evaluation value in the first evaluation value acquisition unit 101, the first evaluation value is calculated using a first loss function, obtains a second evaluation value in the second evaluation value acquisition unit 102, the second evaluation value is calculated using a second loss function, and performs learning by error back propagation based on the first evaluation value and the second evaluation value in the learning execution unit 104. The first evaluation value is a value that evaluates a feature value used to identify an object to be identified in an image, and the second evaluation value is a value that evaluates the continuity of the object to be identified in the image.

[0035] §2 Structure Example

[0036] (1. Hardware structure)

[0037] Next, use Figure 2 An example of the hardware configuration of the learning device 1 and the image segmentation device 2 according to the present embodiment will be described.

[0038] like Figure 2As shown, the learning device 1 is a computer having a processor 11, a main memory 12, an input / output interface 13, a communication interface 14, and a storage device 15 as hardware resources. The storage device 15 is a computer-readable recording medium such as a semiconductor memory (e.g., a volatile memory, a non-volatile memory) or a disk medium (e.g., a magnetic recording medium, an optical magnetic recording medium). The storage device 15 stores a program for causing the processor 11 to execute a learning model for segmenting an object to be recognized in an image, a model to be learned, a data set of images used for learning, and the like. The program is read from the storage device 15 by the main memory 12, and is interpreted and executed by the processor 11, thereby executing various functions.

[0039] The image segmentation device 2 is a computer having a processor 21, a main memory 22, an input / output interface 23, a communication interface 24, and a storage device 25 as hardware resources. The storage device 25 is a computer-readable recording medium such as a semiconductor memory (e.g., a volatile memory, a non-volatile memory) or a disk medium (e.g., a magnetic recording medium, an optical magnetic recording medium). The storage device 25 stores a program for causing the processor 21 to perform segmentation of an identified object in an image using a learned model and the learned model. The program is read from the storage device 25 by the main memory 22, and is interpreted and executed by the processor 21, thereby executing various functions.

[0040] (2. Functional structure)

[0041] Next, use Figure 3 An example of the functional configuration of the learning device 1 and the image segmentation device 2 according to the present embodiment will be described. Figure 3 1 is a block diagram showing functional modules executed by the processor 11 of the learning device 1 and the processor 12 of the image segmentation device 2. Figure 3 As shown, the functional modules executed by the processor 11 of the learning device 1 include a first evaluation value acquisition unit 101, a second evaluation value acquisition unit 102, a third evaluation value acquisition unit 103, and a learning execution unit 104. The second evaluation value acquisition unit 102 includes a first factor product calculation unit 1021, a second factor product calculation unit 1022, and an evaluation value calculation unit 1023.

[0042] Furthermore, the functional blocks executed by the processor 21 of the image segmentation device 2 include a segmentation execution unit 201 .

[0043] §3 Action Examples

[0044] Next, use Figure 4The flowchart of the present embodiment illustrates the steps of model learning performed by the learning device 1. The image data set used for machine learning includes linear objects such as wires and electrical wires. In machine learning, these linear objects are used as recognition objects, and the model is optimized in a manner that enables correct segmentation.

[0045] First, in the first evaluation value acquisition unit 101, a first evaluation value is calculated using a loss function (first loss function) that evaluates a feature quantity for instance segmentation of an image for learning (step S101). As the loss function, for example, a discriminative loss can be used.

[0046] Then, in the second evaluation value acquisition unit 102 , a second evaluation value is calculated using a loss function (second loss function) that evaluates the continuity of the identification target identified in the learning image (step S102 ).

[0047] Regarding the calculation of the second evaluation value, use Figure 5 Detailed description. First, the first element product calculation unit 1021 calculates the first element product matrix (MT1), which is obtained by multiplying the correct answer matrix (GT1) corresponding to the correct answer data (Ground Truth: reference true value) showing the correct position of the segmentation mask relative to the recognition object in the image (the area of ​​the element shown by "1" in the figure) and the respective elements of the matrix (GT2) after the position of the segmentation mask is moved relative to GT1 (step (1)).

[0048] When the recognition target is a linear object, the first element product calculation unit 1021 can set a matrix GT2 in which the position of the correct segmentation mask in the matrix GT1 is shifted in the longitudinal direction of the linear shape.

[0049] Here, the element product matrix MT1 (Mk, Δh, Δw) of the matrix GT1 (M (0, 0)) and the matrix GT2 (M (Δh, Δw)) obtained by shifting the position of the mask by h in the vertical direction and by w in the horizontal direction can be expressed by the following equation.

[0050] [Formula 1]

[0051]

[0052] Then, the second element product calculation unit 1022 calculates the second element product matrix (MT2), which is obtained by multiplying the elements of the prediction matrix (GT3) showing the predicted position of the segmentation mask of the identification object (the area of ​​MS surrounded by a frame line in the figure) and the matrix (GT4) after the position of the segmentation mask is moved relative to GT3 (step (2)).

[0053] The movement of the position of the segmentation mask relative to the matrix GT3 is performed in the same manner as the movement of the position of the segmentation mask relative to the matrix GT1 in step (1). The product matrix MT2 of the elements of the matrix GT3 and the matrix GT4 after the position of the mask is moved vertically by h and horizontally by w can be expressed as follows.

[0054] [Formula 2]

[0055]

[0056] Where, [Formula 3]

[0057]

[0058] For matrix GT3,

[0059] [Formula 4]

[0060]

[0061] For Matrix GT4,

[0062] [Formula 5]

[0063]

[0064] is the element product matrix MT2.

[0065] Then, the evaluation value calculation unit 1023 uses the difference between the first element product matrix MT1 obtained in step (1) and the second element product matrix MT2 obtained in step (2) as the second loss function to calculate the second evaluation value (step (3)). smooth ) can be expressed as follows. In the following formula, A[i,j] represents the i-th row and j-th column element of matrix A. And N corresponds to the batch size during learning. By using L smooth Although learning that takes continuity into consideration can be achieved, it also has the effect of blurring the contour of the object to be recognized. Therefore, a loss function that exerts the effect of emphasizing the contour is used in the subsequent steps.

[0066] [Formula 6]

[0067]

[0068] Then, the third evaluation value acquisition unit 103 calculates the third evaluation value using a loss function (third loss function) that evaluates the feature quantity that emphasizes the contour of the recognized object (step S103). The third evaluation value is obtained by calculating the loss function using M', which is a value obtained by weighting the correct answer data (GT) of the semantic mask of the recognized object by the contour. Specifically, for example, the following focal loss (L edge ).

[0069] [Formula 7]

[0070]

[0071] M' is a semantic GT mask weighted by contours, and is assigned by the following formula. F is an edge image calculated from the correct data and normalized to the range of [0,1]. In the edge image, pixels above the threshold thr are weighted by α (for example, α=2) times to emphasize them.

[0072] [Formula 8]

[0073]

[0074] By calculating the edge-emphasized mask M' and the focal loss of the predicted mask output by the model, it is possible to learn the feature amount of the contour-emphasized portion.

[0075] Then, the learning execution section 104 performs error back propagation based on the first evaluation value calculated in step S101 , the second evaluation value calculated in step S102 , and the third evaluation value calculated in step S103 , thereby optimizing the model (step S104 ).

[0076] Next, use Figure 6 The flowchart of FIG. 1 illustrates the segmentation procedure performed by the image segmentation device 2 according to the present embodiment.

[0077] The image segmentation device 2 acquires image data to be segmented (step S201 ).

[0078] Then, the segmentation execution unit 201 executes instance segmentation of the image data using the model optimized by the learning device 1 (step S202 ).

[0079] As described above, according to this embodiment, in addition to the evaluation value calculated by the loss function such as the discrimination loss in the existing method, the evaluation value calculated by the loss function that evaluates the continuity of the recognition object is also used to optimize the learning model for image segmentation. As a result, a learning model suitable for evaluating the continuity characteristics of linear objects can be generated. In the segmentation of elongated objects such as wires, when the actual Figure 7 When the shape has two points connected as shown in (A), if prediction based on segmentation is performed, it may result in the following Figure 7 (B) shows a shape that is disconnected in the middle. According to this embodiment, by evaluating the model in consideration of continuity, it is possible to obtain a model that does not generate such a mask that is disconnected in the middle.

[0080] In addition, the learning model for image segmentation is optimized by using an evaluation value calculated using a loss function weighted by the contour of the recognized object, so that a model that is also adapted to the features of the irregular contour of a linear object can be obtained. For example, in the case of a fence made of wire, it is sometimes difficult to recognize the contour of the overlapping part with high accuracy. According to this embodiment, even in such a case, a mask can be generated with good accuracy.

[0081] The segmentation model involved in this embodiment can perform instance segmentation with high accuracy and can also be applied to semantic segmentation.

[0082] The embodiments of the present invention have been described in detail above, but the above description is only an example of the present invention in any key point. Of course, various improvements and modifications can be made without departing from the scope of the present invention. In addition, part or all of the above embodiments can also be described as the following supplementary notes, but are not limited to the following descriptions.

[0083] (Note 1)

[0084] A learning device is used to learn a model, wherein the model is a model in an image segmentation device, and the image segmentation device uses the model to segment an identification object in an image.

[0085] The learning device comprises:

[0086] A first evaluation value acquisition unit, which acquires a first evaluation value, wherein the first evaluation value is calculated using a first loss function, and the first loss function evaluates a feature quantity for identifying the identification object in the image;

[0087] a second evaluation value acquiring unit, which acquires a second evaluation value, wherein the second evaluation value is calculated using a second loss function, and the second loss function evaluates the continuity of the identified object identified in the image; and

[0088] The learning execution unit performs learning to optimize the model by performing error back propagation based on the first evaluation value and the second evaluation value.

[0089] (Note 2)

[0090] According to the learning device described in Supplement 1, the learning device includes a third evaluation value acquisition unit, which acquires a third evaluation value, wherein the third evaluation value is calculated using a third loss function, and the third loss function evaluates a feature quantity that emphasizes the contour of the recognition object.

[0091] The learning execution unit performs learning to optimize the model by performing error back propagation based on the first evaluation value, the second evaluation value, and the third evaluation value.

[0092] (Note 3)

[0093] According to the learning device of Supplementary Note 1, the second evaluation value acquisition unit comprises:

[0094] a first element product calculation unit that calculates a first element product matrix obtained by multiplying elements of a correct matrix data indicating a position of a correct segmentation mask of the recognition object and matrix data obtained by shifting the position of the correct segmentation mask;

[0095] a second element product calculation unit that calculates a second element product matrix obtained by multiplying elements of prediction matrix data indicating a predicted position of the segmentation mask of the recognition object and matrix data obtained by shifting the predicted position of the segmentation mask in the same manner as the correct segmentation mask; and

[0096] The evaluation value calculation unit calculates a second evaluation value by using a difference between the first element product matrix and the second element product matrix as the second loss function.

[0097] (Note 4)

[0098] According to the learning device described in Supplementary Note 3, the recognition object is a linear object,

[0099] The first element product calculation unit calculates a first element product matrix obtained by multiplying elements of matrix data obtained by shifting the position of the correct segmentation mask in the length direction of the linear shape.

[0100] (Note 5)

[0101] An image segmentation device uses a model to segment an identifiable object in an image.

[0102] The model is a model learned by the learning device described in Supplement 1,

[0103] The image segmentation device includes a segmentation execution unit configured to segment the identification target in an image using the learned model.

[0104] (Note 6)

[0105] A learning method is provided for learning a model, wherein the model is a model in an image segmentation device, and the image segmentation device uses the model to segment an identifiable object in an image.

[0106] The learning method comprises the following steps:

[0107] Acquire a first evaluation value, where the first evaluation value is calculated using a first loss function, where the first loss function evaluates a feature quantity for identifying the identification object in the image;

[0108] acquiring a second evaluation value, wherein the second evaluation value is calculated using a second loss function, wherein the second loss function evaluates continuity of the identified object identified in the image; and

[0109] The step of optimizing the learning of the model is performed by performing error back propagation based on the first evaluation value and the second evaluation value.

[0110] (Note 7)

[0111] A program that causes a computer to function as a first evaluation value acquisition unit, a second evaluation value acquisition unit, and a learning execution unit, wherein the computer executes learning of a model, the model being a model in an image segmentation device, the image segmentation device using the model to segment an identification object in an image,

[0112] The first evaluation value acquisition unit acquires a first evaluation value, the first evaluation value being calculated using a first loss function, the first loss function evaluating a feature quantity for identifying the recognition object in the image,

[0113] The second evaluation value acquisition unit acquires a second evaluation value, wherein the second evaluation value is calculated using a second loss function, and the second loss function evaluates the continuity of the identified object identified in the image.

[0114] The learning execution unit performs learning to optimize the model by performing error back propagation based on the first evaluation value and the second evaluation value.

[0115] Description of Reference Numerals

[0116] 1…learning device; 2…image segmentation device; 11, 21…processor; 12, 22…main memory; 13, 23…input and output interface; 14, 24…communication interface; 15, 25…storage device; 101…first evaluation value acquisition unit; 102…second evaluation value acquisition unit; 103…third evaluation value acquisition unit; 104…learning execution unit; 201…segmentation execution unit; 1021…first element product calculation unit; 1022…second element product calculation unit; 1023…evaluation value calculation unit.

Claims

1. A learning device for learning a model, wherein the model is a model in an image segmentation device, and the image segmentation device uses the model to segment an identifiable object in an image. The learning device comprises: A first evaluation value acquisition unit, which acquires a first evaluation value, wherein the first evaluation value is calculated using a first loss function, and the first loss function evaluates a feature quantity for identifying the identification object in the image; a second evaluation value acquiring unit, which acquires a second evaluation value, wherein the second evaluation value is calculated using a second loss function, and the second loss function evaluates the continuity of the identified object identified in the image; as well as The learning execution unit performs learning to optimize the model by performing error back propagation based on the first evaluation value and the second evaluation value.

2. The learning device according to claim 1, wherein: The learning device includes a third evaluation value acquisition unit, the third evaluation value acquisition unit acquiring a third evaluation value, the third evaluation value being calculated using a third loss function, the third loss function evaluating a feature quantity that emphasizes the contour of the recognition object, The learning execution unit performs learning to optimize the model by performing error back propagation based on the first evaluation value, the second evaluation value, and the third evaluation value.

3. The learning device according to claim 1, wherein: The second evaluation value acquisition unit includes: a first element product calculation unit that calculates a first element product matrix obtained by multiplying elements of a correct matrix data indicating a position of a correct segmentation mask of the recognition object and matrix data obtained by shifting the position of the correct segmentation mask; a second element product calculation unit that calculates a second element product matrix obtained by multiplying elements of prediction matrix data indicating a predicted position of the segmentation mask of the recognition object and matrix data obtained by moving the predicted position of the segmentation mask in the same manner as the correct segmentation mask; as well as The evaluation value calculation unit calculates a second evaluation value by using a difference between the first element product matrix and the second element product matrix as the second loss function.

4. The learning device according to claim 3, wherein: The recognition object is a linear object. The first element product calculation unit calculates a first element product matrix obtained by multiplying elements of matrix data obtained by shifting the position of the correct segmentation mask in the length direction of the linear shape.

5. An image segmentation device, using a model to segment an identifiable object in an image, The model is a model learned by the learning device according to claim 1, and the image segmentation device includes a segmentation execution unit that uses the learned model to perform segmentation of the recognition target in the image.

6. A learning method for learning a model, wherein the model is a model in an image segmentation device, and the image segmentation device uses the model to segment an identifiable object in an image. The learning method comprises the following steps: Acquire a first evaluation value, where the first evaluation value is calculated using a first loss function, where the first loss function evaluates a feature quantity for identifying the identification object in the image; Obtaining a second evaluation value, where the second evaluation value is calculated using a second loss function, where the second loss function evaluates the continuity of the identified object identified in the image; as well as The learning of optimizing the model is performed by performing error back propagation according to the first evaluation value and the second evaluation value.

7. A program that causes a computer to function as a first evaluation value acquisition unit, a second evaluation value acquisition unit, and a learning execution unit, wherein the computer executes learning of the model, the model being a model in an image segmentation device, and the image segmentation device uses the model to segment an identification object in an image, The first evaluation value acquisition unit acquires a first evaluation value, the first evaluation value being calculated using a first loss function, the first loss function evaluating a feature quantity for identifying the recognition object in the image, The second evaluation value acquisition unit acquires a second evaluation value, wherein the second evaluation value is calculated using a second loss function, and the second loss function evaluates the continuity of the identified object identified in the image. The learning execution unit performs learning to optimize the model by performing error back propagation based on the first evaluation value and the second evaluation value.

Images