Object detection ai learning method and object detection ai learning device

Abstract

Provided are an object detection AI learning method and an object detection AI learning device that suppress variation in a plurality of candidate regions and make it possible to curb erroneous detection of objects. The object detection AI learning method is implemented by the object detection AI learning device comprising an AI inference processing unit, a coordinate loss calculation unit, a variability calculation unit, a loss calculation unit, and a weight update unit, wherein: the AI inference processing unit uses object detection AI to extract, for each object included in an input image, a plurality of candidate regions in which the object is assumed to be located, and infers coordinates with regard to each of the plurality of candidate regions; the coordinate loss calculation unit calculates an offset of the coordinates from true coordinates of the object, as a coordinate loss, for each of the plurality of candidate regions of each of the objects; the variability calculation unit calculates variation of coordinates pertaining to the plurality of candidate regions, as a coordinate variation loss, for each of the objects; the loss calculation unit calculates a loss on the basis of the coordinate loss and the coordinate variation loss; and the weight update unit updates a weight of the object detection AI so as to reduce the loss.

Description

Object detection AI learning method and object detection AI learning device

[0001] The present invention relates to an object detection AI learning method and an object detection AI learning device.

[0002] Patent Document 1 is known as a technology relating to the learning of a learning model that infers an area where an object exists from an image. In Patent Document 1, paragraph 0021 states, "In S301, the image acquisition unit 201 acquires an input image saved in the storage unit 208. The input image is as shown in FIG. 7( a), and an object to be detected is reflected in the image.", and paragraph 0023 states, "In S303, the inference unit 203 infers the object-likelihood, label, and shape. The shape inferred here is a rectangular Bounding Box that surrounds the object. FIG. 4( b) shows Bounding Boxes inferred at some anchors. Each grid 405 indicates an anchor. 406, 408, and 410 are anchors, 407 is the Bounding Box inferred at anchor 406, 409 is the Bounding Box inferred at anchor 408, and 411 is the Bounding Box inferred at anchor 410." Then, paragraph 0032 states, "In S306, the loss calculation unit 306 calculates the loss related to the object-likeness and the loss related to the Bounding Box regression.", and paragraph 0040 states, "In S307, the parameter update unit 207 updates the parameters based on the loss calculated in S306."

[0003] JP 2023-48873 A

[0004] In Patent Document 1, as shown in FIG. 4(b) of the same document, multiple bounding boxes (candidate regions) are inferred and used to train a learning model. When detecting an object that is the same color as the background or when detecting an object that is far away, the boundaries of the object may become unclear, resulting in large variations in the multiple inferred candidate regions. An object detection AI (learning model) that has learned multiple candidate regions with such large variations infers the candidate region of the object within the range of this variation, which may result in erroneously detecting the object as a candidate region located far from the actual position of the object.

[0005] Therefore, an object of the present invention is to provide an object detection AI learning method and an object detection AI learning device that can reduce the variation among multiple candidate regions and suppress erroneous object detection.

[0006] In order to solve the above problem, the object detection AI learning method disclosed herein is an object detection AI learning method realized by an object detection AI learning device having, for example, an AI inference processing unit, a coordinate loss calculation unit, a variability calculation unit, a loss calculation unit, and a weight update unit, in which the AI ​​inference processing unit uses the object detection AI to extract, for each object included in an input image, multiple candidate areas where the object is expected to be located and infer coordinates for each of the multiple candidate areas, the coordinate loss calculation unit calculates, for each of the multiple candidate areas for each object, the deviation of the coordinates from the correct coordinates of the object as a coordinate loss, the variability calculation unit calculates, for each object, the variation of the coordinates for the multiple candidate areas as a coordinate variation loss, the loss calculation unit calculates a loss based on the coordinate loss and the coordinate variation loss, and the weight update unit updates the weights of the object detection AI to reduce the loss.

[0007] In addition, the object detection AI learning device disclosed herein includes, for example, an AI inference processing unit that uses object detection AI to extract, for each object included in an input image, multiple candidate areas where the object is expected to be located and infer coordinates for each of the multiple candidate areas; a coordinate loss calculation unit that calculates, for each of the multiple candidate areas for each object, the deviation of the coordinates from the correct coordinates of the object as a coordinate loss; a variability calculation unit that calculates, for each object, the variation of the coordinates for the multiple candidate areas as a coordinate variation loss; a loss calculation unit that calculates a loss based on the coordinate loss and the coordinate variation loss; and a weight update unit that updates the weights of the object detection AI so as to reduce the loss.

[0008] According to the present invention, it is possible to provide an object detection AI training method and an object detection AI training device that can suppress variation among multiple candidate regions and reduce false detection of objects. Other objects and novel features will become apparent from the description of this specification and the accompanying drawings.

[0009] FIG. 1 is a diagram showing an example of a functional block diagram of an object detection AI learning device of Example 1. FIG. 2 is a diagram showing an example of a flowchart of an object detection AI learning method realized by the object detection AI learning device of Example 1. FIG. 3 is a diagram specifically explaining a candidate area. FIG. 4 is a diagram showing an example of a physical configuration of an object detection AI learning device. FIG. 5 is a diagram showing an example of a functional block diagram of an object detection AI learning device of Example 2. FIG. 6 is a diagram showing an example of a flowchart of an object detection AI learning method realized by the object detection AI learning device of Example 2. FIG. 7 is a diagram showing an example of a functional block diagram of an object detection AI learning device of Example 3. FIG. 8 is a diagram showing an example of a flowchart of an object detection AI learning method realized by the object detection AI learning device of Example 3. FIG. 9 is a diagram showing an example of a functional block diagram of an object detection AI learning device of Example 4.

[0010] The present disclosure will be described in detail below with reference to the drawings. It should be noted that the present disclosure is not limited to the examples described below. These examples are merely illustrative, and the present invention can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Furthermore, in the drawings used in the following description, the same reference numerals are used to designate common devices and equipment, and descriptions of devices, equipment, and operations that have already been described may be omitted.

[0011] 1 is a diagram illustrating an example of a functional block diagram of an object detection AI learning device 100 according to Example 1. The object detection AI learning device 100 includes an AI inference processing unit 101, a variability calculation unit 102, a loss calculation unit 103, a weight update unit 104, and a coordinate loss calculation unit 105.

[0012] 2 is a diagram showing an example of a flowchart of the object detection AI learning method realized by the object detection AI learning device 100 of Example 1. Using FIG. 2, the operation of each component of the object detection AI learning device 100 shown in FIG. 1 will be described.

[0013] In step S101, the AI ​​inference processing unit 101 acquires input data. The input data is data indicating an image that is the target of inference by the object detection AI learning device 100, and includes, for example, image data or point cloud data obtained by Lidar or the like.

[0014] In step S102, the coordinate loss calculation unit 105 acquires correct answer data indicating which part of the image indicated by the input data (hereinafter referred to as the "input image") the object to be detected is located in. Here, the correct answer data is the coordinates (correct answer coordinates) of a rectangular area indicating the position of the object in the input image. The correct answer coordinates are, for example, two-dimensional coordinates with the lower left corner of the input image as the origin, and are expressed by the coordinates of the lower left and upper right corner of the area. Note that the object may be a vehicle, a pedestrian, a sign, a road, etc.

[0015] In step S103, the AI ​​inference processing unit 101 uses a deep neural network (DNN) to extract, for each object included in the input image, multiple candidate areas where the object is expected to be located, and infers coordinates for each of the multiple candidate areas. FIG. 3 is a diagram specifically illustrating candidate areas. The input image 10 shown in FIG. 3 includes vehicles 11A, 11B, and 11C. Here, the candidate area 13 for vehicle 11A will be described as an example. As shown in FIG. 3, the AI ​​inference processing unit 101 extracts multiple candidate areas for each object, such as the candidate area 13. The candidate areas are represented as rectangles, similar to the correct answer data. The bottom left corner of the input image is then used as the origin 12, and the coordinates of each candidate area 13 are inferred. Similar to the correct answer data, the coordinates of the candidate areas are represented by two-dimensional coordinates of the bottom left corner and the top right corner of the candidate area.

[0016] In step S104 of FIG. 2 , the coordinate loss calculation unit 105 calculates the coordinate deviation from the correct coordinate of the object as the coordinate loss for each of the multiple candidate regions for each object. As described above, the correct coordinates and the coordinates of the candidate region are represented by two-dimensional coordinates in the upper right and lower left of the region, and are therefore represented by four components. Therefore, the difference for each component between the correct coordinates and the coordinates of the candidate region is calculated, and the sum of the differences for each component is calculated as the coordinate loss for that candidate region. Note that the coordinate loss for a candidate region is not limited to the sum of the differences for each component; for example, the average value of the differences for each component or the maximum value of the differences for each component can also be used.

[0017] In step S105, the variability calculation unit 102 calculates, for each object, the variability of coordinates for multiple candidate regions that represent the same object as a coordinate variability loss. For example, the variance for each component of the coordinates of the candidate region is calculated, and the sum of the variances for each component is used as the coordinate variability loss. Note that the coordinate variability loss is not limited to this, and for example, the average value of the variances for each component or the maximum value of the variances for each component can also be used as the coordinate variability loss.

[0018] In step S106, the loss calculation unit 103 adds the coordinate loss for each candidate region calculated in step S104 and the coordinate variation loss for each object calculated in step S105, and calculates the sum as the loss.

[0019] In step S107, the weight update unit 104 updates the weights of the DNN by backpropagation or the like so as to reduce the loss calculated in step S106.

[0020] In step S108, the weight update unit 104 determines whether the loss after the weight update is equal to or less than a predetermined value. If it is equal to or less than the predetermined value, the learning is terminated, and if it is greater than the predetermined value, the process returns to step S101.

[0021] FIG. 4 is a diagram showing an example of the physical configuration of the object detection AI learning device 100. The object detection AI learning device 100 is realized by an information processing device 40 including, as shown in FIG. 4 , a processor (CPU or GPU) 41, a memory 42, a storage device 43, an input device 44, an output device 45, a communication device 46, and a bus 47 as its main components. The processor 41 functions as a functional unit that provides a predetermined function by executing processing in accordance with a program loaded in the memory 42. The storage device 43 stores data and programs used by the functional unit. The input device 44 is a keyboard, a pointing device, etc., and the output device 45 is a display, etc. The communication device 46 enables communication with other information processing devices via a network 4. These are connected to each other so that they can communicate with each other via a bus 47.

[0022] Note that object detection AI learning device 100 does not need to be realized by a single information processing device, but may be realized by multiple information processing devices. Also, some or all of the functions of object detection AI learning device 100 may be realized as an application on the cloud.

[0023] Conventionally, when multiple candidate regions where an object contained in an input image is expected to be located are extracted, each candidate region is treated as representing an independent object. Therefore, when multiple candidate regions representing the same object are extracted, there is a problem of falsely detecting the presence of multiple objects. To address this problem, non-maximum suppression (NMS) processing is performed. When candidate regions with large overlapping areas are extracted, one is selected as representing the same object and the others are deleted, thereby preventing such false detection. However, when multiple candidate regions representing the same object but with small overlapping areas are extracted, the candidate regions are not deleted, and the object is detected as being present in multiple locations, even when NMS is used.

[0024] In the present disclosure, coordinate variation is included in the loss, and the weights of the DNN are updated to reduce the loss, thereby reducing the coordinate variation of candidate regions representing the same object. Therefore, when multiple candidate regions representing the same object are extracted, the overlapping areas between these candidate regions become large. As a result, only one candidate region representing the same object is selected by the NMS, thereby reducing false detection of objects.

[0025] Furthermore, safety can be improved by installing an object detection AI that has been trained using the object detection AI learning method disclosed herein in a vehicle and having the object detection AI provide driving assistance.

[0026] An object detection AI learning method according to a second embodiment will now be described. FIG. 5 is a diagram illustrating an example of a functional block diagram of an object detection AI learning device 200 according to the second embodiment. The object detection AI learning device 200 further includes a class loss calculation unit 203, which infers the class (type) of an object included in a candidate region and updates the weights of the DNN while also taking into account the loss related to the class. Below, differences from the first embodiment will be mainly described.

[0027] Fig. 6 is a diagram showing an example of a flowchart of an object detection AI learning method realized by the object detection AI learning device 200 of Example 2. The operation of each component of the object detection AI learning device 200 shown in Fig. 5 will be described using Fig. 6. Note that step S201 is similar to step S101 of Example 1, and therefore description thereof will be omitted.

[0028] In step S202, the coordinate loss calculation unit 204 acquires the correct coordinates of a rectangular area indicating the position of an object in the input image, and the class loss calculation unit 203 acquires the correct data (correct class) of the class of the object included in the rectangular area.

[0029] In step S203, the AI ​​inference processing unit 201 uses a DNN to extract, for each object included in the input image, multiple candidate areas where the object is assumed to be located, infer the coordinates and the class of the object included in each of the multiple candidate areas, and infer a class probability that indicates the likelihood of the class of the candidate area.

[0030] In step S204, similarly to step S104 in the first embodiment, the coordinate loss calculation unit 204 calculates the coordinate loss for each candidate region.

[0031] In step S205, the class loss calculation unit 203 calculates, for each of the multiple candidate regions for each object, the deviation of the class from the correct class of the object included in the candidate region as the class loss. For example, if the correct class is "vehicle," the class of the candidate region is "vehicle," and the class probability is 80%, the class probability of the correct class is 100%, so the class loss value is 0.2, which is 1 minus 0.8.

[0032] In step S206, the variation calculation unit 202 calculates, for each object, the variation in coordinates for multiple candidate regions representing the same object as a coordinate variation loss, similar to step S105 in Example 1. Then, in addition to the coordinate variation loss, the variation calculation unit 202 further calculates, for each object, the variation in class probability for multiple candidate regions representing the same object as a class variation loss. For example, the variance of class probabilities for classes representing the same object can be used as the class variation loss.

[0033] In step S207, the loss calculation unit 205 adds together the coordinate loss for each candidate region calculated in step S204, the class loss for each candidate region calculated in step S205, and the coordinate variation loss and class variation loss for each object calculated in step S206, and calculates the sum as the loss.

[0034] In step S208, the weight update unit 104 updates the weights of the DNN by backpropagation or the like so as to reduce the loss calculated in step S207.

[0035] In step S209, the weight update unit 104 determines whether the loss after the weight update is equal to or less than a predetermined value. If it is equal to or less than the predetermined value, the learning is terminated, and if it is greater than the predetermined value, the process returns to step S201.

[0036] As described above, by updating the weights of the DNN based on losses including the class loss and the class variation loss, the accuracy of inference by the DNN is further improved.

[0037] An object detection AI learning method according to Example 3 will be described. Fig. 7 is a diagram showing an example of a functional block diagram of an object detection AI learning device 300 according to Example 3. The object detection AI learning device 300 has a configuration that further includes a threshold setting unit 302 in addition to the configuration of the object detection AI learning device 200 shown in Fig. 5 .

[0038] Fig. 8 is a diagram showing an example of a flowchart of an object detection AI learning method realized by the object detection AI learning device 300 of Example 3. Using Fig. 8, the operation of each component of the object detection AI learning device 300 shown in Fig. 7 will be described. Note that the flowchart in Fig. 8 differs from the flowchart in Fig. 6 in steps S301, S302, and S303, so only these steps will be described and descriptions of the other steps will be omitted.

[0039] In step S301, the class loss calculation unit 203 determines whether the coordinate variation loss calculated in step S206 exceeds the coordinate variation threshold value set by the variation degree calculation unit 301. The coordinate variation threshold value indicates the upper limit of the allowable range of the coordinate variation loss.

[0040] In step S302, the loss calculation unit 205 calculates the loss based on the determination result in step S301. Specifically, the loss calculation unit 205 calculates the loss from the coordinate loss and class loss of a candidate region indicating an object whose coordinate variation loss is equal to or less than the coordinate variation threshold, and the coordinate variation loss and class variation loss of the object.

[0041] In step S303, the threshold setting unit 302 updates the coordinate variation threshold when a predetermined condition is met. The predetermined condition may be when the number of learning times exceeds a predetermined number (for example, when 10 learning sets have been completed) or when the coordinate variation loss falls below a predetermined value. In such cases, the threshold setting unit 302 increases the coordinate variation threshold.

[0042] In the early stages of learning, object detection AI has low object detection capabilities and therefore low detection accuracy. Such object detection AI can have difficulty extracting candidate regions and inferring coordinates and classes when detecting a black car at night, detecting a distant object, or when an object's shape cannot be accurately detected because part of it is hidden. Even if such object detection AI in the early stages of learning is trained to learn objects that are difficult to infer, it may not learn correctly and may adversely affect the detection accuracy of objects that it had been able to detect correctly.

[0043] Therefore, in this embodiment, by selecting the objects and candidate regions to be used for calculating the loss based on the magnitude of the coordinate variation loss, it is possible to prevent excessive learning.

[0044] Then, by increasing the coordinate variation threshold as learning progresses, learning can be achieved that corresponds to the detection capability of the object detection AI.

[0045] In this embodiment, the target of loss calculation is determined based on the coordinate variation loss, but the present invention is not limited to this, and the target of loss calculation may be determined using any one of coordinate loss, class loss, and class variation loss. Furthermore, the target of loss calculation may be determined using two or more of coordinate variation loss, coordinate loss, class loss, and class variation loss.

[0046] An object detection AI learning method according to Example 4 will be described. Fig. 9 is a diagram illustrating an example of a functional block diagram of an object detection AI learning device 400 according to Example 4. The object detection AI learning device 400 has a configuration that further includes a notification unit 401 in addition to the configuration of the object detection AI learning device 300 in Fig. 7 .

[0047] The notification unit 401 notifies the person in charge of training the object detection AI (the trainer) of candidate areas that indicate objects whose coordinate variation loss is greater than the coordinate variation threshold.

[0048] This allows the learner to select whether to use the candidate area as a learning target for object detection and whether to reset the training data.

[0049] Although the above describes an example in which variance is used as the variation of coordinates and classes, the present invention is not limited to this. For example, it is also possible to obtain the reliability of each coordinate using AI that outputs the reliability of input information, and calculate the loss based on the reliability.

[0050] In addition, in the above steps S108 and S208, whether to terminate learning is determined based on the loss after the weight update, but this is not limiting. For example, the termination of learning can also be determined based on the amount of weight update, the number of learning iterations, the accuracy of object detection, etc.

[0051] Note that the DNN used in the above explanation is an example of object detection AI, and is not limited to this.

[0052] DESCRIPTION OF SYMBOLS 100, 200, 300, 400... Object detection AI learning device, 101, 201... AI inference processing unit, 102, 202, 301... Variability calculation unit, 103, 205... Loss calculation unit, 104, 206... Weight update unit, 105, 204... Coordinate loss calculation unit, 203... Class loss calculation unit, 302... Threshold setting unit, 401... Notification unit

Claims

1. An object detection AI learning method realized by an object detection AI learning device having an AI inference processing unit, a coordinate loss calculation unit, a variability calculation unit, a loss calculation unit, and a weight update unit, wherein the AI ​​inference processing unit uses the object detection AI to extract, for each object included in an input image, multiple candidate areas where the object is expected to be located and infer coordinates for each of the multiple candidate areas; the coordinate loss calculation unit calculates, for each of the multiple candidate areas for each object, the deviation of the coordinates from the correct coordinates of the object as a coordinate loss; the variability calculation unit calculates, for each object, the variation of the coordinates for the multiple candidate areas as a coordinate variation loss; the loss calculation unit calculates a loss based on the coordinate loss and the coordinate variation loss; and the weight update unit updates the weights of the object detection AI to reduce the loss.

2. An object detection AI learning method according to claim 1, wherein the object detection AI learning device further has a class loss calculation unit, the AI ​​inference processing unit uses the object detection AI to infer, for each of the plurality of candidate areas, a class representing the type of object contained in the candidate area, the class loss calculation unit calculates, for each of the plurality of candidate areas for each of the objects, the deviation of the class from the correct class of the object contained in the candidate area as a class loss, and the loss calculation unit calculates the loss based on the coordinate loss, the coordinate variance loss, and the class loss.

3. An object detection AI learning method as described in claim 2, wherein the AI ​​inference processing unit uses the object detection AI to further infer a class probability representing the likelihood of the class for each of the plurality of candidate regions; the variability calculation unit further calculates, for each of the objects, the variability of the class probability for the plurality of candidate regions as a class variability loss; and the loss calculation unit calculates the loss based on the coordinate loss, the coordinate variability loss, the class loss, and the class variability loss.

4. An object detection AI learning method as described in claim 1, characterized in that the loss calculation unit does not include in the loss the coordinate loss of the candidate area indicating the object whose coordinate variation loss is larger than a coordinate variation threshold indicating the upper limit of the allowable range of the coordinate variation loss, and the coordinate variation loss itself.

5. An object detection AI learning method according to claim 4, wherein the coordinate variation threshold is set to be larger when the number of learning rounds exceeds a predetermined number, or when the coordinate variation loss falls below a predetermined value.

6. An object detection AI learning method according to claim 1, wherein the object detection AI learning device further has a notification unit, and the notification unit notifies the candidate area indicating the object whose coordinate variation loss is greater than a coordinate variation threshold indicating the upper limit of the allowable range of the coordinate variation loss.

7. An object detection AI learning device comprising: an AI inference processing unit that uses object detection AI to extract, for each object included in an input image, multiple candidate areas where the object is assumed to be located, and infer coordinates for each of the multiple candidate areas; a coordinate loss calculation unit that calculates, for each of the multiple candidate areas for each object, the deviation of the coordinates from the correct coordinates of the object as a coordinate loss; a variability calculation unit that calculates, for each object, the variability of the coordinates for the multiple candidate areas as a coordinate variability loss; a loss calculation unit that calculates a loss based on the coordinate loss and the coordinate variability loss; and a weight update unit that updates the weights of the object detection AI so as to reduce the loss.

Images