Object detection device, program, object detection system, and object detection method

Abstract

This object detection device (100) is characterized by comprising: an object estimation unit (102) that identifies, from image data showing an image of an object, a rectangle enclosing the object and estimates the type of the object in the identified rectangle, using an estimation model; and a training unit (103) that updates the estimation model by training the estimation model with estimation results from the object estimation unit, using correct-answer data that includes the image data and a correct-answer label including the type of the object and the distance from one intersection of grid lines set to the image shown by the image data to four lines including the four sides of the rectangle enclosing the object, thereby generating a learning model. The object detection device (100) is furthermore characterized in that the value indicating the distance is allowed to include a negative value.

Description

Object detection device, program, object detection system, and object detection method

[0001] The present disclosure relates to an object detection device, a program, an object detection system, and an object detection method.

[0002] 2. Description of the Related Art Solutions such as safety monitoring or support for the monitors monitoring the cameras are being implemented using cameras installed in infrastructure environments such as highways.

[0003] One of the key features of this solution is the ability to detect objects from images captured by a camera, and the detected objects often become input for subsequent applications.

[0004] Deep learning methods are available for detecting objects, and one such method is the anchor-free method, which divides an image by grid lines and uses the object as a learning sample when there is an intersection of the grid lines and an object overlaps with the image (see, for example, Non-Patent Document 1).

[0005] Zhi Tian, ​​Chunhua Shen, Hao Chen Tong He, “FCOS: Fully Convolutional One-Stage Object Detection”, arXiv:1904.01355v5, 20 August 2019

[0006] However, with the anchor-free method, even if an object smaller than the area enclosed by the grid lines is to be learned, there are no suitable grid points, and such small objects cannot be learned. Furthermore, when detecting and learning a small object, it is necessary to set many grid lines and make the area enclosed by the grid lines small, which increases the processing load.

[0007] Therefore, one or more aspects of the present disclosure aim to detect an object that is smaller than an area surrounded by grid lines and to perform learning using that object.

[0008] An object detection device according to one aspect of the present disclosure includes an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimate the type of the object within the identified rectangle, and a learning unit that uses the image data and ground truth data including a ground truth label including the type of the object and a distance from an intersection of one of the grid lines set in the image to four straight lines including the four sides of the rectangle surrounding the object to learn the results of the estimation and update the estimation model to generate a learning model, wherein the value indicating the distance can include a negative value.

[0009] A program according to one aspect of the present disclosure causes a computer to function as an object estimation unit that identifies a rectangle surrounding an object from image data showing an image of the object using an estimation model and estimates the type of the object within the identified rectangle, and a learning unit that learns the results of the estimation using ground truth data that includes the image data and a ground truth label that includes the type of the object and the distance from one intersection of grid lines set in the image to four straight lines that include the four sides of the rectangle surrounding the object, and updates the estimation model to generate a learning model, wherein the value indicating the distance can include a negative value.

[0010] An object detection system according to one aspect of the present disclosure includes an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimate the type of the object within the identified rectangle; and a learning unit that uses the image data and ground truth data including a ground truth label containing the type of the object and a distance from an intersection of one of the grid lines set in the image to four straight lines containing the four sides of the rectangle surrounding the object to learn the results of the estimation and update the estimation model to generate a learning model, wherein the value indicating the distance can include a negative value.

[0011] An object detection method according to one aspect of the present disclosure uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object, estimate the type of the object within the identified rectangle, learns the results of the estimation using correct answer data including the image data and a correct answer label including the type of the object and the distance from an intersection of one of the grid lines set in the image to four straight lines containing the four sides of the rectangle surrounding the object, and updates the estimation model to generate a learning model, characterized in that the value indicating the distance can include a negative value.

[0012] According to one or more aspects of the present disclosure, it is possible to detect an object that is smaller than an area surrounded by grid lines and perform learning using the object.

[0013] FIG. 1 is a block diagram schematically showing the configuration of an object detection device according to embodiments 1 and 2. FIG. 2 is a schematic diagram showing a first example of specifying the position of a bounding box. FIG. 3 is a schematic diagram showing a second example of specifying the position of a bounding box. FIG. 4 is a graph showing functions used in the embodiments. FIG. 5 is a block diagram schematically showing the configuration of a computer. FIG. 6 is a block diagram schematically showing the configuration of an object detection device according to embodiment 3.

[0014] 1 is a block diagram showing a schematic configuration of an object detection device 100 according to embodiment 1. The object detection device 100 includes a correct answer data storage unit 101, an object estimation unit 102, a learning unit 103, and a learning model storage unit 104.

[0015] The correct answer data storage unit 101 stores correct answer data including input data, which is image data showing an image of an object, and output data showing a correct answer label indicating the type of object included in the image data and the position of a bounding box surrounding the object.

[0016] Here, the bounding box is a rectangle that surrounds the object to be detected, and the position of the bounding box is indicated by the distance between the four lines that form the four sides of the bounding box and a reference intersection, which is the intersection of a grid line that is preset for the image.

[0017] FIG. 2 is a schematic diagram showing a first example of how to specify the position of a bounding box. FIG. 3 is a schematic diagram showing a second example of how to specify the position of a bounding box. Five vertical grid lines (solid lines) and five horizontal grid lines (solid lines) are shown in FIGS. 2 and 3. As shown in FIG. 2, when the bounding box BB1 is larger than the area enclosed by the grid lines, the distances l, t, r, and b from the intersection point P1 of the grid lines to the straight lines L1 to L4 containing the sides of the bounding box BB1 can all be expressed as positive values.

[0018] Here, it will be explained whether the distances l, t, r, and b are positive or negative. Distance l: If the left side of the bounding box is to the left of the reference intersection point, distance l is positive; if the left side of the bounding box is to the right of the reference intersection point, distance l is negative. Distance t: If the top side of the bounding box is above the reference intersection point, distance t is positive; if the top side of the bounding box is below the reference intersection point, distance t is negative. Distance r: If the right side of the bounding box is to the right of the reference intersection point, distance r is positive; if the right side of the bounding box is to the left of the reference intersection point, distance r is negative. Distance b: If the bottom side of the bounding box is below the reference intersection point, distance b is positive; if the bottom side of the bounding box is above the reference intersection point, distance b is negative.

[0019] 3, if the bounding box BB2 is smaller than the area enclosed by the grid lines, in other words, if the object to be detected is included in the area enclosed by the grid lines, then of the distances l, t, r, and b from the intersection P2 of the grid lines to the lines L5 to L8 that include each side of the bounding box BB2, the distances indicated by the symbols l and t can be expressed as positive values, but the distances indicated by the symbols r and b can be expressed as negative values. In this embodiment, the distance values ​​can be expressed as negative values.

[0020] Returning to Figure 1, the object estimation unit 102 uses an estimation model to identify a rectangle surrounding an object included in the image represented by the image data, which is input data included in the correct data stored in the correct data storage unit 101, and estimates the type of object within the identified rectangle.

[0021] 2 and 3, the position and size of a bounding box are determined from the intersections of grid lines. The intersections used to determine the position and size of the bounding box are referred to as reference intersections. In the first embodiment, as shown in FIG. 2, when an object is in contact with an intersection, the contacting intersection is used as the reference intersection.

[0022] As shown in Figure 3, if the object does not contact an intersection and the object is included in the area formed by the grid, one of the four intersections in the area including the object is used as the reference intersection. Any intersection may be used as the reference intersection. In this example, the intersection obtained by selecting the distance with the smallest absolute value of the distances l and r and the smallest absolute value of the distances t and b is used as the reference intersection.

[0023] Here, the object estimation unit 102 performs estimation using an anchor-free deep learning method. In the deep learning method, a nonlinear transformation using an activation function is performed. Therefore, a required value is usually calculated by substituting the output value x from the estimation model into exp(x).

[0024] However, as described above, in this embodiment, there are cases where the distance value must be a negative value. Therefore, it is not possible to use negative values ​​for the distances l, t, r, and b simply by using exp(x), as is usually the case. For this reason, in this embodiment, the object estimation unit 102 calculates the distances l, t, r, and b using the function shown in the following equation (1) for the output value x from the estimation model.

[0025] As shown in FIG. 4, the function expressed by equation (1) is a function that takes a smaller value as the output value x from the estimation model becomes smaller, and also outputs negative values ​​within the range of values ​​that the output value can take. Here, the value α is greater than 0. The larger the value α, the less likely it is to become a negative value, so an appropriate value can be specified in advance through experiments, etc. The value β is also greater than 0, and an appropriate value can be specified in advance through experiments, etc.

[0026] As described above, in this embodiment, negative values ​​can be used in the correct label included in the correct data for the distance between the four lines comprising the four sides of the bounding box and the reference intersection, and for the distance value estimated by the object estimation unit 102. Specifically, the object estimation unit 102 can estimate a negative value as the distance value by inputting the estimated value of the estimation model to a function that can output a negative value. This allows the object estimation unit 102 to set grid lines on an image represented by image data as input data, and detect objects that are smaller than the area surrounded by the grid lines.

[0027] Returning to FIG. 1 , the learning unit 103 uses the correct answer data stored in the correct answer data storage unit 101 to learn the results of the estimation made by the object estimation unit 102 and updates the estimation model used in the object estimation unit 102, thereby generating a learning model. The learning model is a trained estimation model. In other words, the learning unit 103 uses the correct answer data including image data and a correct answer label including the type of object and the distance from one intersection of grid lines set in the image represented by the image data to four straight lines including the four sides of a rectangle surrounding the object, thereby generating a learning model.

[0028] The learning model storage unit 104 stores the learning model generated by the learning unit 103 .

[0029] The object detection device 100 described above can be realized by a computer 10 as shown in Fig. 5. The computer 10 includes an auxiliary storage device 11 such as a hard disk drive (HDD) or a solid state drive (SSD), a main storage device 12 such as a volatile or non-volatile memory, and a processor 13 such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0030] For example, the correct answer data storage unit 101 and the learning model storage unit 104 can be realized by the auxiliary storage device 11 or the main storage device 12. Furthermore, the object estimation unit 102 and the learning unit 103 can be realized by the processor 13 reading a program stored in the auxiliary storage device 11 into the main storage device 12 and executing the program.

[0031] Such a program may be provided via a network or may be provided by being recorded on a recording medium, i.e., may be provided as a program product.

[0032] Furthermore, although not shown, the object detection system may be configured by implementing the correct answer data storage unit 101, the object estimation unit 102, the learning unit 103, and the learning model storage unit 104 using multiple computers connected to a network such as the Internet.

[0033] 1, an object detection device 200 according to the second embodiment includes a supervised data storage unit 101, an object estimation unit 202, a learning unit 103, and a learning model storage unit 104. The supervised data storage unit 101, the learning unit 103, and the learning model storage unit 104 of the object detection device 200 according to the second embodiment are similar to the supervised data storage unit 101, the learning unit 103, and the learning model storage unit 104 of the object detection device 100 according to the first embodiment.

[0034] The object estimation unit 202 uses the correct answer data stored in the correct answer data storage unit 101 to identify a bounding box in an image represented by image data as input data, and estimates the type of object contained in the bounding box.

[0035] Here, object estimation unit 202 in embodiment 2 performs substantially the same processing as object estimation unit 102 in embodiment 1, but the method of determining the reference intersection is different from that in embodiment 1. In embodiment 2, as shown in Fig. 2, when an object and an intersection are in contact with each other, the contacting intersection is used as the reference intersection, as in embodiment 1.

[0036] On the other hand, as shown in FIG. 3, if the object does not contact the intersection and the object is included in the area formed by the grid, one of the four intersections in the area containing the object is used as the reference intersection.

[0037] In the second embodiment, the object estimation unit 202 estimates the object using each of the four intersections of the region containing the object. Then, the object estimation unit 202 identifies four bounding boxes from the four intersections, and if the overlap rate of the four bounding boxes is equal to or greater than a predetermined threshold, the object estimation unit 202 uses the intersection with the highest likelihood of estimating the type of object as the reference intersection. Note that if the overlap rate of the four bounding boxes is less than the predetermined threshold, the object estimation unit 202 may select all of the intersections as the reference intersections. This is because there is a possibility that multiple small objects are present in the same region.

[0038] As described above, in the second embodiment, the object estimation unit 202 estimates the object from the four intersections of the area including the object, and merges the estimation results to identify the reference intersection.

[0039] 6 is a block diagram showing a schematic configuration of an object detection device 300 according to embodiment 3. The object detection device 300 includes a supervised data storage unit 101, an object estimation unit 102, a learning unit 103, a learning model storage unit 104, a learning data storage unit 305, a learning data acquisition unit 306, and a supervised data generation unit 307.

[0040] The supervised data storage unit 101, the object estimation unit 102, the learning unit 103, and the learning model storage unit 104 of the object detection device 300 in the third embodiment are similar to the supervised data storage unit 101, the object estimation unit 102, the learning unit 103, and the learning model storage unit 104 of the object detection device 100 in the first embodiment. However, the supervised data storage unit 101 in the third embodiment stores the supervised data generated by the supervised data generation unit 307, as will be described later.

[0041] The training data storage unit 305 stores training data. Here, the training data indicates image data that is input data for the correct answer data, the type of object included in the image indicated by the image data, and the position and size in the image data of a bounding box that surrounds the object.

[0042] The learning data acquisition unit 306 acquires learning data from the learning data storage unit 305 and provides the learning data to the correct answer data generation unit 307 .

[0043] The supervised data generation unit 307 generates supervised data using the learning data provided by the learning data acquisition unit 306. For example, the object estimation unit 102 sets a grid area in an image represented by image data serving as learning data, and when an object is included in an area surrounded by grid lines, identifies a reference intersection from any one of the four intersections in that area, identifies the distance from the reference intersection to four straight lines including the four sides of the bounding box, and estimates the type of the object.

[0044] Then, the correct data generation unit 307 uses the image data as input data and generates correct data by adding output data indicating the correct label indicating the distance and object type identified by the object estimation unit 102 to the input data.

[0045] The type of object and its bounding box can be identified by displaying an image of the image data on a display unit (not shown) and having the operator input the type of object and the bounding box surrounding the object via the input unit.

[0046] The object detection device 300 according to the third embodiment can also be realized by a computer 10 as shown in FIG.

[0047] For example, the learning data storage unit 305 can also be realized by the auxiliary storage device 11 or the main storage device 12. The learning data acquisition unit 306 and the correct answer data generation unit 307 can also be realized by the processor 13 reading a program stored in the auxiliary storage device 11 into the main storage device 12 and executing the program.

[0048] As described above, according to the third embodiment, correct answer data can be easily generated from training data, which is existing image data. Note that the processing by the object estimation unit 202 shown in the second embodiment may also be performed in the third embodiment.

[0049] In the above-described first to third embodiments, it is assumed that each of the object detection devices 100 to 300 is configured by a single computer, but each of the first to third embodiments is not limited to this example. For example, the functions of each unit included in the object detection devices 100 to 300 can be executed by multiple computers connected to a network such as the Internet. In other words, the first to third embodiments can also be configured as an object detection system including multiple computers instead of the object detection devices 100 to 300.

[0050] 100, 200, 300 Object detection device, 101 Correct answer data storage unit, 102, 202 Object estimation unit, 103 Learning unit, 104 Learning model storage unit, 305 Learning data storage unit, 306 Learning data acquisition unit, 307 Correct answer data generation unit.

Claims

1. an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle; a learning unit that uses the image data and correct answer data including a type of the object and a correct answer label including distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object to learn a result of estimating the type of the object and update the estimation model to generate a learned model, The value indicating the distance can include a negative value. An object detection device comprising:

2. The object estimation unit is capable of setting the grid lines on the image and identifying a rectangle smaller than an area enclosed by the grid lines. The object detection device according to claim 1 .

3. When the object is included in the region, the object estimation unit specifies a distance from any one of four intersections of the region to four straight lines including four sides of the rectangle, and estimates the type of the object.

3. The object detection device according to claim 2,

4. the object estimation unit, when the object is included in the area, specifies four rectangles surrounding the object from four intersections of the area, specifies a distance from one intersection used in specifying one of the specified rectangles to four straight lines including four sides of one of the specified four rectangles, and when an overlap rate of the specified four rectangles is equal to or greater than a predetermined threshold, uses, as a result of estimation of the type of the object, a distance from one of the four intersections that has the highest likelihood of estimating the type of the object; 3. The object detection device according to claim 2,

5. The object estimation unit estimates a negative value as the value indicating the distance by inputting the estimated value of the estimation model to a function that can output a negative value. The object detection device according to claim 1 ,

6. Computer, an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object, and estimates the type of the object within the identified rectangle; a learning unit that learns a result of estimating the type of the object using correct answer data including the image data and a correct answer label including a distance from an intersection of one of the grid lines set in the image to four straight lines including the four sides of a rectangle surrounding the object, and updates the estimation model to generate a learning model; The value indicating the distance can include a negative value. A program characterized by.

7. an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle; a learning unit that uses the image data and correct answer data including a type of the object and a correct answer label including distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object to learn a result of estimating the type of the object and update the estimation model to generate a learned model, The value indicating the distance can include a negative value. An object detection system comprising:

8. Using an estimation model, a rectangle surrounding the object is identified from image data showing an image of the object, and a type of the object within the identified rectangle is estimated; The object detection method includes: learning a result of estimating the type of the object using correct answer data including the image data and a correct answer label including a type of the object and distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object; and updating the estimation model to generate a learned model, The value indicating the distance can include a negative value. The object detection method is characterized by: