Object detection device, program, object detection system, and object detection method

Abstract

This object detection device (100) is characterized by comprising: an object estimation unit (102) that identifies, from image data showing an image of an object, a rectangle enclosing the object and estimates the type of the object in the identified rectangle, using an estimation model; and a training unit (103) that updates the estimation model by training the estimation model with estimation results from the object estimation unit, using correct-answer data that includes the image data and a correct-answer label including the type of the object and the distance from one intersection of grid lines set to the image shown by the image data to four lines including the four sides of the rectangle enclosing the object, thereby generating a learning model. The object detection device (100) is furthermore characterized in that the value indicating the distance is allowed to include a negative value.

Description

[Technical Field]

[0001] The present disclosure relates to an object detection device, a program, an object detection system, and an object detection method. [Background technology]

[0002] 2. Description of the Related Art Solutions such as safety monitoring or support for the monitors monitoring the cameras are being implemented using cameras installed in infrastructure environments such as highways.

[0003] One of the key features of this solution is the ability to detect objects from images captured by a camera, and the detected objects often become input for subsequent applications.

[0004] Deep learning methods are available for detecting objects, and one such method is the anchor-free method, which divides an image by grid lines, and uses any intersections of the grid lines where an object overlaps as a learning sample (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Zhi Tian, ​​Chunhua Shen, Hao Chen Tong He, “FCOS: Fully Convolutional One-Stage Object Detection”, arXiv:1904.01355v5, 20 August 2019 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the anchor-free method, even if an attempt is made to learn an object smaller than the area enclosed by the grid lines, there are no suitable grid points, and such a small object cannot be learned. Furthermore, when trying to detect and learn small objects, it is necessary to set many grid lines and make the area enclosed by the grid lines small, which increases the processing load.

[0007] Therefore, one or more aspects of the present disclosure aim to detect an object that is smaller than an area surrounded by grid lines and to perform learning using that object. [Means for solving the problem]

[0008] An object detection device according to one aspect of the present disclosure includes an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle; and an object estimation unit that uses the image data and correct answer data including a correct answer label including the type of the object and distances from one intersection of grid lines set in the image to four straight lines including four sides of the rectangle surrounding the object to estimate the type of the object within the identified rectangle. Types of objects The present invention is characterized in that it comprises a learning unit that learns the results of the estimation and updates the estimation model to generate a learning model, and that the value indicating the distance can include a negative value.

[0009] A program according to one aspect of the present disclosure includes a computer that includes an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle, and a computer that uses the image data and correct answer data including the type of the object and a correct answer label including distances from one intersection of grid lines set on the image to four straight lines including four sides of the rectangle surrounding the object to estimate the type of the object within the identified rectangle. Types of objects The device functions as a learning unit that learns the results of estimation and updates the estimation model to generate a learning model, and is characterized in that the value indicating the distance can include negative values.

[0010] The object detection system according to one aspect of the present disclosure includes an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle; and an object estimation unit that uses the image data and correct answer data including a correct answer label including the type of the object and distances from one intersection of grid lines set in the image to four straight lines including four sides of the rectangle surrounding the object to estimate the type of the object within the identified rectangle. Types of objects The present invention is characterized in that it comprises a learning unit that learns the results of the estimation and updates the estimation model to generate a learning model, and that the value indicating the distance can include a negative value.

[0011] An object detection method according to one aspect of the present disclosure includes: identifying a rectangle surrounding an object from image data showing an image of the object using an estimation model; estimating the type of the object within the identified rectangle; and using the image data and correct answer data including a correct answer label including the type of the object and distances from one intersection of grid lines set in the image to four straight lines including four sides of the rectangle surrounding the object, Types of objects This object detection method generates a learning model by learning the results of estimation and updating the estimation model, and is characterized in that the value indicating the distance can include negative values. [Effects of the Invention]

[0012] According to one or more aspects of the present disclosure, it is possible to detect an object that is smaller than an area surrounded by grid lines and perform learning using the object. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram schematically illustrating a configuration of an object detection device according to first and second embodiments. [Figure 2] FIG. 1 is a schematic diagram illustrating a first example of locating a bounding box. [Figure 3] FIG. 10 is a schematic diagram illustrating a second example of locating a bounding box. [Figure 4]10 is a graph showing a function used in the embodiment. [Figure 5] FIG. 1 is a block diagram illustrating a schematic configuration of a computer. [Figure 6] FIG. 11 is a block diagram schematically illustrating the configuration of an object detection device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Embodiment 1 FIG. 1 is a block diagram schematically illustrating the configuration of an object detection device 100 according to the first embodiment. The object detection device 100 includes a correct answer data storage unit 101, an object estimation unit 102, a learning unit 103, and a learning model storage unit 104.

[0015] The correct answer data storage unit 101 stores correct answer data including input data, which is image data showing an image of an object, and output data showing a correct answer label indicating the type of object included in the image data and the position of a bounding box surrounding the object.

[0016] Here, the bounding box is a rectangle that surrounds the object to be detected, and the position of the bounding box is indicated by the distance between the four lines that form the four sides of the bounding box and a reference intersection, which is the intersection of a grid line that is preset for the image.

[0017] Fig. 2 is a schematic diagram showing a first example of how to specify the position of a bounding box. Fig. 3 is a schematic diagram showing a second example of how to specify the position of a bounding box. Five vertical grid lines (solid lines) and five horizontal grid lines (solid lines) are shown in Fig. 2 and Fig. 3. As shown in Fig. 2, when the bounding box BB1 is larger than the area enclosed by the grid lines, the distances l, t, r, and b from the intersection point P1 of the grid lines to the straight lines L1 to L4 that include each side of the bounding box BB1 can all be expressed as positive values.

[0018] Here, it will be explained whether the distances l, t, r, and b are positive or negative values. Distance l: If the left side of the bounding box is to the left of the reference intersection point, distance l is positive; if the left side of the bounding box is to the right of the reference intersection point, distance l is negative. Distance t: If the top edge of the bounding box is above the reference intersection point, distance t is positive; if the top edge of the bounding box is below the reference intersection point, distance t is negative. Distance r: If the right side of the bounding box is to the right of the reference intersection point, the distance r is positive; if the right side of the bounding box is to the left of the reference intersection point, the distance r is negative. Distance b: If the bottom edge of the bounding box is below the reference intersection point, distance b is positive; if the bottom edge of the bounding box is above the reference intersection point, distance b is negative.

[0019] However, as shown in FIG. 3, if the bounding box BB2 is smaller than the area surrounded by the grid lines, in other words, if the object to be detected is included in the area surrounded by the grid lines, of the distances l, t, r, and b from the intersection point P2 of the grid lines to each of the straight lines L5 to L8 containing each side of the bounding box BB2, the distances indicated by the symbols l and t can be expressed as positive values, but the distances indicated by the symbols r and b can be expressed as negative values. In this embodiment, the distance value can be expressed as a negative value.

[0020] Returning to Figure 1, the object estimation unit 102 uses an estimation model to identify a rectangle surrounding an object included in the image represented by the image data, which is input data included in the correct data stored in the correct data storage unit 101, and estimates the type of object within the identified rectangle.

[0021] 2 and 3, the position and size of the bounding box are determined from the intersections of the grid lines. The intersections used to determine the position and size of the bounding box are referred to as reference intersections. In the first embodiment, as shown in FIG. 2, when an object and an intersection are in contact with each other, the contacting intersection is used as the reference intersection.

[0022] As shown in Figure 3, if the object does not contact the intersection and the object is included in the area formed by the grid, one of the four intersections in the area including the object is used as the reference intersection. Any intersection may be used as the reference intersection. Here, for example, the intersection obtained by selecting the distance with the smallest absolute value of the distances l and r and the smallest absolute value of the distances t and b is used as the reference intersection.

[0023] Here, the object estimation unit 102 performs estimation using an anchor-free deep learning method. Deep learning methods use nonlinear transformations using activation functions, so the required values ​​are usually calculated by substituting the output value x from the estimation model into exp(x).

[0024] However, as mentioned above, in this embodiment, there are cases where the distance value must be a negative value, so it is not possible to use negative values ​​for the distances l, t, r, and b by simply using exp(x) as in the usual case. For this reason, in this embodiment, the object estimation unit 102 calculates the distances l, t, r, and b using the function shown in the following equation (1) for the output value x from the estimation model.

number

[0025] As shown in Figure 4, the function expressed by equation (1) is a function that takes a smaller value as the output value x from the estimation model becomes smaller, and also outputs negative values ​​within the range of values ​​that the output value can take. Here, the value α is greater than 0. The larger the value α, the less likely it is to become a negative value, so an appropriate value may be specified in advance by experiment or the like. The value β is also a value greater than 0, and an appropriate value may be specified in advance through experiments or the like.

[0026] As described above, in this embodiment, negative values ​​can be used in the correct label included in the correct data for the distance between the four lines comprising the four sides of the bounding box and the reference intersection, and for the distance value estimated by the object estimation unit 102. Specifically, the object estimation unit 102 can estimate a negative value as the distance value by inputting the estimated value of the estimation model to a function that can output a negative value. This allows the object estimation unit 102 to set grid lines on an image represented by image data as input data and detect objects that are smaller than the area surrounded by the grid lines.

[0027] Returning to FIG. 1 , learning unit 103 uses the correct answer data stored in correct answer data storage unit 101 to learn the results of estimation by object estimation unit 102 and update the estimation model used in object estimation unit 102, thereby generating a learning model. The learning model is a trained estimation model. In other words, learning unit 103 uses correct answer data including image data and a correct answer label including the type of object and the distance from one intersection of grid lines set in the image represented by the image data to four straight lines including the four sides of a rectangle surrounding the object, thereby generating a learning model.

[0028] The learning model storage unit 104 stores the learning model generated by the learning unit 103.

[0029] The object detection device 100 described above can be realized by a computer 10 as shown in FIG. The computer 10 includes an auxiliary storage device 11 such as an HDD (Hard Disk Drive) and an SSD (Solid State Drive), a main storage device 12 such as a volatile or non-volatile memory, and a processor 13 such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

[0030] For example, the correct answer data storage unit 101 and the learning model storage unit 104 can be realized by the auxiliary storage device 11 or the main storage device 12. The object estimation unit 102 and the learning unit 103 can be realized by the processor 13 reading a program stored in the auxiliary storage device 11 into the main storage device 12 and executing the program.

[0031] Such a program may be provided via a network or may be provided by being recorded on a recording medium, i.e., may be provided as a program product.

[0032] Furthermore, although not shown, the object detection system may be configured by implementing the correct answer data storage unit 101, the object estimation unit 102, the learning unit 103, and the learning model storage unit 104 using multiple computers connected to a network such as the Internet.

[0033] Embodiment 2 As shown in FIG. 1, object detection device 200 according to the second embodiment includes supervised data storage unit 101, object estimation unit 202, learning unit 103, and learning model storage unit 104. The correct data storage unit 101, the learning unit 103, and the learning model storage unit 104 of the object detection device 200 according to the second embodiment are the same as the correct data storage unit 101, the learning unit 103, and the learning model storage unit 104 of the object detection device 100 according to the first embodiment.

[0034] The object estimation unit 202 uses the correct answer data stored in the correct answer data storage unit 101 to identify a bounding box in an image represented by image data as input data, and estimates the type of object contained in the bounding box.

[0035] Here, object estimation section 202 in the second embodiment performs substantially the same processing as object estimation section 102 in the first embodiment, but the method of determining the reference intersection is different from that in the first embodiment. In the second embodiment, as shown in FIG. 2, when an object and an intersection are in contact with each other, the contacting intersection is used as the reference intersection, as in the first embodiment.

[0036] On the other hand, as shown in Figure 3, if the object does not contact the intersection and the object is included in the area formed by the grid, one of the four intersections in the area containing the object is used as the reference intersection.

[0037] In the second embodiment, object estimation unit 202 estimates an object using each of the four intersections of an area including the object. Then, object estimation unit 202 identifies four bounding boxes from the four intersections, and if the overlap rate of the four bounding boxes is equal to or greater than a predetermined threshold, uses the intersection with the highest likelihood of estimating the type of object as the reference intersection. Note that if the overlap rate of the four bounding boxes is less than a predetermined threshold, the object estimation unit 202 may select all of the intersections as the reference intersections, since there is a possibility that multiple small objects are in the same region.

[0038] As described above, in the second embodiment, the object estimation unit 202 estimates the object from the four intersections of the region including the object, and merges the estimation results to identify the reference intersection.

[0039] Embodiment 3 FIG. 6 is a block diagram schematically showing the configuration of object detection device 300 according to the third embodiment. The object detection device 300 includes a supervised data storage unit 101, an object estimation unit 102, a learning unit 103, a learning model storage unit 104, a learning data storage unit 305, a learning data acquisition unit 306, and a supervised data generation unit 307.

[0040] The correct data storage unit 101, the object estimation unit 102, the learning unit 103, and the learning model storage unit 104 of the object detection device 300 in embodiment 3 are similar to the correct data storage unit 101, the object estimation unit 102, the learning unit 103, and the learning model storage unit 104 of the object detection device 100 in embodiment 1. However, the supervised data storage unit 101 in the third embodiment stores supervised data generated by the supervised data generating unit 307, as will be described later.

[0041] The training data storage unit 305 stores training data. Here, the training data indicates image data that is input data for the correct answer data, the type of object included in the image indicated by the image data, and the position and size in the image data of a bounding box that surrounds the object.

[0042] The learning data acquisition unit 306 acquires learning data from the learning data storage unit 305 and provides the learning data to the correct answer data generation unit 307 .

[0043] The correct answer data generating unit 307 generates correct answer data using the learning data provided by the learning data acquiring unit 306 . For example, the object estimation unit 102 sets a grid area in an image represented by image data as learning data, and if an object is included in an area surrounded by grid lines, it identifies a reference intersection from any one of the four intersections in that area, identifies the distance from that reference intersection to the four straight lines containing the four sides of the bounding box, and estimates the type of object.

[0044] Then, the correct data generation unit 307 uses the image data as input data and generates correct data by adding output data indicating the correct label indicating the distance and object type identified by the object estimation unit 102 to the input data.

[0045] The type of object and its bounding box can be identified by displaying an image of the image data on a display unit (not shown) and having the operator input the type of object and the bounding box surrounding the object via the input unit.

[0046] The object detection device 300 according to the third embodiment can also be realized by a computer 10 as shown in FIG.

[0047] For example, the learning data storage unit 305 can also be realized by the auxiliary storage device 11 or the main storage device 12. The learning data acquisition unit 306 and the correct answer data generation unit 307 can also be realized by the processor 13 reading a program stored in the auxiliary storage device 11 into the main storage device 12 and executing the program.

[0048] As described above, according to the third embodiment, correct answer data can be easily generated from training data, which is existing image data. In the third embodiment, the processing by the object estimation unit 202 shown in the second embodiment may also be performed.

[0049] In the above-described first to third embodiments, it is assumed that each of object detection devices 100 to 300 is configured by one computer, but each of the first to third embodiments is not limited to this example. For example, the functions of each unit included in object detection devices 100 to 300 can be executed by multiple computers connected to a network such as the Internet. In other words, embodiments 1 to 3 can also be configured as an object detection system including multiple computers instead of object detection devices 100 to 300. [Explanation of symbols]

[0050] 100, 200, 300 Object detection device, 101 Correct answer data storage unit, 102, 202 Object estimation unit, 103 Learning unit, 104 Learning model storage unit, 305 Learning data storage unit, 306 Learning data acquisition unit, 307 Correct answer data generation unit.

Claims

1. an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle; a learning unit that uses the image data and correct answer data including the type of the object and a correct answer label including distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object to learn the result of estimating the type of the object and update the estimation model, thereby generating a learned model; The value indicating the distance can include a negative value. An object detection device characterized by:

2. The object estimation unit is configured to set the grid lines on the image and identify a rectangle smaller than an area surrounded by the grid lines.

2. The object detection device according to claim 1,

3. When the object is included in the region, the object estimation unit specifies the distance from any one of the four intersections of the region to four straight lines including the four sides of the rectangle, and estimates the type of the object.

3. The object detection device according to claim 2, wherein:

4. the object estimation unit, when the object is included in the region, estimates the object from each of four intersections of the region, thereby specifying, at each of the four intersections, four rectangles surrounding the object estimated from each of the four intersections; specifies, for each of the four specified rectangles, a distance from one of the intersections used in specifying one of the four specified rectangles to four straight lines including four sides of the one rectangle; and, when an overlap rate at which all of the specified four rectangles overlap is equal to or greater than a predetermined threshold, uses, as a result of estimating the type of the object, a distance from one of the four intersections which has the highest likelihood of estimating the type of the object; 3. The object detection device according to claim 2, wherein:

5. The object estimation unit estimates a negative value as the value indicating the distance by inputting the estimated value of the estimation model to a function that can output a negative value. The object detection device according to claim 1 ,

6. Computer, an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data representing an image of the object and estimates the type of the object within the identified rectangle; and a learning unit that uses the image data and correct answer data including the type of the object and a correct answer label including distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object to learn the result of estimating the type of the object and update the estimation model, thereby generating a learned model; The value indicating the distance can include a negative value. A program characterized by.

7. an object estimation unit that uses an estimation model to identify a rectangle surrounding an object from image data showing an image of the object and estimates the type of the object within the identified rectangle; a learning unit that uses the image data and correct answer data including the type of the object and a correct answer label including distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object to learn the result of estimating the type of the object and update the estimation model, thereby generating a learned model; The value indicating the distance can include a negative value. An object detection system comprising:

8. Using an estimation model, a rectangle surrounding the object is identified from image data showing an image of the object, and the type of the object within the identified rectangle is estimated; an object detection method for generating a learned model by learning a result of estimating the type of the object using correct answer data including the image data and a correct answer label including the type of the object and distances from one intersection of grid lines set in the image to four straight lines including four sides of a rectangle surrounding the object, and updating the estimation model, The value indicating the distance can include a negative value. An object detection method characterized by:

Images