Learning device, learning method, program, and object detection device

Abstract

To provide a learning device, a learning method, a program, and an object detection device for suitably generating a learned model for discriminating an object existing on a road.SOLUTION: A learning device 100 includes: a captured image acquisition unit 110 for acquiring a captured image captured on a road; a CG image addition unit 140 for adding a computer graphic image of an object existing on the road relative to a real image based on the captured image; and a learning processing unit 150 for learning a parameter of a learned model so as to output a type of the object when an image is input with the type of an added computer graphic image being teacher data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a learning device, a learning method, a program, and an object detection device. [Background technology]

[0002] A technology has been disclosed in which training data is created for 3D graphics drawn by a rendering unit, and a deep learning recognition unit is trained using the training data (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 171005 Summary of the Invention [Problem to be solved by the invention]

[0004] With conventional technology, it was sometimes not possible to generate an appropriate trained model for identifying objects on the road from images captured by a camera mounted on a moving vehicle.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a learning device, a learning method, a program, and an object detection device that can preferably generate a trained model for discriminating objects present on the road. [Means for solving the problem]

[0006] The learning device, learning method, program, and object detection device according to the present invention employ the following configuration. (1): A learning device according to one embodiment of the present invention includes an image acquisition unit that acquires an image of a road, a CG image addition unit that adds a computer graphic image of an object present on the road to an actual image based on the image, and a learning processing unit that uses the type of the added computer graphic image as training data and learns parameters of a trained model so as to output the type of the object when an image is input.

[0007] (2): In the above aspect (1), a sunlight direction estimation unit is further provided which estimates the sunlight direction based on an actual image based on the captured image, and the CG image addition unit adds a shadow based on the sunlight direction to the computer graphics image of the object.

[0008] (3): In the above aspect (1) or (2), the captured image is captured by a camera mounted on a moving body, and the device further includes a pitching amount estimation unit that estimates the pitching amount of the moving body for each imaging time point based on the captured image, and a first correction unit that performs a first correction on the captured image to cancel out the pitching amount, thereby generating the actual image.

[0009] (4): In the aspect of (3) above, a second correction unit is further provided that performs a second correction on the real-life image to which the computer graphic image has been added to undo the first correction to generate a training image, and the learning processing unit learns parameters of the trained model using the training image as training data.

[0010] (5): In the above aspect (1) or (2), the captured image is an image captured by a camera mounted on a moving body, the real-life image is the captured image, and the system further includes a pitching amount estimation unit that estimates the pitching amount of the moving body for each imaging time point based on the captured image, and the CG image adding unit adds the computer graphic image at a position in the real-life image according to the pitching amount.

[0011] (6): In any of the above aspects (1) to (5), the captured image is an image captured by a camera mounted on a moving body, and the device further includes a movement amount acquisition unit that acquires the amount of movement of the moving body, and the CG image addition unit determines the position and size of the computer graphic image based on the amount of movement of the moving body.

[0012] (7): A learning method according to another aspect of the present invention is a learning method executed using a computer, comprising: acquiring an image of a road; adding a computer graphic image of an object on the road to an actual image based on the acquired image; and learning parameters of a trained model using the type of the added computer graphic image as training data so as to output the type of the object when an image is input.

[0013] (8): A program according to another aspect of the present invention is a program executed by a computer, which causes the computer to acquire an image of a road, add a computer graphic image of an object on the road to an actual image based on the captured image, and learn parameters of a trained model using the type of the added computer graphic image as training data so as to output the type of the object when an image is input.

[0014] (9): An object detection device according to another aspect of the present invention is an object detection device mounted on a moving body, which inputs an image captured by a camera mounted on the moving body of at least a road in the direction of travel of the moving body into the trained model trained by a learning device according to any one of aspects (1) to (6) above, and determines whether an object on the road shown in the image is an object with which the moving body should avoid contact.

[0015] (10): A learning device according to another aspect of the present invention includes an image acquisition unit that acquires an image of a road, a CG image addition unit that adds a computer graphic image of an object on the road to an actual image based on the image, and a learning processing unit that uses the position of the added computer graphic image as training data and learns parameters of a trained model so as to output the position of the object when an image is input.

[0016] (11): A learning method according to another aspect of the present invention is a learning method executed by a computer, comprising: acquiring an image of a road; adding a computer graphic image of an object on the road to an actual image based on the captured image; and learning parameters of a trained model using the position of the added computer graphic image as training data so as to output the position of the object when an image is input.

[0017] (12): A program according to another aspect of the present invention is a program executed by a computer, which causes the computer to acquire an image of a road, add a computer graphic image of an object on the road to an actual image based on the captured image, and learn parameters of a trained model using the position of the added computer graphic image as training data so as to output the position of the object when an image is input. Effect of the Invention

[0018] According to the above aspects (1) to (8) and (10) to (12), a trained model for discriminating objects present on a road can be suitably generated. According to the above aspect (9), objects on the road can be appropriately identified using a trained model that has been suitably trained. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example of the configuration of a learning device 100. [Diagram 2] 2 is a diagram illustrating an example of a processing flow by each unit of the learning device 100. FIG. [Diagram 3] FIG. 13 is a diagram showing how the position of the road surface in the corrected image IM2 changes depending on the gradient. [Figure 4] 10 is a diagram for explaining the process of the pitching amount estimating unit 120. FIG. [Diagram 5] FIG. 11 is a diagram for explaining the content of area correction. [Figure 6] FIG. 13 is a diagram for explaining the contents of up / down shift processing. [Figure 7] 11 is a diagram for explaining the content of a process for calculating a deviation amount d. FIG. [Figure 8] 11A and 11B are diagrams for explaining a process of estimating a pitching amount. [Figure 9] 11A and 11B are diagrams for explaining a process of estimating a pitching amount. [Figure 10] 11 is a diagram for explaining the processing of a lateral movement amount estimation unit 131. FIG. [Figure 11] FIG. 13 is a diagram for explaining the contents of left / right shift and image cutting out processing. [Figure 12] 11 is a diagram for explaining the processing of a speed estimation unit 132. FIG. [Figure 13] FIG. 13 is a diagram for explaining the contents of the enlargement / reduction and image cutout processing. [Figure 14] 1 is a diagram illustrating an example of a configuration and a usage environment of an object detection device 200. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, with reference to the drawings, embodiments of a learning device, a learning method, a program, and an object detection device of the present invention will be described. The learning device is a device that generates a trained model for object discrimination used by an object detection device mounted on a moving body. A moving body is, for example, a four-wheeled vehicle, a two-wheeled vehicle, micromobility, a robot, or other object that moves by itself, or a portable device such as a smartphone that is placed on a moving body that moves by itself or is carried by a person. In the following description, the moving body is a four-wheeled vehicle, and the moving body is referred to as a "vehicle".

[0021] [Learning device] 1 is a diagram showing an example of the configuration of a learning device 100. The object detection device 200 includes, for example, a captured image acquisition unit 110, a pitching amount estimation unit 120, a first correction unit 122, a second correction unit 124, a solar radiation direction estimation unit 126, a movement amount acquisition unit 130, a CG (computer graphic) image addition unit 140, a learning processing unit 150, and a storage unit 170. The movement amount acquisition unit 130 includes, for example, a lateral movement amount estimation unit 131 and a speed estimation unit 132. The CG image addition unit 140 includes, for example, a CG image generation unit 141, a CG image movement amount / magnification ratio calculation unit 142, a road surface position estimation unit 143, a CG image position identification unit 144, and an image synthesis unit 145. Each unit other than the storage unit 170 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transient storage medium) such as a hard disk drive (HDD) or a flash memory, or may be stored in a removable storage medium (a non-transient storage medium) such as a DVD or a CD-ROM, and may be installed by mounting the storage medium in a drive device. The storage unit 170 stores data and information such as an actual driving image 171 and machine learning model definition information 172.

[0022] The captured image acquisition unit 110 acquires captured images, for example, of at least the road in the traveling direction of the vehicle, captured by a camera mounted on the vehicle while the vehicle is moving (including a case where the vehicle is temporarily stopped). The captured images are moving images in which images are successively displayed in time series. The captured images may be captured by a fixed-point observation camera or may be captured by a smartphone camera. For example, an actual traveling image 171 stored in the storage unit 170 is an example of this captured image. The captured image acquisition unit 110 reads out the actual traveling image 171 from the storage unit 170 and provides it to other functional units (for example, expands it in a shared area of ​​a RAM (Random Access Memory)). The actual traveling image 171 is captured as the vehicle moves by a vehicle equipped with a camera that captures the traveling direction. The actual driving image 171 may be provided to the learning device 100 by an in-vehicle communication device via a network such as a WAN (Wide Area Network) or a LAN (Local Area Network), or it may be transported to the learning device 100 by various portable storage devices and stored in the memory unit 170.

[0023] The subsequent processing of the functional units will be described with reference to Fig. 2. Fig. 2 is a diagram illustrating the flow of processing by each unit of the learning device 100. Here, the actual driving image 171 is referred to as actual driving image IM1.

[0024] The pitching amount estimation unit 120 estimates the pitching amount of the vehicle in the actual driving image IM1 at each imaging time point. For example, the pitching amount estimation unit 120 compares the actual driving image IM1 at imaging time (hereinafter simply referred to as time) k with the actual driving image IM1 at time k-1, and estimates the pitching amount of the vehicle between time k and time k-1 as the pitching amount at time k. The pitching amount estimation unit 120 performs this processing for each time. The pitching amount is the amount of rotation around an axis whose axis is the left-right direction of the vehicle. If a pitching amount exists, vertical shaking occurs in the actual driving image IM1 between time k and time k-1, so it is necessary to perform processing to correct the pitching amount for processing by the CG image addition unit 140 described later. Details of the processing by the pitching amount estimation unit 120 will be described later.

[0025] The first correction unit 122 performs a first correction on the actual driving image IM1 to cancel out the up and down fluctuation of the image according to the pitching amount estimated by the pitching amount estimation unit 120. For example, the first correction unit 122 determines the amount of correction for the first correction using a table or map that specifies how much each pixel on the image fluctuates up and down in response to the pitching amount. The image on which the first correction by the first correction unit 122 has been performed is referred to as a corrected image IM2. The corrected image IM2 is an example of a "real image based on a captured image."

[0026] The second corrector 124 performs a second correction to undo the first correction on a CG-added image IM3 in which a CG image has been added to the corrected image IM2 by the CG image adder 140, thereby generating a learning image IM4.

[0027] The sunlight direction estimation unit 126 estimates the sunlight direction in the scenery captured in the actual driving image IM1 based on the corrected image IM2. For example, the sunlight direction estimation unit 126 recognizes, in the corrected image IM2, a shadow portion included in the corrected image IM2 and a three-dimensional object that is estimated to have created the shadow, and converts the sunlight direction based on the recognized positional relationship between the two from the image plane to real space, thereby deriving the sunlight direction in real space.

[0028] The movement amount acquisition unit 130 estimates the movement amount of the vehicle. Specifically, the movement amount acquisition unit 130 estimates the following two movement amounts.

[0029] The lateral movement amount estimation unit 131 of the movement amount acquisition unit 130, for example, compares the actual traveling image IM1 at time k with the actual traveling image IM1 at time k-1, and estimates the lateral movement amount of the vehicle between time k and time k-1 (the movement amount in the direction perpendicular to the vehicle center axis, or the movement amount in the road width direction). Details of the processing of the lateral movement amount estimation unit 131 will be described later.

[0030] The speed estimation unit 132 of the movement amount acquisition unit 130, for example, compares the actual driving image IM1 at time k with the actual driving image IM1 at time k-1, and estimates the speed of the vehicle at time k. The process of the speed estimation unit 132 will be described in detail later.

[0031] A CG image adding unit 140 adds CG images of objects existing on the road to the corrected image IM2.

[0032] The CG image generating unit 141 of the CG image adding unit 140 reads out template images of objects that the vehicle should avoid contacting, such as some objects (e.g., tires, cardboard boxes, bicycles, steel beams, etc.) that may exist on the road as fallen objects or abandoned objects, from the storage unit 170, converts or re-renders the template images to a size according to a magnification ratio described later, and further adds a shadow based on the sunlight direction estimated by the sunlight direction estimating unit 126 to generate a CG image. The CG image generating unit 141 also generates CG images of objects that the vehicle does not need to avoid, such as road markings, manholes, and road materials. The CG image generating unit 141 generates CG images of road markings that were not collected as captured images, faded road markings, and fallen objects that frequently appear, in various angles and states (states where crushed, cracked, dirty, etc. have occurred). This makes it possible to expand the range covered by the learning process compared to the case where a trained model is generated using only captured images, and to reduce the probability of recognition as an unlearned object, thereby suppressing the occurrence of erroneous avoidance by the vehicle.

[0033] The CG image movement amount / magnification ratio calculation unit 142 of the CG image adding unit 140 determines the position and size of the CG image based on the movement amount of the vehicle estimated by the movement amount acquisition unit 130. Specifically, the CG image movement amount / magnification ratio calculation unit 142 adds the displacement amount on the image based on the distance traveled by the vehicle from time k-1 to time k and the lateral movement amount (obtained by projecting the displacement amount on the imaginary plane seen from above onto the image plane) to the position of the CG image at time k-1 to calculate the position of the CG image at time k. The CG image movement amount / magnification ratio calculation unit 142 outputs the calculated position of the CG image at time k to the CG image position identification unit 144. In addition, the CG image movement amount / magnification ratio calculation unit 142 calculates the magnification ratio of the CG image at time k-1 based on the distance traveled by the vehicle from time k-1 to time k. The CG image movement amount / magnification ratio calculation unit 142 outputs the magnification ratio to the CG image generating unit 141. The magnification factor will be greater than 1 if the vehicle is moving forward and less than 1 (i.e., reduced) if the vehicle is moving backward.

[0034] The road surface position estimation unit 143 of the CG image addition unit 140 estimates the position of the road surface in the real space in the traveling direction of the moving object 1 based on the corrected image IM2, and constructs a correspondence relationship between each position on the road surface and each position in the corrected image IM2. FIG. 3 is a diagram showing how the position of the road surface in the corrected image IM2 changes depending on the gradient. As shown in the figure, if the road surface in the real space in the traveling direction of the moving object 1 is an uphill or downhill (especially when the position where the vehicle is located is flat and a gradient occurs in the traveling direction side), or curves to the left or right, the position of the road surface in the corrected image IM2 changes, and therefore the position on the image to which the CG image should be added changes. The road surface position estimation unit 143 performs the above processing to recognize the position on the image. The processing of the road surface position estimation unit 143 may construct a correspondence relationship between each position on the image in an ideal state in which the road surface is horizontal and not curved, and each position on the corrected image IM2 showing the actual state.

[0035] The CG image position identification unit 144 of the CG image addition unit 140 identifies the position of the CG image on the image based on the calculated position (corresponding to the position in the ideal state described above) obtained from the CG image movement amount / magnification ratio calculation unit 142 and the position of the road surface estimated by the road surface position estimation unit 143.

[0036] The image synthesis section 145 of the CG image adding section 140 superimposes (adds) the CG image generated by the CG image generating section 141 onto the corrected image IM2 at the position of the CG image specified by the CG image position specifying section 144. This generates a CG-added image IM3. The CG image adding section 140 may erase and overwrite the image of the road surface at the position occupied by the CG image, or may add pixel values ​​of the CG image while leaving the image of the road surface.

[0037] As described above, the second corrector 124 performs the second correction on the CG-added image IM3 to undo the first correction, thereby generating a learning image IM4.

[0038] The learning processing unit 150 learns the parameters of the machine learning model defined by the machine learning model definition information 172. The machine learning model definition information 172 is information that defines the number of nodes, connection relationships, and the like of the machine learning model. The machine learning model is, for example, a DNN (Deep Neural Network) as shown in FIG. 2. The learning processing unit 150 learns the parameters of the machine learning model by using the learning image IM4 as learning data and the type of the added CG image as teacher data, using a method such as backpropagation, so as to output the type of an object (an object with which the vehicle should avoid contact) when an image is input. The machine learning model that has finished learning the parameters is used in the vehicle-mounted device as a learned model. Therefore, it can be said that the learning processing unit 150 learns the parameters of the learned model.

[0039] Alternatively (or in addition), the learning processing unit 150 may learn the parameters of a machine learning model using the learning image IM4 as learning data and the position of the added CG image as teacher data, using a technique such as backpropagation, so as to output the position of an object (an object with which the vehicle must avoid contact) when an image is input.

[0040] In the above description, the CG image adding unit 140 adds a CG image of an object existing on the road to the corrected image IM2, which is obtained by performing a first correction on the actual driving image IM1 to cancel out the vertical fluctuation of the image according to the pitching amount estimated by the pitching amount estimating unit 120, and performs a second correction on the CG-added image IM3 to which the CG image has been added, which undoes the first correction, to generate the learning image IM4. Alternatively, the CG image adding unit 140 may add the CG image to the actual driving image IM1 at a position where the position of the CG image specified by the CG image position specifying unit 144 has been corrected according to the pitching amount estimated by the pitching amount estimating unit 120. In this case, the second correction is not necessary.

[0041] The above-described processing makes it possible to appropriately generate a trained model for discriminating objects present on the road.

[0042] The processing of pitching amount estimation unit 120, lateral movement amount estimation unit 131, and speed estimation unit 132 will be described in more detail below. FIG. 4 is a diagram for explaining the processing of pitching amount estimation unit 120. Pitching amount estimation unit 120 grayscales actual traveling images IM1 at time k and time k-1, respectively. Hereinafter, the grayscaled images will be referred to as grayscale images. Pitching amount estimation unit 120 performs image cropping processing on the grayscale image at time k, deleting unnecessary portions and cropping the image.

[0043] The pitching amount estimation unit 120 performs area correction based on the vehicle speed for the grayscale image at time k-1. The vehicle speed may be a processing result of the speed estimation unit 132 described later, or may be obtained from a speed sensor together with imaging by a camera in the vehicle (provided together with the captured image). FIG. 5 is a diagram for explaining the content of area correction. The pitching amount estimation unit 120 calculates the distance traveled by the vehicle from time k-1 to time k from the vehicle speed, and enlarges the grayscale image at time k-1 with an enlargement ratio based on the traveled distance (i.e., an enlargement ratio based on the speed). This is because, for the portion captured of the same location, if the vehicle is moving forward, the distance at time k should be closer than that at time k-1, and the image at time k is more enlarged. The pitching amount estimation unit 120 performs area correction so that the same object appears at the same size in the grayscale images at time k and time k-1, and allows accurate comparison between the images. The pitching amount estimation unit 120 trims the top, bottom, left and right edges of the enlarged image to return it to its original size.

[0044] Returning to FIG. 4, the pitching amount estimation unit 120 performs up-down shift processing on the grayscale image at time k-1 after the area correction to generate N images. FIG. 6 is a diagram for explaining the content of the up-down shift processing. The pitching amount estimation unit 120 generates a plurality of (here, N) comparison target images with different amounts of shift in the upward and downward directions for the grayscale image at time k-1 after the area correction. In the figure, the dashed line indicates the grayscale image before the shift is generated. Then, N difference images are generated in which the pixel difference between the pixel difference between the area cut out (cut-out area) at a determined position from the grayscale image at time k and the cut-out area of ​​the comparison target image is used as the pixel (difference image generation processing in FIG. 3). Returning to FIG. 4, the pitching amount estimation unit 120 performs binarization processing by comparing the pixel value of each pixel of the difference image with a threshold value, and assigning a pixel value of 1 if the pixel value is equal to or greater than the threshold value, and assigning a pixel value of zero if the pixel value is less than the threshold value. Then, the pitching amount estimation unit 120 selects the difference target image with the smallest sum of pixel values ​​(the largest number of pixels with zero), and calculates the amount of deviation that led to the generation of the difference target image. Fig. 7 is a diagram for explaining the content of the process for calculating the amount of deviation d. In the illustrated example, the difference target image from the comparison target image with the amount of deviation d of minus 2, that is, shifted two pixels downward, is selected as the image with the smallest sum of pixel values.

[0045] Next, the pitching amount estimation unit 120 estimates the pitching amount based on the shift amount d. FIG. 8 and FIG. 9 are diagrams for explaining the process of estimating the pitching amount. The relationship between the shift amount d and the pitching amount θ is expressed by the following formula. In the formula, l is the distance in real space to the vanishing point VP in the image (for example, about 100 to 200 [m]), and h is a value obtained by multiplying the height from the lower end of the image to the vanishing point VP by two. The pitching amount estimation unit 120 obtains the pitching amount θ based on this formula. Note that h changes depending on the size of the image, but since the pitching amount θ is substantially returned to the shift amount on the image and used, no problem occurs in this respect.

[0046] tan{(α / 2)+θ}={d+(h / 2)} / l

[0047] 10 is a diagram for explaining the processing of the lateral movement amount estimation unit 131. The lateral movement amount estimation unit 131 grayscales the actual traveling images IM1 at time k and time k-1 (the grayscale processing may be common to the pitching amount estimation unit 120. The same applies to the speed estimation unit 132). The lateral movement amount estimation unit 131 performs an image cropping process for deleting unnecessary portions from the grayscale image at time k-1 and cropping it out.

[0048] The lateral movement amount estimation unit 131 performs area correction based on the vehicle speed for the grayscale image at time k. This is the same as the pitching amount estimation unit 120, and the direction of enlargement / reduction is the same as that of the pitching amount estimation unit 120.

[0049] The lateral movement amount estimation unit 131 performs left-right shifting and image cutting processing on the grayscale image at time k after the area correction, and generates M images. FIG. 11 is a diagram for explaining the contents of the left-right shifting and image cutting processing. The lateral movement amount estimation unit 131 generates a plurality of (here, M) comparison target images with different amounts of shifting in the left and right directions for the grayscale image at time k after the area correction. In the figure, the dashed line indicates the grayscale image before the shifting is caused. Then, M difference images are generated in which the pixel difference between the pixel difference between the cut-out area (cut-out area) of the comparison target image and the area cut-out area of ​​the comparison target image is used as the pixel (difference image generation processing in FIG. 10). Returning to FIG. 10, the lateral movement amount estimation unit 131 performs binarization processing by comparing the pixel value of each pixel of the difference image with a threshold value, and assigning a pixel value of 1 if the pixel value is equal to or greater than the threshold value, and assigning a pixel value of zero if the pixel value is less than the threshold value. Then, the lateral movement amount estimation unit 131 selects the difference target image having the smallest sum of pixel values ​​(the largest number of pixels that are zero), and calculates the lateral movement amount ΔY based on the shift amount that led to the generation of that difference target image.

[0050] 12 is a diagram for explaining the processing of the speed estimation unit 132. The speed estimation unit 132 converts the actual traveling image IM1 at time k and time k-1 into grayscale images. The speed estimation unit 132 performs an image cropping process for cropping the grayscale image at time k-1 by deleting unnecessary portions.

[0051] The speed estimation unit 132 performs enlargement / reduction and image cutout processing on the grayscale image at time k to generate R images. FIG. 13 is a diagram for explaining the contents of the enlargement / reduction and image cutout processing. The speed estimation unit 132 generates a plurality of (here, R) comparison target images with different enlargement ratios (reduction ratios) in the enlargement direction and the reduction direction for the grayscale image at time k. In the figure, the dashed line indicates the grayscale image before enlargement or reduction. Then, R difference images are generated in which the pixel difference between a region cut out at a determined position from the grayscale image at time k-1 (cutout region) and the cutout region of the comparison target image is used as a pixel (difference image generation processing in FIG. 10). Returning to FIG. 12, the speed estimation unit 132 performs binarization processing by comparing the pixel value of each pixel of the difference image with a threshold value, and assigning a pixel value of 1 if the pixel value is equal to or greater than the threshold value, and assigning a pixel value of zero if the pixel value is less than the threshold value. Then, the speed estimation unit 132 selects the difference target image having the smallest sum of pixel values ​​(the largest number of pixels that are zero), and calculates the speed V based on the enlargement ratio (reduction ratio) at which the difference target image was generated.

[0052] According to the embodiment of the learning device described above, the device is equipped with an image acquisition unit 110 that acquires an image of at least the road in the direction of travel of the moving body, the image being captured by a camera mounted on the moving body while the moving body is moving, a CG image addition unit 140 that adds a computer graphic image of an object present on the road to an actual image IM2 based on the captured image, and a learning processing unit 150 that learns parameters of the learned model so as to output the position of the object when an image is input, using the position of the added computer graphic image as training data, thereby making it possible to suitably generate a learned model for discriminating objects present on the road.

[0053] [Object detection device] Hereinafter, an embodiment of an object detection device using a trained model generated by the learning device 100 will be described. The object detection device is mounted on, for example, a moving body. The moving body is, for example, a four-wheeled vehicle, a two-wheeled vehicle, micromobility, a robot, etc. In the following description, the moving body is a four-wheeled vehicle, and the moving body is referred to as a "vehicle".

[0054] 14 is a diagram showing an example of the configuration and usage environment of the object detection device 200. The object detection device 200 communicates with the camera 10, a driving control device 300, a notification device 310, and the like.

[0055] The camera 10 is attached to the rear surface of the windshield of the vehicle, captures an image of at least the road in the traveling direction of the vehicle, and outputs the captured image to the object detection device 200. Note that a sensor fusion device or the like may be interposed between the camera 10 and the object detection device 200, but a description thereof will be omitted.

[0056] The driving control device 300 is, for example, an automatic driving control device that drives the vehicle autonomously, a driving support device that performs vehicle distance control, automatic braking control, lane change control, etc. The notification device 310 is a speaker, vibrator, light-emitting device, display device, etc. for outputting information to the vehicle occupants.

[0057] The object detection device 200 includes, for example, an acquisition unit 210, a low-resolution processing unit 220, a high-resolution processing unit 230, and a storage unit 250. The storage unit 250 stores a trained model 252 trained by the learning device 100. The acquisition unit 210, the low-resolution processing unit 220, and the high-resolution processing unit 230 are each realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, or GPU, or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device having a non-transient storage medium), or may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium), and may be installed by mounting the storage medium in a drive device.

[0058] The acquisition unit 210 acquires a captured image from the camera 10. The acquisition unit 210 stores (data of) the acquired captured image in a working memory such as a RAM.

[0059] The low-resolution processing unit 220 performs thinning processing on the captured image to generate a low-resolution image with lower image quality than the captured image. The low-resolution image is, for example, an image with fewer pixels than the captured image. The low-resolution processing unit 220, for example, extracts an area including a characteristic part from the low-resolution image and outputs it to the high-resolution processing unit 230 as a focus area. There are no particular restrictions on a specific example of the process for extracting this area, and any method may be adopted.

[0060] The high resolution processing unit 230 cuts out a portion of the captured image that corresponds to the location of interest, and inputs the image of that portion to the trained model 252. The trained model 252 determines whether the image captured at the location of interest is a road marking, a fallen object (learned in the learning device 100 using a CG image), or is unknown (an unlearned object).

[0061] The discrimination result of the high resolution processing unit 230 is output to the driving control device 300 and / or the notification device 310. The driving control device 300 performs automatic brake control, automatic steering control, etc., to avoid contact between the vehicle and the object (actually an area on the image) discriminated as a "falling object" and an unknown object (an object that has not been learned). The notification device 310 outputs an alarm in various ways when the TTC (Time To Collision) between the object (same as above) discriminated as a "falling object" and the vehicle falls below a threshold value.

[0062] According to the embodiment of the object detection device described above, it is possible to appropriately distinguish objects on the road using the trained model 252 that has been suitably trained.

[0063] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Acquire an image of the road, adding a computer graphic image of an object existing on the road to the actual image based on the captured image; Using the type of the added computer graphic image as training data, parameters of a trained model are trained so as to output the type of the object when an image is input. The learning device is configured as follows.

[0064] The above-described embodiment can also be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Acquire an image of the road, adding a computer graphic image of an object existing on the road to the actual image based on the captured image; Using the position of the added computer graphic image as training data, parameters of a trained model are trained so as to output the position of the object when an image is input. The learning device is configured as follows.

[0065] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0066] 10. Camera 100 Learning Device 110 Image acquisition unit 120 Pitching amount estimation unit 122 First Correction Section 124 Second Correction Section 126 Solar radiation direction estimation unit 130 Movement amount estimation section 140 CG Image Addition Section 150 Learning processing unit 170 Storage section 200 Object detection device 252 trained models

Claims

1. an image acquisition unit for acquiring an image of a road; a CG image adding unit that adds a computer graphic image of an object existing on the road to the actual image based on the captured image; a learning processing unit that learns parameters of a trained model so as to output a type of the object when an image is input, using the type of the added computer graphic image as training data; and A learning device comprising: The captured image is an image captured by a camera mounted on a moving object, a pitching amount estimation unit that estimates a pitching amount of the moving object for each imaging time point based on the captured images; a first correction unit that performs a first correction on the captured image to cancel out the pitching amount and generates the actual image; The learning device further comprises:

2. A solar radiation direction estimation unit that estimates a solar radiation direction based on an actual image based on the captured image, the CG image adding unit adds a shadow based on the sunlight direction to the computer graphic image of the object; The learning device according to claim 1.

3. a second correction unit that performs a second correction on the real-life image to which the computer graphic image has been added to undo the first correction to generate a learning image; The learning processing unit learns parameters of the learned model using the learning images as learning data.

3. The learning device according to claim 1 or 2.

4. An image acquisition unit for acquiring an image of a road surface; a CG image adding unit that adds a computer graphic image of an object existing on the road to the actual image based on the captured image; a learning processing unit that learns parameters of a trained model so as to output a type of the object when an image is input, using the type of the added computer graphic image as training data; and A learning device comprising: The captured image is an image captured by a camera mounted on a moving object, the actual image is the captured image, A pitching amount estimation unit is further provided for estimating a pitching amount of the moving body for each imaging time point based on the captured image, the CG image adding unit adds the computer graphic image to the real-life image at a position corresponding to the pitching amount. Learning device.

5. Further comprising a movement amount acquisition unit that acquires a movement amount of the moving body, the CG image adding unit determines a position and a size of the computer graphic image based on the amount of movement of the moving object; A learning device according to any one of claims 1 to 4.

6. 1. A computer-implemented method of learning, comprising: Obtaining an image of a road surface; adding a computer graphic image of an object existing on the road to the actual image based on the captured image; learning parameters of a trained model so as to output the type of the object when an image is input, using the type of the added computer graphic image as training data; Equipped with The captured image is an image captured by a camera mounted on a moving object, estimating a pitching amount of the moving object for each imaging time point based on the captured images; performing a first correction on the captured image to cancel out the pitching amount, thereby generating the actual image; A learning method that further includes the above.

7. A program executed by a computer, the program comprising: Obtaining an image of a road surface; adding a computer graphic image of an object existing on the road to the actual image based on the captured image; learning parameters of a trained model so as to output the type of the object when an image is input, using the type of the added computer graphic image as training data; Run the command, The captured image is an image captured by a camera mounted on a moving object, estimating a pitching amount of the moving object for each imaging time point based on the captured images; performing a first correction on the captured image to cancel out the pitching amount, thereby generating the actual image; A program that further executes the above.

8. An object detection device mounted on a moving body, A captured image of at least a road in a traveling direction of the moving body by a camera mounted on the moving body is input to the trained model trained by the learning device according to any one of claims 1 to 5, thereby determining whether an object on the road shown in the captured image is an object with which the moving body should avoid contact. Object detection device.

9. an image acquisition unit for acquiring an image of a road; a CG image adding unit that adds a computer graphic image of an object existing on the road to the actual image based on the captured image; a learning processing unit that learns parameters of a trained model so as to output the position of the object when an image is input, using the position of the added computer graphic image as training data; and A learning device comprising: The captured image is an image captured by a camera mounted on a moving object, a pitching amount estimation unit that estimates a pitching amount of the moving object for each imaging time point based on the captured images; a first correction unit that performs a first correction on the captured image to cancel out the pitching amount and generates the actual image; The learning device further comprises:

10. An image acquisition unit for acquiring an image of a road surface; a CG image adding unit that adds a computer graphic image of an object existing on the road to the actual image based on the captured image; a learning processing unit that learns parameters of a trained model so as to output the position of the object when an image is input, using the position of the added computer graphic image as training data; and A learning device comprising: The captured image is an image captured by a camera mounted on a moving object, the actual image is the captured image, A pitching amount estimation unit is further provided for estimating a pitching amount of the moving body for each imaging time point based on the captured image, the CG image adding unit adds the computer graphic image to the real-life image at a position corresponding to the pitching amount. Learning device.

11. 1. A computer-implemented method of learning, comprising: Obtaining an image of a road surface; adding a computer graphic image of an object existing on the road to the actual image based on the captured image; learning parameters of a trained model so as to output the position of the object when an image is input, using the position of the added computer graphic image as training data; Equipped with The captured image is an image captured by a camera mounted on a moving object, estimating a pitching amount of the moving object for each imaging time point based on the captured images; performing a first correction on the captured image to cancel out the pitching amount, thereby generating the actual image; A learning method that further includes:

12. A program executed by a computer, the program comprising: Obtaining an image of a road surface; adding a computer graphic image of an object existing on the road to the actual image based on the captured image; learning parameters of a trained model so as to output the position of the object when an image is input, using the position of the added computer graphic image as training data; Run the command, The captured image is an image captured by a camera mounted on a moving object, estimating a pitching amount of the moving object for each imaging time point based on the captured images; performing a first correction on the captured image to cancel out the pitching amount, thereby generating the actual image; A program that further executes the above.

Images