Image generation method and image generation device

Abstract

In the present invention: in a first image (41) acquired from an imaging device (11) mounted on a vehicle, a first region (A1), in which at least one of a stationary object and a road surface is displayed, and a second region (A2), which is other than the first region (A1) and includes at least a moving object, are identified; if map information (13), which includes stationary object information relating to the stationary object, includes registration of the stationary object being present at a second position on a map corresponding to a first position in the first region (A1), then correspondence display (81a, 82a, 83a, 86a, 86b, 87a, 91a) corresponding only to the stationary object present at the second position are projected onto the first position; and a second image (44, 45) is generated with the correspondence display (81a, 82a, 83a, 86a, 86b, 87a, 91a) projected therein.

Description

Image generation method and image generation device

[0001] The present invention relates to an image generation method and an image generation device.

[0002] A method for labeling images for machine learning is known (Patent Document 1), in which images of roadside objects such as traffic signs are acquired from an on-board camera, the relative position of the roadside object with respect to the vehicle is determined based on the position and orientation of the vehicle in a defined coordinate system and the position information of the roadside object, and the roadside object on the image is labeled taking into account the determined relative position.

[0003] Special Publication No. 2022-514891

[0004] In the above-mentioned conventional technology, if a part of a roadside object is obstructed by another object when viewed from an on-board camera, the roadside object is labeled as an image of the roadside object that includes a different object. Therefore, when an image labeled using the above-mentioned conventional technology is used for machine learning, there is a problem that the roadside object cannot be accurately detected.

[0005] The problem to be solved by the present invention is to provide an image generation method and an image generation device that can generate images for machine learning that suppress erroneous detection of stationary objects.

[0006] The present invention solves the above problem by distinguishing between a first area in an image acquired from an imaging device mounted on a vehicle, in which at least one of a stationary object and the road surface is displayed, and a second area other than the first area, which includes at least a moving object, and if map information including stationary object information regarding stationary objects registers that a stationary object exists at a second position on the map corresponding to a first position in the first area, projecting a corresponding display corresponding only to the stationary object present at the second position onto the first position.

[0007] According to the present invention, it is possible to generate images for machine learning that suppress false detection of stationary objects.

[0008] 1 is a block diagram showing an example of an embodiment of an image generation system according to the present invention; FIG. 2 is an example of a first image acquired from the imaging device of FIG. 1; FIG. 3 is an example of an image in which a corresponding display of a stationary object is projected onto the first image of FIG. 2; FIG. 4 is an example of a first region and a second region in the first image of FIG. 2; FIG. 5 is an example of a second image generated from the first image of FIG. 2; FIG. 6 is another example of a second image generated from the first image of FIG. 2; FIG. 7 is a flowchart showing an example of a processing procedure in the image generation system of FIG. 1; FIG. 8 is a flowchart showing an example of a processing procedure in the driving assistance device of FIG. 1; FIG. 9 is a flowchart showing an example of a processing procedure in the map generation device of FIG. 1;

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [Configuration of Image Generation System] Fig. 1 is a block diagram showing an example of an embodiment of an image generation system 10 according to the present invention. The image generation system 10 is a system that generates training data for supervised learning, consisting of an image (hereinafter also referred to as a first image) captured by an imaging device mounted on a vehicle and an image (hereinafter also referred to as a second image) on which a corresponding display corresponding to a stationary object (hereinafter also referred to as a stationary object) is projected. In the training data, the first image is a sample problem, and the second image is a correct answer to the sample problem. The vehicle is not particularly limited, and the first image may be acquired from the imaging devices of multiple vehicles.

[0011] The target object is an object that exists around a vehicle equipped with an imaging device (hereinafter also referred to as the equipped vehicle), and includes lane boundaries, center lines, road markings, medians, guardrails, curbs, road signs, traffic lights, crosswalks, etc. The target object also includes obstacles that may affect the driving of the equipped vehicle, such as other vehicles other than the equipped vehicle, motorcycles, bicycles, pedestrians, etc.

[0012] Stationary objects are not particularly limited as long as they are objects that exist on or around the road and whose position does not change, and include objects that are fixed, attached, or installed on the road or its surroundings. Stationary objects also include objects related to the detection of the driving range of the equipped vehicle. Such objects include road markings, road signs, traffic lights, structures (e.g., guardrails and blocks) installed on sidewalks adjacent to roads that indicate the boundary between the road and the sidewalk, and landmark buildings. Note that road markings are traffic signs installed on the road surface, and road signs are signboards installed on the shoulders of roads or above the road.

[0013] A corresponding display corresponding to a stationary object is a display indicating the presence of a stationary object. In the second image, a display such as a solid line, a dashed line, a dot, or a rectangle is projected at the location where the stationary object is located or at a location corresponding to the stationary object based on map information (specifically, location information regarding the location of the stationary object). The shape and color of the corresponding display can be appropriately set within a range that allows appropriate machine learning to be performed using training data including the second image on which the corresponding display is projected. Note that instead of projecting the corresponding display onto the second image, the corresponding display may be displayed on the second image, or the corresponding display may be placed on the second image.

[0014] As shown in FIG. 1 , the image generation system 10 includes an imaging device 11, a distance measuring device 12, map information 13, a self-location detection device 14, and an image generation device 15. The distance measuring device 12 and the self-location detection device 14 are mounted on a vehicle, and the map information 13 is stored in an on-board storage medium of the vehicle or in a storage medium outside the vehicle. The image generation device 15 may be mounted on the vehicle or may be provided outside the vehicle (e.g., in a remote location away from the vehicle). These devices are connected to each other by a Controller Area Network (CAN) or other on-board LAN or via a network so that they can exchange information with each other. The network refers to a telecommunications network such as the Internet, and the communication format is not particularly limited.

[0015] The imaging device 11 is mounted on the vehicle and captures images of objects around the vehicle, and is a camera equipped with an imaging element such as a CCD, an infrared camera, or the like. The ranging device 12 is a device that acquires the relative distance and relative speed between the ranging device 12 and an object, and includes a millimeter-wave radar, a LiDAR (Light Detection and Ranging) unit, or the like. In order to reduce blind spots when detecting an object, multiple imaging devices 11 and ranging devices 12 are installed at the front, right and left sides, rear, and other locations of the vehicle.

[0016] The map information 13 includes information on nodes corresponding to points where the vehicle's traveling direction changes (intersections, branching points, etc.) and information on links corresponding to road sections connecting the nodes. Node information includes location information (e.g., latitude and longitude) and information on entering and exiting intersections, while link information includes road width and curvature radius, roadside structures, road traffic regulations, etc. Furthermore, the map information 13 may be high-precision map information that includes road information, facility information, and their attribute information, and that allows tracking of the movement trajectory for each lane.

[0017] The self-position detection device 14 is a positioning system that detects the current position of the vehicle and calculates the current position of the vehicle from, for example, radio waves received from a satellite for the GPS (Global Positioning System). The image generation device 15 acquires detection results from the imaging device 11, the distance measurement device 12, and the self-position detection device 14 at predetermined time intervals (for example, every 0.1 to 1 millisecond), and also acquires map information 13 from a storage medium (not shown) as needed.

[0018] The image generation device 15 is a device that controls the devices that make up the image generation system 10 to cooperate with each other to generate a second image. The image generation device 15 is, for example, a computer, and includes a CPU (Central Processing Unit) that serves as a processor, a ROM (Read Only Memory) that stores a program, and a RAM (Random Access Memory) that functions as an accessible storage device. The CPU of the image generation device 15 is an operating circuit that executes the program stored in the ROM to generate a second image. The program includes a discrimination unit 21 and a generation unit 22 that are functional blocks for generating the second image, and a learning unit 23 that performs machine learning. Figure 1 illustrates these functional blocks in an extracted form for convenience.

[0019] [Functions of the Image Generating Device] Each functional block shown in Fig. 1 will be described below with reference to Figs. 2 to 6. Fig. 2 is an example of an image (first image) acquired from the imaging device 11, which is an image of the area ahead of the mounted vehicle stopped in front of an intersection captured by a camera (imaging device 11) attached to the top of the windshield. A portion of the body of the stopped mounted vehicle 50 is displayed at the bottom of the first image 41 shown in Fig. 2, another vehicle 51 stopped in front of the mounted vehicle 50 is displayed in the center of the first image 41, and another vehicle 52 stopped to the right and in front of the mounted vehicle 50 is displayed on the right side of the first image 41.

[0020] 2, a lane boundary line 61 and a lane boundary line 62 are displayed on the left and right sides of the loaded vehicle 50, and a stop line 63 is displayed between the other vehicles 51 and 52. Also, a part of an arrow sign 64 indicating the direction in which the vehicle can proceed is displayed on the road surface between the loaded vehicle 50 and the other vehicles 51, a crosswalk 65 is displayed above the stop line 63, and a stop line 66 is displayed above the crosswalk 65. Furthermore, a stop line 67 is displayed above the stop line 66, located beyond the intersection (i.e., located on the opposite side of the intersection from the loaded vehicle 50).

[0021] A curb 71 on the shoulder of the road is displayed on the left side of the lane boundary line 61, and a curb 72 on the shoulder of the intersecting lane that intersects the lane in which the loaded vehicle 50 is traveling at the intersection is displayed on the left side of the first image 41. A traffic light 73 is displayed above the other vehicle 51, and a traffic light 74 is displayed to the left of the traffic light 73 (above the curb 72). A sign 75, which is a road sign indicating that a turn is prohibited, is displayed to the left of the traffic light 74. A building 76, which serves as a landmark when traveling around the intersection, is displayed on the upper right side of the first image 41.

[0022] 3 is an example of an image in which the corresponding indicators of stationary objects used to detect the driving range of the equipped vehicle 50 are directly projected onto the first image 41 shown in FIG. When projecting the corresponding indicators of stationary objects onto the first image 41, the position information of the stationary objects expressed in the coordinate system (e.g., the global coordinate system) of the map information 13 is converted into the coordinate system (e.g., the screen coordinate system) of the first image 41, and the corresponding indicators are projected based on the converted position information of the stationary objects. On the image 42 shown in FIG. 3, corresponding indicators 81, 82, 84, and 85 corresponding to lane boundary lines, corresponding indicators 83, 86, and 87 corresponding to stop lines, and corresponding indicators 91, 92, and 93 corresponding to curbs are projected. Note that in FIGS. 2 to 6, lane boundary lines and stop lines are represented by solid lines, and curbs are represented by dashed lines.

[0023] 3 projects corresponding indicators of stationary objects that are not displayed in the first image 41 (i.e., not captured by the imaging device 11). For example, for lane boundary lines 61 and 62, corresponding indicators 81 and 82 are projected even in the portions that are blocked by the vehicle 50. Similarly, corresponding indicators 84 and 85 of lane boundary lines that are blocked by the other vehicle 52 and not captured are projected superimposed on the other vehicle 52. Furthermore, for stop lines 63, 66, and 67, corresponding indicators 83, 86, and 87 are projected even in the portions that are blocked by the other vehicles 51 and 52. Furthermore, a corresponding indicator 93 of a curb that is not captured by the imaging device 11 is projected.

[0024] When generating a model (e.g., a neural network) for detecting stationary objects from a first image 41, if the model is trained using training data in which the first image 41 shown in Fig. 2 is used as an example and the image 42 shown in Fig. 3 is used as a correct answer, a model that outputs an image in which corresponding indications of stationary objects not displayed in the input image are projected will be generated, and stationary objects will not be detected accurately. Therefore, the image generation device 15 of this embodiment generates training data for generating a model that can accurately detect stationary objects from the first image 41 by detecting and deleting corresponding indications of stationary objects not displayed in the first image 41 from the image 42 using the functions of the functional blocks shown in Fig. 1.

[0025] The discrimination unit 21 acquires a first image 41 from the imaging device 11 and distinguishes between a first region in the acquired first image 41, in which at least one of a stationary object and a road surface is displayed, and a second region other than the first region. The second region may include at least a moving object, such as a vehicle or a pedestrian. The discrimination unit 21 includes a discrimination model for discriminating between the first region and the second region in the first image 41, and discriminates between the first region and the second region using the discrimination model. The discrimination model is a trained model that has been trained in advance to discriminate between the first region and the second region in the first image 41, and the type of model is not particularly limited. For example, training data is used to train the discrimination model, using the first image 41 as an example, in which an image obtained by extracting only a portion of the first image 41 in which at least one of a stationary object and a road surface is displayed is used as a correct answer.

[0026] The discrimination model is, for example, a model that labels each pixel in the first image 41 as belonging to a first region or a second region, and is specifically a convolutional neural network that segments the first image 41 into the first region and the second region by semantic segmentation. This convolutional neural network has an encoder-decoder structure. When the first image 41 is input to the input layer, a feature map smaller in size than the first image 41 is generated in the convolutional layer, which corresponds to the encoder. Then, the smaller feature map is enlarged to the size of the original first image 41 in the pooling layer, which corresponds to the decoder. When the feature map is generated in the pooling layer, each pixel is labeled as belonging to the first region or the second region. Then, an image segmented into the first region and the second region is output from the output layer.

[0027] 4 is an example of an image output from the discrimination model when the first image 41 shown in Fig. 2 is input to the discrimination model, and in the image 43, the non-hatched portion is the first region A1, and the hatched portion is the second region A2. In the image 43, the portions corresponding to the road surface, lane boundary lines 61, 62, stop lines 63, 66, 67, arrow sign 64, crosswalk 65, and curb 71 of the first image 41 shown in Fig. 2 are classified as the first region A1, and the remaining portions are classified as the second region A2.

[0028] The discrimination unit 21 may detect objects around the vehicle using at least one of the imaging device 11 and the distance measuring device 12, and discriminate between the first area A1 and the second area A2 based on the detection results of the objects and stationary object information. For example, when the discrimination unit 21 detects objects around the mounted vehicle 50 from the detection results of the imaging device 11 and the distance measuring device 12, the discrimination unit 21 labels pixels in a portion of the first image 41 corresponding to the detected objects as the second area A2. In the image 43 shown in FIG. 4 , portions corresponding to another vehicle 51 located in front of the mounted vehicle 50 and another vehicle 52 located to the right and in front of the mounted vehicle 50 are classified as the second area A2.

[0029] The stationary object information is information about stationary objects, and is registered in advance in the map information 13. The stationary object information includes position information about the location of the stationary object, type information about the type of the stationary object, shape information about the shape of the stationary object, etc., and includes at least the position information and type information of the stationary object displayed in the first area A1. The type information includes information about the type of stationary object, such as a road marking, curb, traffic light, or road sign, and information about whether the stationary object is an obstacle to the travel of the mounted vehicle 50.

[0030] When a stationary object present in the first area A1 is identified as a stationary object used to detect the driving range of the mounted vehicle 50 (hereinafter also simply referred to as the driving range), the discrimination unit 21 determines, from the type information, whether the stationary object is used to detect the driving range. If the discrimination unit 21 determines that the stationary object is a first stationary object used to detect the driving range, it labels the pixels of the first image 41 in a portion corresponding to the first stationary object as the first area A1. On the other hand, if the discrimination unit 21 determines that the stationary object is a second stationary object not used to detect the driving range, it labels the pixels of the first image 41 in a portion corresponding to the second stationary object as the second area A2. In the image 43 shown in FIG. 4 , the portions corresponding to traffic lights 73 and 74, a sign board 75, and a building 76 are classified as portions corresponding to the second stationary object and belong to the second area A2.

[0031] The generation unit 22 determines whether a stationary object exists at a second position on the map corresponding to the first position in the first area A1. The first position is a position on the first image 41 and is represented by a two-dimensional coordinate system such as an image coordinate system or a normalized image coordinate system. On the other hand, the second position is an actual position used in the map information 13 or the like and is represented by a three-dimensional coordinate system such as a world coordinate system or a camera coordinate system. The first position and the second position can be converted to each other by a known coordinate conversion method using a rotation matrix, a translation vector, or the like. Furthermore, the first position may be a position corresponding to a pixel in the first image 41, or may be represented by a matrix (x, y) indicating the vertical and horizontal positions of the pixel.

[0032] For example, the generation unit 22 converts the first position expressed in the two-dimensional coordinate system of the first image 41 into the three-dimensional coordinate system used in the map information 13, and calculates a second position corresponding to the first position in the first area A1. Next, the generation unit 22 detects the current position of the mounted vehicle 50 from the current position information acquired from the self-position detection device 14, and acquires position information of stationary objects around the current position of the mounted vehicle 50 from the map information 13. Then, it is determined from the position information whether or not a stationary object exists at the second position.

[0033] If a stationary object is present at the second position, the generation unit 22 determines that the presence of a stationary object at the second position is registered in the map information 13. In this case, the generation unit 22 projects a corresponding display of the stationary object present at the second position onto the first position. On the other hand, if no stationary object is present at the second position, the generation unit 22 determines that the absence of a stationary object at the second position is registered in the map information 13. In this case, the generation unit 22 does not project a corresponding display onto the first position. Furthermore, the generation unit 22 does not project a corresponding display onto the second area A2. In this manner, the generation unit 22 generates the second image by projecting a corresponding display of the stationary object present in the first area A1. Furthermore, the generation unit 22 may display the generated second image on a display device (not shown) of the vehicle 50. Note that, if the map information indicates that a stationary object exists at a second position corresponding to the first position, the generation unit 22 may project a corresponding display corresponding only to the stationary object present at the second position onto the first position.

[0034] As an example, a process for generating a second image from the image 43 shown in FIG. 4 and the corresponding indicators 81 to 87 and 91 to 93 shown in FIG. 3 will be described. The corresponding indicators 81 to 87 and 91 to 93 are superimposed on the image 43. The generator 22 maintains the portions of the corresponding indicators 81 to 87 and 91 to 93 that are superimposed on the first area A1 and deletes the portions that are superimposed on the second area A2. FIG. 5 shows a second image 44 generated by deleting the corresponding indicators that are superimposed on the second area A2. The second image 44 shown in FIG. 5 includes projected corresponding indicators 81a and 82a corresponding to lane boundary lines, corresponding indicators 83a, 86a, 86b, and 87a corresponding to stop lines, and corresponding indicator 91a corresponding to a curb. However, the corresponding indicators corresponding to stationary objects that are not displayed in the first image 41 (i.e., not captured by the image capture device 11) are not displayed.

[0035] It is sufficient that the corresponding indicators of the stationary objects are projected onto the second image, and indicators other than the corresponding indicators may be projected as necessary, or projection of such indicators may be omitted. For example, the generation unit 22 may delete the hatching indicating the second region A2 from the second image 44 shown in FIG. 5 and generate a second image that displays only the corresponding indicators 81 a, 82 a, 83 a, 86 a, 86 b, 87 a, and 91 a. The generation unit 22 may also generate the second image 45 shown in FIG. 6 by superimposing the corresponding indicators 81 a, 82 a, 83 a, 86 a, 86 b, 87 a, and 91 a shown in FIG. 5 on the first image 41 shown in FIG.

[0036] The learning unit 23 performs machine learning using the second images 44 and 45 generated by the generation unit 22, generates a detection model that detects stationary objects and their types from the first image 41, and outputs the generated detection model to the driving assistance device 31 and the map generation device 32. The detection model is a trained model that has been trained in advance to detect stationary objects and their types from the first image 41, and the type of model is not particularly limited. The detection model is, for example, a neural network that includes an input layer, at least one intermediate layer, and an output layer, with each layer including at least one neuron. When input data including the first image 41 is input to the input layer, a feature map of the first image 41 is generated in the intermediate layer, and output data including the detected stationary objects and their attributes is output from the output layer.

[0037] The learning unit 23 associates type information regarding the type of stationary object with the corresponding display in the second images 44 and 45, and trains a detection model using the second images 44 and 45 in which the type information is associated with the corresponding display. For example, the learning unit 23 uses the first image 41 shown in FIG. 2 as an example and trains the detection model using training data in which type information is associated with the corresponding displays 81a, 82a, 83a, 86a, 86b, 87a, and 91a in the second image 45 shown in FIG. 6 as a correct answer. The learning unit 23 may also train the detection model by transfer learning using the discrimination model included in the discrimination unit 21. For example, the learning unit 23 replaces the final layer of the discrimination model and transfers the stationary object detection results obtained by the discrimination model to a model that detects the type of stationary object. This reduces the cost (e.g., time) required for training the detection model.

[0038] The driving assistance device 31 is a computer including a CPU, ROM, and RAM, and uses the detection model learned by the learning unit 23 to detect lane boundary lines on the road surface from the first image 41, recognize the lane in which the vehicle is traveling, and perform driving assistance for the vehicle so that the vehicle travels along the recognized lane. For example, the driving assistance device 31 detects a lane boundary line 61 on the left side of the equipped vehicle 50 and a lane boundary line 62 on the right side from the first image 41 shown in FIG. 2 , and recognizes the area between the lane boundary lines 61 and 62 as a lane. The driving assistance device 31 controls the travel of the equipped vehicle 50 by autonomous driving control so that the equipped vehicle 50 travels between the lane boundary lines 61 and 62. Note that the vehicle that is the target of driving assistance by the driving assistance device 31 is not limited to the equipped vehicle 50 and is not particularly limited.

[0039] The ROM of the driving assistance device 31 includes a program for executing autonomous driving control using a CPU, which is an operating circuit. Autonomous driving control refers to autonomously controlling the driving behavior of a vehicle using a vehicle controller, and this driving behavior includes all driving behaviors such as acceleration, deceleration, starting, stopping, and steering. Autonomously controlling driving behavior refers to the controller controlling driving behavior using a vehicle device. The controller controls these driving behaviors within a predetermined range, and driving behaviors that are not controlled by the controller are manually operated by the driver. When the vehicle is driven manually by the driver without autonomous driving control, the controller does not autonomously control the driving behavior, and the driving behavior of the vehicle is controlled by the driver's operation.

[0040] Furthermore, the detection model learned by the learning unit 23 may be mounted on a vehicle. A vehicle equipped with the detection model detects at least one of road markings and road signs from the first image 41, recognizes at least one of a driving lane and a stop line based on the detection results of the road markings and road signs, and drives along the driving lane and / or stops before the stop line by autonomous driving control. For example, a vehicle equipped with the detection model detects a stop line 63 from the first image 41, and drives by autonomous driving control to stop before the stop line 63. The vehicle may also be equipped with a driving assistance device 31.

[0041] On the other hand, the map generation device 32 is a computer equipped with a CPU, ROM, and RAM, and uses the detection model trained by the learning unit 23 to detect at least one of road signs and road signs from the first image 41, and compares the road sign and road sign detection results with stationary object information. If there is a discrepancy between the road sign and road sign detection results and the stationary object information, the map generation device 32 generates map information 13 of the surroundings of the vehicle based on the road sign and road sign detection results. Alternatively or additionally, if there is a discrepancy between the road sign and road sign detection results and the stationary object information, the map generation device 32 updates the map information 13 based on the road sign and road sign detection results.

[0042] The ROM of the map generating device 32 stores a program for causing the CPU, which is an operating circuit, to generate or update the map information 13. As an example, if a crosswalk 65 is detected in the first image 41 and the crosswalk 65 is not registered in the stationary object information, the map generating device 32 registers in the map information 13 that the crosswalk 65 exists at a position on the map corresponding to the position on the image where the crosswalk 65 is displayed. As another example, if a road sign is detected in the first image 41 and the regulation content of the detected road sign differs from the regulation content of the road sign registered in the stationary object information, the map generating device 32 updates the regulation content of the road sign registered in the stationary object information to the regulation content of the detected road sign.

[0043] [Processing in Image Generation System] The information processing procedure in the image generation system 10 will be described with reference to Fig. 7. The processing described below is executed at predetermined time intervals by a processor (CPU) included in the image generation device 15.

[0044] First, in step S1, the discrimination unit 21 acquires a first image 41 from the imaging device 11, and in the subsequent step S2, detects objects around the mounted vehicle 50 from the detection results of the imaging device 11 and the distance measuring device 12. In the subsequent step S3, the discrimination unit 21 uses a discrimination model to discriminate between a first area A1 and a second area A2 in the first image 41. In step S4, the generation unit 22 acquires current position information of the mounted vehicle 50 from the self-position detection device 14, and in the subsequent step S5, acquires stationary object information from the map information 13, and in the subsequent step S6, calculates a second position on the map corresponding to the first position on the image.

[0045] In step S7, the generation unit 22 determines whether or not a stationary object exists at a second position corresponding to the first position. If it is determined that a stationary object exists at the second position, the process proceeds to step S8, where a corresponding display of the stationary object is projected at the first position, and then proceeds to step S9. On the other hand, if it is determined that a stationary object does not exist at the second position, the process proceeds to step S9. In step S9, the generation unit 22 determines whether or not a corresponding display of all stationary objects in the first region has been projected. If it is determined that not all corresponding displays of stationary objects in the first region have been projected, the process proceeds to step S7. On the other hand, if it is determined that all corresponding displays of stationary objects in the first region have been projected, the process outputs second images 44 and 45, and then proceeds to step S10. In step S10, the learning unit 23 associates type information with the corresponding displays of the second images 44 and 45, and then ends the process.

[0046] 8 and 9, the information processing procedures in the driving assistance device 31 and the map generation device 32 will be described. The processing described below is executed at predetermined time intervals by processors (CPUs) provided in the driving assistance device 31 and the map generation device 32.

[0047] 8 is a flowchart showing an example of a processing procedure in the driving assistance device 31. First, in step S21, road signs and road traffic signs are detected using a detection model, and then in step S22, the driving lane and stop line are recognized based on the detection results of the road signs and road traffic signs. Then, in step S23, autonomous driving control is executed based on the recognized driving lane and stop line. Then, the processing ends.

[0048] 9 is a flowchart showing an example of a processing procedure in the map generating device 32. First, in step S31, road signs and road signs are detected using a detection model, and then in step S32, the road sign and road sign detection results are compared with stationary object information. In step S33, it is determined whether the road sign and road sign detection results and the stationary object information differ. If it is determined that the detection results and the stationary object information differ, the process proceeds to step S34, where map information 13 is generated or updated based on the road sign and road sign detection results. On the other hand, if it is determined that the detection results and the stationary object information do not differ, the process ends.

[0049] According to the present embodiment, in an image generation method executed by an image generation device 15, the image generation device 15 distinguishes between a first area A1 in which at least one of a stationary object and a road surface is displayed and a second area A2 other than the first area A1 and including at least a moving object in a first image 41 acquired from an imaging device 11 mounted on a vehicle, and if map information 13 including stationary object information regarding the stationary object indicates that the stationary object is present at a second position on the map corresponding to a first position in the first area A1, the image generation device 15 projects corresponding indicators 81a, 82a, 83a, 86a, 86b, 87a, and 91a corresponding only to the stationary object present at the second position onto the first position to generate second images 44 and 45 on which the corresponding indicators 81a, 82a, 83a, 86a, 86b, 87a, and 91a are projected. This makes it possible to generate images for machine learning that suppress false detection of stationary objects.

[0050] In the image generating method of this embodiment, the image generating device 15 detects objects around the vehicle using at least one of the imaging device 11 and the distance measuring device 12 mounted on the vehicle, and distinguishes between the first area A1 and the second area A2 based on the detection results of the objects and the stationary object information. This makes it possible to distinguish between the first area A1 and the second area A2 taking into account the objects around the vehicle.

[0051] In the image generating method of this embodiment, the stationary object information includes type information regarding the type of the stationary object displayed in the first area A1, which allows the type of the detected stationary object to be recognized.

[0052] In the image generation method of this embodiment, the stationary objects include at least one of road markings, road signs, traffic lights 73 and 74, structures that are provided on sidewalks adjacent to the road and indicate the boundary between the road and the sidewalk, and landmark buildings 76. This allows for more accurate detection of stationary objects.

[0053] In the image generating method of this embodiment, the image generating device 15 displays the second images 44, 45 on a display device of the vehicle, thereby making it possible to present the second images 44, 45 to the vehicle occupants.

[0054] According to the present embodiment, there is provided a learning method executed by the image generation device 15, in which type information regarding the type of the still object is associated with the corresponding indication of the second images 44, 45 generated by the above-described image generation method, and the second images 44, 45 in which the type information is associated with the corresponding indication are used to learn a detection model that detects the still object and its type from the first image 41. In this way, a detection model trained using the second images 44, 45 can be generated.

[0055] In the learning method of this embodiment, the image generating device 15 includes a discrimination model that discriminates between the first region A1 and the second region A2 in the first image 41, and the detection model, and trains the detection model by transfer learning using the discrimination model. This reduces the cost (e.g., time) required for training the detection model.

[0056] According to this embodiment, a driving assistance device 31 is provided that uses the detection model trained by the above-described learning method to detect lane boundary lines 61, 62 on the road surface from the first image 41, recognize the lane in which the vehicle is traveling, and performs driving assistance for the vehicle so that the vehicle travels along the lane. This allows driving assistance to be performed using the detection model trained using the second images 44, 45.

[0057] According to this embodiment, a vehicle is provided that uses the detection model trained by the above-described learning method to detect at least one of road marks and road signs of the road from the first image 41, recognizes at least one of a driving lane and stop lines 63, 66, 67 based on the detection results of the road marks and the road signs, and drives by autonomous driving control so as to drive along the driving lane and / or stop before the stop lines 63, 66, 67. This allows the vehicle to drive by autonomous driving control using the detection model trained using the second images 44, 45.

[0058] According to this embodiment, a map generating device 32 is provided that uses the detection model trained by the above-described learning method to detect at least one of road marks and road signs of the road from the first image 41, compares the detection results of the road marks and the road signs with the stationary object information, and if there is a discrepancy between the detection results of the road marks and the road signs and the stationary object information, generates the map information 13 of the surroundings of the vehicle based on the detection results of the road marks and the road signs, or updates the map information 13 based on the detection results of the road marks and the road signs. This makes it possible to generate or update the map information 13 using the detection model trained using second images 44 and 45.

[0059] Furthermore, according to the present embodiment, an image generation device 15 is provided, which includes: a discrimination unit 21 that discriminates between a first area A1 in which at least one of a stationary object and a road surface is displayed and a second area A2 other than the first area A1 and including at least a moving object in a first image 41 acquired from an imaging device 11 mounted on a vehicle; and a generation unit 22 that, if map information 13 including stationary object information regarding the stationary object indicates that the stationary object exists at a second position on the map corresponding to a first position of the first area A1, projects corresponding indicators 81a, 82a, 83a, 86a, 86b, 87a, and 91a corresponding only to the stationary object present at the second position onto the first position to generate second images 44 and 45 on which the corresponding indicators 81a, 82a, 83a, 86a, 86b, 87a, and 91a are projected. This makes it possible to generate machine learning images that suppress false detection of stationary objects.

[0060] 10...image generation system, 11...imaging device, 12...distance measuring device, 13...map information, 14...self-position detection device, 15...image generation device, 21...discrimination unit, 22...generation unit, 23...learning unit, 31...driving assistance device, 32...map generation device 41...first image, 42, 43...image, 44, 45...second image 50...equipped vehicle, 51, 52...other vehicles, 61, 62...lane boundary line, 63, 66, 67...stop line, 64...arrow sign, 65...pedestrian crossing, 71, 72...curb, 73, 74...traffic light, 75...sign, 76...building 81, 82, 83, 84, 85, 86, 87, 91, 92, 93, 81a, 82a, 83a, 86a, 86b, 87a, 91a...corresponding display A1...first area, A2...second area

Claims

1. In an image generation method executed by an image generation device, the image generation device discriminates, in a first image acquired from an imaging device mounted on a vehicle, a first region in which at least one of a stationary object and a road surface is displayed, and a second region that is outside the first region and includes at least a moving object, and when the stationary object is registered as existing at a second position on a map corresponding to a first position of the first region in map information including stationary object information regarding the stationary object, projects a corresponding display corresponding only to the stationary object existing at the second position at the first position to generate a second image on which the corresponding display is projected. Image generation method.

2. The image generation device detects an object around the vehicle using at least one of the imaging device and a distance measuring device mounted on the vehicle, and discriminates the first region and the second region based on a detection result of the object and the stationary object information. The image generation method according to claim 1.

3. The stationary object information includes type information regarding a type of the stationary object displayed in the first region. The image generation method according to claim 1 or 2.

4. The stationary object includes at least one of a road surface marking, a road sign, a traffic signal, a structure provided on a sidewalk adjacent to the road and indicating a boundary between the road and the sidewalk, and a building serving as a landmark. The image generation method according to any one of claims 1 to 3.

5. The image generation device displays the second image on a display device of the vehicle. The image generation method according to any one of claims 1 to 4.

6. Associating type information regarding a type of the stationary object with the corresponding display in the second image generated by the image generation method according to any one of claims 1 to 5, and using the second image in which the type information is associated with the corresponding display to train a detection model for detecting the stationary object and its type from the first image. Learning method executed by the image generation device.

7. The image generation device includes a discrimination model for discriminating the first region and the second region in the first image, and the detection model, and trains the detection model by transfer learning using the discrimination model. The learning method according to claim 6.

8. Using the detection model learned by the learning method according to claim 6 or 7, detecting a lane boundary line of the road surface from the first image to recognize a lane in which the vehicle travels, and executing driving assistance for the vehicle so that the vehicle travels along the lane, a driving assistance device.

9. Using the detection model learned by the learning method according to claim 6 or 7, detecting at least one of the road surface markings and road signs of the road from the first image, recognizing at least one of the driving lane and the stop line based on the detection results of the road surface markings and the road signs, and a vehicle that travels by autonomous driving control so as to travel along the driving lane and / or stop in front of the stop line.

10. Using the detection model learned by the learning method according to claim 6 or 7, detecting at least one of the road surface markings and road signs of the road from the first image, comparing the detection results of the road surface markings and the road signs with the stationary object information, and when there is a difference between the detection results of the road surface markings and the road signs and the stationary object information, generating the map information around the vehicle based on the detection results of the road surface markings and the road signs or updating the map information based on the detection results of the road surface markings and the road signs, a map generation device.

11. A discrimination unit that discriminates a first region in which at least one of a stationary object and a road surface is displayed and a second region that is outside the first region and includes at least a moving object in a first image acquired from an imaging device mounted on a vehicle, and a generation unit that generates a second image on which a corresponding display is projected by projecting a corresponding display corresponding only to the stationary object existing at the second position on the map corresponding to the first position in the first region when the stationary object is registered as existing at the second position on the map corresponding to the first position in the map information including the stationary object information regarding the stationary object, an image generation device.

Images