Image generation method and image generation device

Abstract

In the present invention, when, while generating a second image (42, 42a) in which a corresponding display corresponding to a stationary object registered in map information (13) is projected to a first image (41) acquired from a camera (11) mounted on a vehicle (50) within the imaging range of the camera (11), it is determined that the difference between a first distance (D1) from the stationary object existing at a first position (P1) registered in the map information (13) to a mounting position (P3) of the camera (11) and a second distance (D2) from the subject of the camera (11) captured at a second position (A) corresponding to the first position (P1) in the first image (41) to the mounting position (P3) of the camera (11) is equal to or greater than a predetermined distance, the corresponding display of the whole or part of the stationary object present at the first position (P1) is not projected to the second position (A) of the first image (41). When it is determined that the difference between the first distance (D1) and the second distance (D2) is less than the predetermined distance, the corresponding display of the stationary object present at the first position (P1) is projected to the second position (A) of the first image (41).

Description

Image generation method and image generation device

[0001] The present invention relates to an image generation method and an image generation device.

[0002] A method for labeling images for machine learning is known, in which an image of a roadside object is acquired from an on-board camera, the position of the roadside object in the image is correlated with the position information of the roadside object taking into account the position and orientation of the vehicle, and the roadside object in the image is labeled taking into account the correlated position (Patent Document 1).

[0003] Special Publication No. 2022-514891

[0004] In the above-mentioned conventional technology, when a part of a roadside object is obstructed by another object, an image of the roadside object that includes a different object is labeled as a roadside object. Therefore, when an image labeled by the above-mentioned conventional technology is used for machine learning, there is a problem that the roadside object cannot be accurately detected.

[0005] The problem that the present invention aims to solve is to provide an image generation method and an image generation device that, when used in machine learning, can generate images that suppress erroneous detection of stationary objects on or around a road.

[0006] The present invention solves the above problem by, when a second image is generated on a first image acquired from a camera mounted on a vehicle, projecting a corresponding display corresponding to a stationary object registered in map information within the camera's imaging range, if it is determined that the difference between a first distance from a stationary object located at a first position registered in the map information to the camera's mounting position and a second distance from the camera's subject captured at a second position corresponding to the first position in the first image to the camera's mounting position is greater than a predetermined distance, not projecting a corresponding display of all or part of the stationary object located at the first position onto the second position in the first image, and if it is determined that the difference between the first distance and the second distance is less than the predetermined distance, projecting a corresponding display of the stationary object located at the first position onto the second position in the first image.

[0007] According to the present invention, when used in machine learning, it is possible to generate an image that suppresses erroneous detection of stationary objects that exist on or around the road.

[0008] 1 is a block diagram showing an example of an embodiment of an image generation system according to the present invention. FIG. 1 is a diagram showing an example of a first image acquired from the camera of FIG. 1. FIG. 2 is a diagram showing an example of an image in which a corresponding display of a stationary object is projected onto the first image of FIG. 2. FIG. 3 is a diagram showing a first distance and a second distance at position A of FIG. 3. FIG. 4 is a diagram showing an example of a second image generated from the first image of FIG. 2. FIG. 5 is a diagram showing another example of a second image generated from the first image of FIG. 2. FIG. 6 is a diagram showing an example of a measurement result of a distance measuring device at position B of FIG. 3. FIG. 7 is a flowchart (part 1) showing an example of a processing procedure in the image generation system of FIG. 1. FIG. 8 is a flowchart (part 2) showing an example of a processing procedure in the image generation system of FIG. 1. FIG. 9 is a flowchart showing another example of a processing procedure in the image generation system of FIG. 1.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [Configuration of Image Generation System] Figure 1 is a block diagram showing an example of an embodiment of an image generation system 10 according to the present invention. The image generation system 10 is a system that generates training data for supervised learning, consisting of an image captured by a camera mounted on a vehicle (hereinafter also referred to as a first image) and an image (hereinafter also referred to as a second image) on which a corresponding display corresponding to a stationary object (hereinafter simply referred to as a stationary object) present on or around the road is projected. In the training data, the first image is a sample problem, and the second image is a correct answer to the sample problem. The vehicle is not particularly limited, and the first image may be acquired from on-board cameras of multiple vehicles.

[0011] The target object is an object that exists around the vehicle on which the camera is mounted (hereinafter also referred to as the mounted vehicle), and includes lane lines, center lines, road markings, medians, guardrails, curbs, road signs, traffic lights, crosswalks, etc. The target object also includes obstacles that may affect the driving of the mounted vehicle, such as other vehicles other than the mounted vehicle, motorcycles, bicycles, pedestrians, etc.

[0012] The stationary object is not particularly limited as long as it exists on or around the road and its position does not change, and includes objects that are fixed, attached, or installed on the road or its surroundings. Stationary objects also include objects related to the detection of the driving range of the equipped vehicle (for example, at least one of road markings, road signs, traffic lights, and landmark buildings). The stationary objects and their location information are included (registered) in the map information described below.

[0013] A corresponding display corresponding to a stationary object is a display indicating the presence of a stationary object. In the second image, a display such as a solid line, a dashed line, a dot, or a rectangle is projected at the location where the stationary object is located or at a location corresponding to the stationary object based on map information (specifically, the location information of the stationary object). The shape and color of the corresponding display can be appropriately set within a range that allows appropriate machine learning to be performed using training data including the second image on which the corresponding display is projected. Note that instead of projecting the corresponding display onto the second image, the corresponding display may be displayed on the second image, or the corresponding display may be placed on the second image.

[0014] As shown in FIG. 1 , the image generation system 10 includes a camera 11, a distance measuring device 12, map information 13, a self-location detection device 14, and an image generation device 15. The camera 11 is an on-board camera, and the distance measuring device 12 and the self-location detection device 14 are mounted on the vehicle. The map information 13 is stored in an on-board storage medium of the vehicle or in a storage medium outside the vehicle. The image generation device 15 may be mounted on the vehicle or may be provided outside the vehicle (e.g., in a remote location away from the vehicle). These devices are connected to each other by a Controller Area Network (CAN) or other on-board LAN or via a network so that they can exchange information with each other. The network refers to a telecommunications network such as the Internet, and the communication format is not particularly limited.

[0015] The camera 11 is a device that captures images of objects around the vehicle, and examples thereof include a camera equipped with an imaging element such as a CCD, an infrared camera, etc. The distance measuring device 12 is a device that acquires the relative distance between the distance measuring device 12 and the object, and examples thereof include a millimeter-wave radar and a LiDAR (Light Detection and Ranging) unit. The distance measuring device 12 also includes at least one of a stereo camera, radar, and LiDAR mounted on the vehicle. Multiple cameras 11 and distance measuring devices 12 are installed at the front, right side, left side, rear, etc. of the vehicle to reduce blind spots when detecting objects.

[0016] The map information 13 includes information on nodes corresponding to points where the vehicle's traveling direction changes (intersections, branching points, etc.) and information on links corresponding to road sections connecting the nodes. Node information includes location information (e.g., latitude and longitude) and information on entering and exiting intersections, while link information includes road width and curvature radius, roadside structures, road traffic regulations, etc. Furthermore, the map information 13 may be high-precision map information that includes road information, facility information, and their attribute information, and that allows tracking of the movement trajectory for each lane.

[0017] The self-position detection device 14 is a positioning system that detects the current position of the vehicle, and calculates the current position of the vehicle from, for example, radio waves received from a satellite for the GPS (Global Positioning System). The image generation device 15 acquires detection results from the camera 11, the distance measurement device 12, and the self-position detection device 14 at predetermined time intervals (for example, every 0.1 to 1 millisecond), and acquires map information 13 from a storage medium (not shown) as needed.

[0018] The image generation device 15 controls the various devices constituting the image generation system 10 to cooperate with each other, and generates a second image, based on the first image and the map information 13, of a stationary object present in the imaging range of the camera 11, in which a corresponding display corresponding to the stationary object is projected. The image generation device 15 is, for example, a computer, and includes a central processing unit (CPU) as a processor, a read-only memory (ROM) in which a program is stored, and a random access memory (RAM) that functions as an accessible storage device. The CPU of the image generation device 15 is an operating circuit for executing a program stored in the ROM to generate a second image from the first image, and the program includes a calculation unit 21, a determination unit 22, and a generation unit 23, which are functional blocks for generating the second image. These functional blocks are illustrated in FIG. 1 for convenience.

[0019] [Functions of the Image Generating Device] Each functional block shown in Fig. 1 will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of an image (first image) acquired from the camera 11, which is an image of the area ahead of the mounted vehicle stopped in front of an intersection captured by the camera 11 attached to the top of the windshield. A portion of the body of the stopped mounted vehicle 50 is displayed at the bottom of the first image 41 shown in Fig. 2. Another vehicle 51 stopped in front of the mounted vehicle 50 is displayed in the center of the first image 41, and another vehicle 52 stopped to the right and in front of the mounted vehicle 50 is displayed on the right side of the first image 41.

[0020] A lane boundary line 61 is displayed on the left side of the mounted vehicle 50, a lane boundary line 62 is displayed on the right side of the mounted vehicle 50, and a stop line 63 is displayed between the other vehicles 51, 52. A portion of a road marking 64 indicating the direction in which the vehicle may proceed is displayed between the mounted vehicle 50 and the other vehicle 51. A crosswalk 65 is displayed above the stop line 63, and a stop line 66 is displayed above the crosswalk 65. A stop line 67 is displayed above the stop line 66, located beyond the intersection (i.e., on the opposite side of the intersection from the mounted vehicle 50).

[0021] A curb 71 on the shoulder of the road is displayed to the left of the lane boundary line 61, and a curb 72 on the shoulder of the intersecting lane that intersects the lane in which the loaded vehicle 50 is traveling at the intersection is displayed to the left of the first image 41. A traffic light 73 is displayed above the other vehicle 51, and a traffic light 74 is displayed to the left of the traffic light 73 (above the curb 72). A road sign 75 indicating that a turn is prohibited is displayed to the left of the traffic light 74. A building 76 that serves as a landmark when traveling around the intersection is displayed in the upper right side of the first image 41.

[0022] When generating the second image from the first image 41 shown in FIG. 2 , for example, corresponding indicators for stationary objects are projected onto the first image 41 based on position information of the stationary objects converted from the three-dimensional coordinate system used in the map information 13 to the two-dimensional coordinate system of the first image 41. FIG. 3 is a diagram showing an example of an image in which corresponding indicators for stationary objects are projected onto the first image 41. In the image 41x shown in FIG. 3 , corresponding indicators for stationary objects used to detect the driving range of the vehicle 50 are projected. Specifically, corresponding indicators 81, 82, 84, and 85 corresponding to lane boundary lines for recognizing lanes, corresponding indicators 83, 86, and 87 corresponding to stop lines for recognizing stopping positions, and corresponding indicators 91, 92, and 93 corresponding to curbs for recognizing the driving range of the road including side roads are projected. In the image 41x, the corresponding indicators for lane boundary lines and stop lines are solid lines, and the corresponding indicators for curbs are dashed lines.

[0023] 3, corresponding indicators for stationary objects not displayed in the first image 41 (i.e., not captured by the camera 11) are projected. For example, the lane boundary lines 61 and 62 are partially obscured by the vehicle 50 and not displayed in the first image 41. However, in the image 41x, portions of corresponding indicators 81 and 82 corresponding to the lane boundary lines 61 and 62 are projected and superimposed on the body of the vehicle 50. Similarly, corresponding indicators 84 and 85 corresponding to lane boundary lines not captured by the camera 11 are projected and superimposed on the other vehicle 52. Furthermore, portions of corresponding indicators 83 and 86 corresponding to the stop lines 63 and 66 are projected and superimposed on the other vehicles 51 and 52, and a portion of corresponding indicator 87 corresponding to the stop line 67 is projected and superimposed on the other vehicle 51. Furthermore, a corresponding indicator 93 corresponding to a curb not captured by the camera 11 is projected on the image 41x.

[0024] When machine learning is performed in a neural network having an input layer, at least one intermediate layer, and an output layer using training data in which the first image 41 is used as an example and the second image is used as the correct answer for the first image 41, when an image is input to the input layer of the trained neural network, the trained neural network outputs an image from the output layer on which a display (solid line, dashed line, dot, etc.) corresponding to the still object displayed in the input image is projected. In other words, the trained neural network becomes a trained model that detects still objects displayed in the input image.

[0025] For example, if machine learning is performed on a neural network using training data in which the first image 41 shown in FIG. 2 is used as an example and the image 41x shown in FIG. 3 is used as the correct answer for the first image 41, the trained neural network will output an image, such as image 41x, in which a corresponding indication of a still object not displayed in the input image (i.e., not captured by camera 11) is projected for the image input to the input layer. In other words, the trained neural network will become a model that cannot accurately detect the still object displayed in the input image. Therefore, the image generation device 15, using the functions of the functional blocks shown in FIG. 1, detects a corresponding indication in image 41x that hinders proper machine learning and deletes the detected corresponding indication.

[0026] The calculation unit 21 calculates a first distance from a stationary object located at a first position registered in the map information 13 to the mounting position of the camera 11. The first position is an actual position used in the map information 13 or the like, and is expressed by a three-dimensional coordinate system such as a world coordinate system or a camera coordinate system. The mounting position of the camera 11 is, for example, the optical center of the camera 11, and is determined in advance as a relative position with respect to the current position of the mounted vehicle 50.

[0027] Furthermore, the calculation unit 21 calculates a second distance from the subject of the camera 11, which is captured (or projected) at a second position corresponding to the first position in the first image 41, to the mounting position of the camera 11. The calculation unit 21 calculates the second distance based on, for example, the measurement results of a distance measuring device 12 (stereo camera, radar, etc.) mounted on the mounted vehicle 50, the measurement results of a ToF (Time-of-Flight) camera, the measurement results of a camera that can measure the distance to the subject using aberration, etc.

[0028] The second position is a position on the first image 41 and is represented by a two-dimensional coordinate system such as an image coordinate system or a normalized image coordinate system. The first position and the second position can be converted into each other by a known coordinate conversion method using a rotation matrix, a translation vector, or the like. The second position may be a position corresponding to a pixel on the first image 41, or may be represented by a matrix (x, y) indicating the vertical and horizontal positions of the pixel. The subject of the camera 11 is an object captured by the camera 11 and is not particularly limited. Note that the subject of the camera 11 includes a stationary object.

[0029] As an example, the calculation unit 21 detects the current position of the mounted vehicle 50 from the current position information acquired from the self-position detection device 14, acquires position information of stationary objects around the current position of the mounted vehicle 50 from the map information 13, and detects a first position where the stationary object is located. Then, it calculates a first distance from the mounting position of the camera 11 to the first position where the stationary object is located.

[0030] Next, the calculation unit 21 converts the first position expressed in the three-dimensional coordinate system used in the map information 13 into the two-dimensional coordinate system of the first image 41, and calculates a second position in the first image 41 corresponding to the first position. Then, the calculation unit 21 calculates a second distance to the subject captured (or displayed) at the second position based on position information regarding the ranging points of the objects around the vehicle 50 acquired from the ranging device 12 and position information regarding the mounting position of the camera 11. For example, the calculation unit 21 acquires point cloud data arranged two-dimensionally in the left-right and up-down directions of the vehicle 50 from the ranging device 12, converts the coordinate system of the point cloud data into the coordinate system of the first image 41, and calculates the second distance based on position information of the ranging points corresponding to the second position on the first image. The ranging points of the object are points on the object whose distances to the ranging device 12 are measured.

[0031] An example of the first distance and the second distance will be described with reference to FIG. 4 . FIG. 4 is a diagram showing the first distance and the second distance at the second position A in FIG. 3 , and is a side view of a main part of the other vehicle 51 as viewed from the right side. As shown in the image 41x in FIG. 3 , a stop line 63 is located at the first position P1 on the road surface R of the road shown in FIG. 4 , as shown in the image 41x in FIG. 3 . Position P2 is the actual position of the other vehicle 51 (subject) displayed at the second position A corresponding to the first position P1, and position P3 is the mounting position of the camera 11. In this case, the calculation unit 21 calculates the first distance D1 from the position information of the first position P1 registered in the map information 13 and the position information of the mounting position P3 of the camera 11. Furthermore, the calculation unit 21 acquires the position information of the position P2 from the distance measuring device 12 and performs coordinate conversion of the position information to calculate the second distance D2 from the mounting position P3 of the camera 11 to the position P2 from the distance measuring device 12 to the position P2.

[0032] The determination unit 22 determines whether the difference between the first distance D1 and the second distance D2 is equal to or greater than a predetermined distance. The predetermined distance can be set to an appropriate value within a range in which appropriate machine learning can be performed using training data including the second image on which the corresponding display is projected, and is, for example, 0.25 to 1 m. For example, in the example shown in FIG. 4 , the difference between the first distance D1 and the second distance D2 exceeds a predetermined distance (e.g., 50 cm), so the determination unit 22 determines that the difference between the first distance D1 and the second distance D2 is equal to or greater than the predetermined distance.

[0033] When the generation unit 23, which generates the second image based on the first image 41 and the map information 13, determines that the difference between the first distance D1 and the second distance D2 is less than a predetermined distance, it projects a corresponding indication of a stationary object located at the first position P1 at the second position A on the first image 41. On the other hand, when the generation unit 23 determines that the difference between the first distance D1 and the second distance D2 is equal to or greater than the predetermined distance, it does not project a corresponding indication of all or part of the stationary object located at the first position P1 at the second position A on the first image 41. In the example shown in FIG. 4 , the determination unit 22 determines that the difference between the first distance D1 and the second distance D2 is equal to or greater than the predetermined distance, so the generation unit 23 does not project a straight line (corresponding indication 83) corresponding to the stop line 63 located at the first position P1 at the second position A on the first image 41.

[0034] Furthermore, if the generation unit 23 determines that the difference between the first distance D1 and the second distance D2 is less than a predetermined distance, the generation unit 23 may maintain the corresponding indication displayed at the second position A on the first image 41, and if the generation unit 23 determines that the difference between the first distance D1 and the second distance D2 is equal to or greater than the predetermined distance, the generation unit 23 may delete the corresponding indication displayed at the second position A on the first image 41. For example, as in the image 41x shown in FIG. 3 , in a case where a straight line corresponding to the stop line 63 is projected at the second position A based on the position information of the stop line 63, if the determination unit 22 determines that the difference between the first distance D1 and the second distance D2 at the second position A is equal to or greater than the predetermined distance, the generation unit 23 deletes the straight line projected at the second position A.

[0035] 5A is a diagram showing an example of a second image generated from the first image 41 of FIG. 2 and the map information 13. Unlike the image 41x shown in FIG. 3, the second image 42 shown in FIG. 5A projects only indicators corresponding to stationary objects displayed in the first image 41 (i.e., captured by the camera 11). Specifically, indicators 81a and 82a corresponding to the lane boundary lines 61 and 62, respectively, indicator 83a corresponding to the stop line 63, indicators 86a and 86b corresponding to the stop line 66, indicator 87a corresponding to the stop line 67, and indicators 91 and 92 corresponding to the curbs 71 and 72, respectively, are projected. In other words, indicators corresponding to stationary objects not displayed in the first image 41 (i.e., not captured by the camera 11) (specifically, some of the indicators 81, 82, 83, 86, and 87, and all of the indicators 84, 85, and 93) are deleted from the image 41x.

[0036] 5B is a diagram showing another example of a second image generated from the first image 41 of FIG. 2 and the map information 13. The second image 42a shown in FIG. 5B is an image in which only the corresponding display projected onto the second image 42 shown in FIG. 5A is displayed, with all displays other than the corresponding display deleted. Using the second image 42a can simplify the training data, improving the efficiency of machine learning for stationary object detection. The generation unit 23 then generates training data consisting of the first image 41 and the second images 42 and 42a, and transmits the training data to the stationary object detection device 31 shown in FIG. 1.

[0037] The stationary object detection device 31 is a model such as a neural network, and becomes a trained model for detecting stationary objects by performing machine learning using the training data generated by the generation unit 23. The stationary object detection device 31 transmits the stationary object detection results to the cruise control device 32. The cruise control device 32 controls the vehicle's onboard devices to cooperate with each other to control the vehicle's travel and drive the vehicle to a set destination. The cruise control device 32 acquires the stationary object detection results from the stationary object detection device 31 and uses information about the detected stationary objects to cause the vehicle to travel autonomously. Note that the vehicle that is driven autonomously by the cruise control device 32 is not limited to the onboard vehicle 50.

[0038] If the distance measuring device 12 has not measured the second distance D2, the calculation unit 21 may obtain a third distance to a subject captured (or displayed) near the second position from the distance measuring device 12 and estimate the second distance D2 from the third distance. Figure 6 shows an example of the measurement results of the distance measuring device 12 at position B in Figure 3 , where the distances to the subjects are measured by the distance measuring device 12 at distance measuring points Q1, Q2, Q3, Q4, and Q5. In this case, the second distance D2 to the other vehicle 51 is not measured by the distance measuring device 12 at position P4 on the other vehicle 51. Therefore, the calculation unit 21 obtains the distances to the subject at distance measuring points Q2 and Q3 on both sides of position P4 (or closest to position P4) (i.e., the third distance) and estimates the second distance D2 to the other vehicle 51 at position P4 from the distances to the subject at distance measuring points Q2 and Q3. As an example, the calculation unit 21 linearly interpolates the measurement results between the distance measurement points Q2 and Q3 to estimate the second distance D2 at the position P4.

[0039] The generator 23 determines whether the position P2 of the subject captured at the second position A on the first image 41 is higher than the road surface by a predetermined height or more. If the generator 23 determines that the position P2 of the subject is higher than the road surface by a predetermined height or more, it is not necessary to project a corresponding display of all or part of the stationary object present at the first position P1 onto the second position A on the first image 41. On the other hand, if the generator 23 determines that the position P2 of the subject captured at the second position A on the first image 41 is lower than the predetermined height above the road surface, the generator 23 projects a corresponding display of the stationary object present at the first position P1 onto the second position A on the first image 41. The predetermined height can be set to an appropriate value within a range in which appropriate machine learning can be performed using training data including the second images 42 and 42a on which the corresponding display is projected, and is, for example, 0.2 to 1 m. As an example, in the example shown in FIG. 4 , the height H from the road surface R to the position P2 is equal to or greater than a predetermined height (e.g., 30 cm), so a straight line corresponding to the stop line 63 present at the first position P1 is not projected onto the second position A on the image 41x.

[0040] When generating training data consisting of the first image 41 and the second images 42, 42a, the generation unit 23 calculates a first error in the first distance D1 due to the estimation accuracy of the current position of the mounted vehicle 50, and may set a smaller weighting for use in machine learning for the corresponding display projected at the first position P1 when the first error is large than when the first error is small. For example, when radio waves cannot be received from a GPS satellite and the accuracy of the current position information acquired from the self-position detection device 14 is low, the generation unit 23 sets a coefficient greater than 0 and less than 1 (e.g., 0.8) as a weighting for the corresponding display so that the output value from the output layer of the neural network is small.

[0041] Furthermore, when generating training data consisting of the first image 41 and the second images 42, 42a, the generation unit 23 may calculate a second error in the second distance D2 due to the measurement accuracy of the distance measuring device 12, and may set a smaller weighting for use in machine learning for the corresponding display projected at the first position P1 when the second error is large than when the second error is small. For example, when the accuracy of the position information of the subject acquired from the distance measuring device 12 is low, such as when the second distance D2 is estimated from the third distance to the subject near the second position A on the first image 41, the generation unit 23 sets a coefficient greater than 0 and less than 1 (e.g., 0.8) as a weighting for the corresponding display so that the output value from the output layer of the neural network is small.

[0042] 7A to 7B and 8, the information processing procedure in the image generation system 10 will be described. The processing described below is executed by a processor (CPU) provided in the image generation device 15.

[0043] 7A and 7B are flowcharts showing an example of a processing procedure in the image generation system 10. First, in step S11 of Fig. 7A, the calculation unit 21 detects the current position of the mounted vehicle 50 based on the current position information of the mounted vehicle 50 acquired from the self-position detection device 14, acquires position information of a stationary object from the map information 13 in step S12, detects a first position P1 where the stationary object exists in step S13, and calculates a first distance D1 from the first position P1 to the mounting position P3 of the camera 11 in step S14. In the subsequent step S15, the calculation unit 21 converts the coordinate system of the position information of the stationary object into the coordinate system of the first image 41, and in step S16, calculates a second position A on the first image 41 corresponding to the first position P1 where the stationary object is located, and in step S17, converts the coordinate system of the position information of the ranging point into the coordinate system of the first image 41, and in step S18, calculates a second distance D2 from the position P2 of the subject projected at the second position A on the first image 41 to the mounting position P3 of the camera 11 from the position information of the ranging point corresponding to the second position A.

[0044] In step S19 of FIG. 7B , the determination unit 22 determines whether the difference between the first distance D1 and the second distance D2 is equal to or greater than a predetermined distance. If the determination unit 22 determines that the difference between the first distance D1 and the second distance D2 is equal to or greater than the predetermined distance, the process proceeds to step S22, where the corresponding display is not projected at the second position A on the first image 41 (or the corresponding display of the stationary object at the second position A on the first image 41 is deleted). On the other hand, if the determination unit 22 determines that the difference between the first distance D1 and the second distance D2 is less than the predetermined distance, the process proceeds to step S20, where the generation unit 23 determines whether the position P2 of the subject displayed at the second position A on the first image 41 is higher than the road surface R by a predetermined height or more. If the generation unit 23 determines that the position P2 of the subject displayed at the second position A on the first image 41 is higher than the road surface R by a predetermined height or more, the process proceeds to step S22. On the other hand, if the generation unit 23 determines that the position P2 of the subject displayed at the second position A on the first image 41 is at a height less than a predetermined height above the road surface R, it proceeds to step S21 and projects a corresponding display of the stationary object at the second position A on the first image 41 (or maintains the corresponding display of the stationary object at the second position A on the first image 41).

[0045] In step S23, the generation unit 23 determines whether processing has been performed for all first positions P1. If it is determined that processing has been performed for all first positions P1, the process proceeds to step S24, where the generation unit 23 outputs second images 42, 42a onto which corresponding representations of stationary objects are projected. On the other hand, if it is determined that processing has not been performed for some first positions P1, the process proceeds to step S13 in FIG. 7A.

[0046] FIG. 8 is a flowchart showing another example of a processing procedure in the image generation system 10. First, in step S31, the generation unit 23 calculates a first error of the first distance D1, and in step S32, determines whether the first error is greater than a predetermined error. The predetermined error can be set to an appropriate value within a range in which appropriate machine learning can be performed using training data including the second images 42, 42a on which the corresponding display is projected. If it is determined that the first error is greater than the predetermined error, the process proceeds to step S36, in which the generation unit 23 sets a small weight for the corresponding display. On the other hand, if it is determined that the first error is equal to or less than the predetermined error, the process proceeds to step S33.

[0047] In step S33, the generation unit 23 calculates a second error of the second distance D2, and in step S34, determines whether the second error is greater than a predetermined error. If it is determined that the second error is greater than the predetermined error, the process proceeds to step S36. On the other hand, if it is determined that the second error is equal to or less than the predetermined error, the process proceeds to step S35, where the generation unit 23 maintains the weighting of the corresponding display or sets a large weighting.

[0048] [Embodiment of the Present Invention] According to the present embodiment, there is provided an image generation method executed by an image generation device 15 that generates second images 42, 42a by projecting corresponding indications corresponding to stationary objects registered in map information 13 within an imaging range of a first image 41 acquired from a camera 11 mounted on a vehicle 50 onto the first image 41. The image generation device 15 determines whether a difference between a first distance D1 from the stationary object present at a first position P1 registered in the map information 13 to the mounting position of the camera 11 and a second distance D2 from a subject of the camera 11 captured at a second position A corresponding to the first position P1 in the first image 41 to the mounting position P3 is equal to or greater than a predetermined distance. If it is determined that the difference is equal to or greater than the predetermined distance, the image generation device 15 does not project the corresponding indications of all or part of the stationary object present at the first position P1 onto the second position A in the first image 41. If it is determined that the difference is less than the predetermined distance, the image generation device 15 projects the corresponding indications of the stationary object present at the first position P1 onto the second position A in the first image 41. This makes it possible to generate second images 42, 42a that, when used in machine learning, suppress false detection of stationary objects fixed to the road.

[0049] In the image generation method of this embodiment, the stationary objects include at least one of road markings 64, road signs 75, traffic lights 73 and 74, and landmark buildings 76. This allows for more accurate detection of stationary objects when training data consisting of the first image 41 and the second images 42 and 42a is used for machine learning.

[0050] In the image generating method of this embodiment, the image generating device 15 determines whether the position P2 of the subject captured at the second position A in the first image 41 is higher than the road surface R by a predetermined height or more, and if it determines that the position P2 of the subject captured at the second position A is higher than the road surface R by the predetermined height or more, does not project the corresponding display of all or part of the stationary object present at the first position P1 onto the second position A. This makes it possible to more accurately detect stationary objects captured (or displayed) in the first image 41.

[0051] In the image generation method of this embodiment, when generating training data consisting of the first image 41 and the second images 42, 42a, the image generation device 15 calculates a first error of the first distance D1 due to the estimation accuracy of the current position of the vehicle, and when the first error is large, sets a smaller weight to be used in machine learning for the corresponding display projected at the second position A of the first image 41 than when the first error is small. This makes it possible to improve the accuracy of detecting stationary objects.

[0052] In the image generation method of this embodiment, when generating teacher data consisting of the first image 41 and the second images 42, 42a, the image generation device 15 calculates a second error of the second distance D2 due to the measurement accuracy of the distance measuring device 12 mounted on the vehicle 50, and when the second error is large, sets a smaller weight to be used in machine learning for the corresponding display projected at the second position A of the first image 41 than when the second error is small. This makes it possible to improve the accuracy of detecting stationary objects.

[0053] In the image generating method of this embodiment, when the distance measuring device 12 mounted on the vehicle 50 has not measured the second distance D2, the image generating device 15 acquires a third distance to the subject captured near the second position A in the first image 41 from the distance measuring device 12, and estimates the second distance D2 from the third distance. This makes it possible to calculate the second distance D2 to the subject even for positions where the second distance D2 has not been measured.

[0054] In the image generating method of this embodiment, the distance measuring device 12 includes at least one of a stereo camera, a radar, and a lidar mounted on the vehicle, thereby enabling more accurate measurement of the distance to the subject.

[0055] Furthermore, according to this embodiment, there is provided a stationary object detection device 31 that performs machine learning using training data consisting of the first image 41 and the second images 42, 42a generated by the above-described image generation method, thereby enabling more accurate detection of stationary objects.

[0056] Furthermore, according to this embodiment, there is provided a driving control device 32 that causes the vehicle to drive autonomously using the detection results of the stationary object detection device 31. This makes it possible to more accurately recognize the driving range of the vehicle from the detection results of the stationary objects.

[0057] According to the present embodiment, an image generating device 15 generates second images 42, 42a by projecting corresponding displays corresponding to stationary objects registered in map information 13 within the imaging range of a first image 41 acquired from a camera 11 mounted on a vehicle 50, the second images 42, 42a being obtained by projecting corresponding displays corresponding to stationary objects registered in map information 13 within the imaging range of the camera 11, and the second images 42, 42a are obtained by projecting corresponding displays corresponding to stationary objects registered in map information 13 within the imaging range of the camera 11. The ... and a generation unit (23) that, when the determination unit (22) determines that the difference is equal to or greater than the predetermined distance, does not project the corresponding indication of all or part of the stationary object present at the first position (P1) at the second position (A) of the first image (41), and, when the determination unit (22) determines that the difference is less than the predetermined distance, projects the corresponding indication of the stationary object present at the first position (P1) at the second position (A) of the first image (41). This makes it possible to generate second images (42, 42a) that suppress erroneous detection of stationary objects fixed to a road when used in machine learning.

[0058] DESCRIPTION OF SYMBOLS 10...Image generation system 11...Camera, 12...Range measuring device, 13...Map information, 14...Self-position detection device, 15...Image generation device 21...Calculation unit, 22...Determination unit, 23...Generation unit 31...Stationary object detection device, 32...Driving control device 41...First image, 41x...Image, 42, 42a...Second image 50...Equipped vehicle, 51, 52...Other vehicles 61, 62...Lane boundary line, 63, 66, 67...Stop line, 64...Road marking, 65...Pedestrian crossing 71, 72...Curb, 73, 74...Traffic signal, 75...Road sign, 76...Building 81, 81a, 82, 82a, 83, 83a, 84, 85, 86, 86a, 86b, 87, 87a, 91, 92, 93...Corresponding display A...second position, D1...first distance, D2...second distance, H...height, P1...first position, P2, P4...position of subject, P3...mounting position of camera, Q1, Q2, Q3, Q4, Q5...focusing point, R...road surface

Claims

1. In an image generation method executed by an image generation device that generates a second image by projecting a corresponding display corresponding to a stationary object registered in map information within an imaging range of a camera onto a first image acquired from the camera mounted on a vehicle, the image generation device determines whether a difference between a first distance from the stationary object existing at a first position registered in the map information to the mounting position of the camera and a second distance from a subject of the camera imaged at a second position corresponding to the first position in the first image to the mounting position is equal to or greater than a predetermined distance. When it is determined that the difference is equal to or greater than the predetermined distance, the corresponding display of all or part of the stationary object existing at the first position is not projected onto the second position of the first image. When it is determined that the difference is less than the predetermined distance, the corresponding display of the stationary object existing at the first position is projected onto the second position of the first image.

2. The image generation method according to claim 1, wherein the stationary object includes at least one of a road surface marking, a road sign, a traffic signal, and a landmark building.

3. The image generation device determines whether a position of the subject imaged at the second position of the first image is equal to or higher than a predetermined height from a road surface. When it is determined that the position of the subject imaged at the second position is equal to or higher than the predetermined height from the road surface, the corresponding display of all or part of the stationary object existing at the first position is not projected onto the second position. The image generation method according to claim 1 or 2.

4. When generating teacher data including the first image and the second image, the image generation device calculates a first error of the first distance due to an estimation accuracy of a current position of the vehicle. When the first error is large, a weight used for machine learning for the corresponding display projected onto the second position of the first image is set to be smaller than when the first error is small. The image generation method according to any one of claims 1 to 3.

5. When generating the teacher data consisting of the first image and the second image, the image generation device calculates a second error of the second distance due to the measurement accuracy of the distance measurement device mounted on the vehicle. When the second error is large, a smaller weight for machine learning is set for the corresponding display projected at the second position of the first image than when the second error is small. The image generation method according to any one of claims 1 to 4.

6. When the distance measurement device mounted on the vehicle has not measured the second distance, the image generation device acquires a third distance from the distance measurement device to the subject shown near the second position of the first image, and estimates the second distance from the third distance. The image generation method according to any one of claims 1 to 5.

7. The distance measurement device includes at least one of a stereo camera, a radar, and a lidar mounted on the vehicle. The image generation method according to claim 5 or 6.

8. A stationary object detection device that performs machine learning using the teacher data consisting of the first image and the second image generated by the image generation method according to any one of claims 1 to 7.

9. A travel control device that autonomously travels the vehicle using the detection result of the stationary object detection device according to claim 8.

10. An image generation device that generates a second image in which a corresponding display corresponding to a stationary object registered in map information is projected onto a first image acquired from a camera mounted on a vehicle, the first distance from the stationary object existing at the first position registered in the map information to the mounting position of the camera, and a determination unit that determines whether or not a difference from a second distance from the subject of the camera shown at a second position corresponding to the first position in the first image to the mounting position is equal to or greater than a predetermined distance; a generation unit that, when the determination unit determines that the difference is equal to or greater than the predetermined distance, does not project all or part of the corresponding display of the stationary object existing at the first position at the second position of the first image, and when the determination unit determines that the difference is less than the predetermined distance, projects the corresponding display of the stationary object existing at the first position at the second position of the first image. An image generation device comprising:

Images