Simulation system, information generation program, and simulation method

Abstract

To provide a technology of a simulation system that can evaluate control of a vehicle on the assumption of a recognition error of information on an object.SOLUTION: A simulation system comprises: a LiDAR processing unit 11 that generates LiDAR information on the basis of a point group generated from reflection light generated by laser light being reflected in an object; a sensor processing unit 12 that generates sensor information on the basis of at least one of a reflection wave generated by an ultrasound wave being reflected in the object, and an image where a camera photographs a range including the object; a learning unit 14 that implemented supervised machine learning, letting the LiDAR information be an input value, and the sensor value be a correct value; a virtual generation unit 15 that generates virtual environment information serving as information on a position of an object existing within a virtual image; and a simulation unit 16 that performs a simulation for control of a vehicle, using virtual sensor information serving as information on the position of the object output from the learning unit with the virtual environment information as an input value.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a simulation system, an information generation program, and a simulation method. [Background technology]

[0002] Patent Document 1 discloses an image generation system and a simulation system. The image generation system generates a virtual image that corresponds to an image output by a near-infrared sensor or a LiDAR sensor.

[0003] The image generation system of Patent Document 1 sets a scenario that defines where objects such as roads and pedestrians are located and how they move. In the image generation system, a 3D modeling unit creates the shape of an object in a virtual space according to the scenario. The image generation system generates 3DCG based on the 3D shape generated by the 3D modeling unit. The image generation system extracts components included in the 3DCG as component images and generates a depth image in which the depth of the object in the component image is defined. The image generation system can artificially generate images similar to real-life images under conditions that are difficult to reproduce, such as severe weather conditions. The generated artificial images can be applied to AI recognition technology used in autonomous driving.

[0004] The simulation system of Patent Document 1 recognizes a specific object from an image generated by an image generation system, and generates a control signal for controlling the operation of a vehicle using the recognition result. The simulation changes or corrects the position information of the vehicle based on the control signal. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-60512 A Summary of the Invention [Problem to be solved by the invention]

[0006] In recent years, a method for evaluating vehicle control by generating information on an object using a simulation tool and inputting the information on the object to a vehicle control unit that controls the vehicle has been known. A camera and an ultrasonic sensor are often used as components of a vehicle. Therefore, the information on the object used for controlling the vehicle is often generated based on data acquired by either the camera or the ultrasonic sensor, or data generated by a combination of data acquired by the camera and the ultrasonic sensor. Therefore, in the simulation tool, it may be required to reproduce the information on the object generated based on data acquired by at least one of the camera and the ultrasonic sensor.

[0007] Here, it is known that information on an object generated based on an ultrasonic sensor or a camera contains many errors in recognition of the object's position, type, etc., compared with information on an object generated based on a sensor that irradiates laser light, such as a LiDAR sensor. However, in a conventional simulation system, information containing errors in recognition of the object's position, type, etc., is not generated. Therefore, a simulation is not performed assuming errors in recognition, and the evaluation of vehicle control is limited. The depth image of Patent Document 1 is also not generated assuming errors in recognition of the object's information. Therefore, a simulation system capable of evaluating vehicle control assuming errors in recognition of the object's information is desired. [Means for solving the problem]

[0008] The present disclosure can be realized in the following forms.

[0009] According to one embodiment of the present disclosure, a simulation system (1) is provided. This simulation system includes a LiDAR processing unit (11) that generates LiDAR information, which is information on the position of an object relative to a predetermined position, based on a point cloud generated from reflected light generated by reflecting laser light irradiated by a LiDAR sensor (LS) on the object, and a camera that captures the object such that at least a part of the detection range of the LiDAR sensor and at least a part of the detection range overlap, the reflected wave of ultrasonic waves emitted from an ultrasonic sensor (US) that emits ultrasonic waves so that at least a part of the detection range overlaps with the detection range of the LiDAR sensor, the reflected wave being reflected by the object, and the detection range of the LiDAR sensor and at least a part of the detection range overlaps with the detection range of the LiDAR sensor. a learning unit (14) that has performed supervised machine learning with the LiDAR information as an input value and the sensor information as a correct answer value; a virtual generation unit (15) that generates virtual environment information that is information on the position of an object present in a virtual image with the vehicle as a reference; and a simulation unit (16) that performs a simulation for controlling the vehicle using virtual sensor information that is information on the position of the object output from the learning unit with the virtual environment information as an input value.

[0010] It is considered that the sensor information generated based on at least one of an ultrasonic sensor and a camera is more likely to have errors in recognition of the position and type of an object, or to be undetected, compared to LiDAR information generated based on a LiDAR sensor that irradiates laser light. According to the simulation system of this embodiment, the learning unit learns by using LiDAR information that is considered to be close to the information of the actual object as an input value, and sensor information that is considered to have a large error in recognition from the information of the actual object as a correct answer value. When virtual environment information, which is information on a virtual environment, is input to the learned learning unit, the virtual environment information is converted into virtual sensor information that is information close to information on an object generated based on at least one of an ultrasonic sensor and a camera. In other words, information on an artificially generated object can be converted into information that includes errors in recognition and undetection. Furthermore, a simulation for controlling a vehicle can be performed using the generated virtual sensor information.

[0011] According to another embodiment of the present disclosure, an information generation program is provided. This information generation program causes a computer to function as a LiDAR processing unit that generates LiDAR information, which is information on the position of an object based on a predetermined position, based on a point cloud generated from reflected light generated by a laser light irradiated by a LiDAR sensor being reflected by the object, a sensor processing unit that generates sensor information, which is information on the position of the object based on at least one of a reflected wave of an ultrasonic wave emitted from an ultrasonic sensor and reflected by the object and an image captured by a camera of a range including the object, a learning unit that performs supervised machine learning using the LiDAR information as an input value and the sensor information as a correct answer value, a virtual generation unit that generates virtual environment information, which is information on the position of an object present in a virtual image based on a vehicle, and a simulation unit that performs a simulation for controlling the vehicle using virtual sensor information, which is information on the position of the object output from the learning unit using the virtual environment information as an input value.

[0012] According to another embodiment of the present disclosure, a simulation method is provided, which generates LiDAR information, which is information on the position of the object based on a predetermined position, based on a point cloud generated from reflected light generated by reflecting laser light irradiated by a LiDAR sensor on the object, generates sensor information, which is information on the position of the object based on the predetermined position, based on at least one of a point cloud generated from reflected waves of ultrasonic waves oscillated from an ultrasonic sensor that oscillates ultrasonic waves so that at least a part of the detection range of the LiDAR sensor overlaps with the object, and an image captured by a camera that captures an image of a range including the object so that at least a part of the detection range of the LiDAR sensor overlaps with at least a part of the detection range of the LiDAR sensor, performs supervised machine learning using the LiDAR information as an input value and the sensor information as a correct answer value, generates virtual environment information, which is information on the position of an object present in a virtual image based on a vehicle, inputs the virtual environment information to a learning unit that has already performed the supervised machine learning, and performs a simulation for controlling the vehicle using the virtual sensor information, which is information on the position of the object output from the learning unit. [Brief description of the drawings]

[0013] [Figure 1] 2 is a diagram for explaining a simulation system of the present embodiment, a LiDAR sensor, an ultrasonic sensor, a camera, and a vehicle control unit. [Diagram 2] A diagram explaining LiDAR point clouds. [Diagram 3] FIG. 4 is a diagram for explaining a captured image. [Figure 4] 11 is a diagram for explaining linking of LiDAR point cloud data, image data, and US distance data by a synchronization processing unit. [Diagram 5] A diagram showing a virtual image. [Figure 6] 1 is a flowchart illustrating an example of a simulation method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] A. This embodiment: A1. Configuration of this embodiment: The simulation system 1 generates information including the position and type of an object included in a virtual image. The simulation system 1 performs a simulation for controlling a vehicle using the generated information. The generated information will be described in detail later. As shown in FIG. 1, the simulation system 1 includes a synchronization processing unit 10, a LiDAR processing unit 11, a sensor processing unit 12, an information storage unit 13, a learning unit 14, a virtual generation unit 15, a simulation unit 16, an evaluation unit 17, and a storage device 18. In FIG. 1, arrows indicate that information can be transmitted and received between each component.

[0015] The synchronization processing unit 10 links the data of the point cloud generated by the LiDAR sensor LS, the data of the distance to the object acquired by the two ultrasonic sensors US, and the data of the image captured by the camera CM. In the following, the point cloud generated by the LiDAR sensor LS is referred to as the LS point cloud, and the distance acquired by the two ultrasonic sensors US is referred to as the US distance. Before describing the details of the function of the synchronization processing unit 10, the LiDAR sensor LS, the LS point cloud, the ultrasonic sensor US, the US distance, the camera CM, and the image captured by the camera CM will be described.

[0016] The LiDAR sensor LS generates an LS point cloud from the reflected light of the irradiated laser light reflected by the object. FIG. 2 shows the LS point cloud. The LS point cloud is a set of points indicating positions where at least a part of the object identified by the reflected light may exist within a range detectable by the LiDAR sensor LS. The points constituting the LS point cloud have information on distance based on the LiDAR sensor LS. The object in this specification is a vehicle, a pedestrian, a traffic light, a road, a building, etc. In this embodiment, the LiDAR sensor LS irradiates laser light at a certain time interval, and multiple LS point clouds are generated. The LiDAR sensor LS of this embodiment irradiates laser light at a predetermined interval. Each of the multiple LS point clouds is associated with the time of generation. In this embodiment, the LiDAR sensor LS is mounted on a vehicle, which is a moving body. Note that the vehicle on which the LiDAR sensor LS is mounted is different from the vehicle to be controlled, as described above.

[0017] The ultrasonic sensor US shown in FIG. 1 emits ultrasonic waves. The ultrasonic sensor US receives a reflected wave of the emitted ultrasonic waves reflected by an object, and generates US distance data, which is data of the US distance. In this embodiment, the US distance data is generated by two adjacent ultrasonic sensors. First, the two ultrasonic sensors each receive a reflected wave of the ultrasonic waves emitted from one ultrasonic sensor and reflected by the object. Then, information on the distance to the object is generated in each of the two ultrasonic sensors. The US distance data is information on the distance to the object in each of the two ultrasonic sensors. The US distance data may include information on the distance between a plurality of objects. In this embodiment, ultrasonic waves are emitted from the ultrasonic sensor US at a certain time interval, and a plurality of US distance data are generated. The ultrasonic sensor US of this embodiment emits ultrasonic waves at a predetermined interval. Each of the plurality of US distance data is associated with a time at which it was generated. Even if the irradiation of laser light by the LiDAR sensor LS and the emission of ultrasonic waves by the ultrasonic sensor US are performed at the same time, the times at which they are generated may differ. The ultrasonic sensor US emits ultrasonic waves so that its detection range overlaps with the detection range of the LiDAR sensor LS. The ultrasonic sensor US is mounted on the same vehicle as the vehicle on which the LiDAR sensor LS is mounted.

[0018] The camera CM shown in FIG. 1 captures images so that its detection range overlaps with the detection range of the LiDAR sensor LS. In this embodiment, the camera CM also captures images so that its detection range overlaps with the detection range of the ultrasonic sensor US. FIG. 3 shows an image captured so that its detection range overlaps with the detection range of the LiDAR sensor LS and the detection range of the ultrasonic sensor US. Note that FIG. 2 does not show the LiDAR point cloud reflected at the crosswalk. In this embodiment, the camera CM captures multiple images at predetermined intervals. Each of the multiple images is associated with the time when it was captured.

[0019] The LiDAR sensor LS, the ultrasonic sensor US, and the camera CM are each disposed at different positions on the same vehicle. Specifically, the LiDAR sensor LS and the camera CM are disposed one by one under the front emblem of the vehicle. The ultrasonic sensor is disposed on the front bumper of the vehicle.

[0020] A worker inputs a plurality of LS point cloud data, a plurality of US distance data, and a plurality of image data into the synchronization processing unit 10. The synchronization processing unit 10 links the LS point cloud data, the US distance data, and the image data that were generated close to each other among the plurality of LS point cloud data, the plurality of US distance data, and the plurality of image data captured by the camera CM.

[0021] The synchronization processing unit 10 links two pieces of data that have been generated the closest to each other among a plurality of LS point group data, a plurality of US distance data, and a plurality of image data. First, the synchronization processing unit 10 links the LS point group data generated at a certain time to the US distance data generated the closest to the time. In the following, a case where the LS point group data A1 generated at a certain time shown in FIG. 4 is linked to the US distance data B1 generated at a different time will be described. As shown in FIG. 4, the synchronization processing unit 10 links the LS point group data A1 to the US distance data B1 generated the closest to the time when the LS point group data A1 was generated. The synchronization processing unit 10 links each of the LS point group data A1 and the US distance data B1 that are linked to each other, and the image data C1 captured the closest to the time when the LS point group data A1 was generated to the LS point group data A1 and the US distance data B1. In this embodiment, the data of the LS point cloud shown in FIG. 2 is data A1 in FIG. 4, the US distance data is data B1 in FIG. 4, and the data of the image shown in FIG. 3 is data C1 in FIG.

[0022] Similarly, the synchronization processing unit 10 links other LS point group data, US distance data, and image data. For example, the synchronization processing unit 10 links the LS point group data A2 and the US distance data B2 generated at the time closest to the time when the LS point group data A2 was generated. Then, the synchronization processing unit 10 links each of the mutually linked LS point group data A2 and US distance data B2 with the image data C1 captured at the time closest to the time when the LS point group data A2 and US distance data B2 were generated. In this way, the same image data C1 may be linked to different LS point group data and US distance data.

[0023] In addition, the image captured at the time closest to the time when the LS point cloud was generated may be different from the image captured at the time closest to the time when the US distance data was generated. In this case, the data of the image captured at the shorter time between the time when the LS point cloud was generated and the time when the most recent image was captured, or between the time when the US distance data was generated and the time when the most recent image was captured, is linked to the LS point cloud data and the US distance data. If the time between the time when the LS point cloud was generated and the time when the most recent image was captured is the same as the time between the time when the US distance data was generated and the time when the most recent image was captured, each of the data of the two images captured at different times is linked to the LS point cloud data and the US distance data.

[0024] A combination of LS point cloud data, US distance data, and image data that are linked to each other is referred to as a "synchronization group." The synchronization processing unit 10 shown in FIG. 1 assigns numbers to the synchronization groups to distinguish each synchronization group. The numbers indicate the order in which the LS point clouds were acquired. For example, as shown in FIG. 4, the number of the synchronization group to which the LS point cloud data A1 generated at the earliest time after the start of detection by the LiDAR sensor LS, the ultrasonic sensor US, and the camera CM, the US distance data B1, and the image data C1 are linked is 1. In this embodiment, multiple synchronization groups are created. The synchronization processing unit 10 in FIG. 1 transmits each of the multiple LS point cloud data to the LiDAR processing unit 11 together with information on the synchronization group to which it belongs. The synchronization processing unit 10 also transmits each of the multiple US distance data and the multiple image data to the sensor processing unit 12 together with information on the synchronization group to which it belongs.

[0025] The LiDAR processing unit 11 generates LiDAR information, which is information on the position of an object based on a predetermined position, based on a point cloud generated from the reflected light reflected by the object among the LS point clouds. Hereinafter, the point cloud generated from the reflected light reflected by the object is referred to as a LiDAR point cloud. In this embodiment, the predetermined position refers to the center of the rear wheel axle of the vehicle. In this embodiment, the LiDAR processing unit 11 generates multiple pieces of LiDAR information based on the LS point clouds belonging to each of multiple synchronization groups.

[0026] The generation of LiDAR information based on the LS point cloud shown in FIG. 2 will be described. First, the LiDAR processing unit 11 clusters the LS point cloud to generate one or more groups. The clustering is performed by an algorithm for clustering a point cloud. As shown in FIG. 2, a group 1 and a group 2 are generated. The LiDAR processing unit 11 recognizes one group as being generated based on a point cloud generated from reflected light generated by a laser light being reflected by an object. In this embodiment, as shown in FIG. 2, the LiDAR processing unit 11 surrounds the clustered LiDAR point cloud with a three-dimensional rectangular box and assigns numbers to each of the multiple groups to distinguish the multiple groups. As shown in FIG. 2, the LS point cloud is divided into a LiDAR point cloud belonging to group 1 and a LiDAR point cloud belonging to group 2. The LiDAR processing unit 11 determines a representative point in each group. Information on the position of the representative point is LiDAR information, which is information on the position of the object based on a predetermined position.

[0027] The LiDAR information is expressed by an orthogonal coordinate system based on a predetermined position. In this embodiment, the right direction based on the predetermined position is the positive direction of the X axis of the orthogonal coordinate system, the depth direction based on the predetermined position is the positive direction of the Y axis, and the height direction based on the predetermined position is the Z axis direction. The depth direction is the forward direction of the vehicle on which the LiDAR sensor LS is mounted.

[0028] In this embodiment, the LiDAR processing unit 11 further generates information on the height, width, and type of the group, and the reliability of each piece of information. The information on the height and width of the group is generated using an algorithm that calculates the height and width of the group. The type of the group is determined by an algorithm that determines the type of the group. In this embodiment, the type of group 1 shown in FIG. 2 is determined to be "vehicle", and the type of group 2 is determined to be "pedestrian". The reliability is generated by an algorithm that calculates the reliability of the generated information. The information generated by the LiDAR processing unit 11, including the LiDAR information, is linked to the number of the synchronization group to which it belongs.

[0029] The sensor processing unit 12 shown in Fig. 1 generates sensor information, which is information on the position of an object based on a predetermined position, based on the US distance data and an image captured by the camera CM. In this embodiment, the sensor processing unit 12 generates sensor information, which is information on the position of an object based on a predetermined position, based on the US distance data belonging to the same synchronization group and an image captured by the camera CM. The sensor processing unit 12 generates sensor information for each of multiple synchronization groups.

[0030] The generation of sensor information based on the image shown in FIG. 3 will be described. First, the sensor processing unit 12 calculates the position of the object based on a predetermined position based on the US distance data. Specifically, the sensor processing unit 12 calculates the position of the object by the trilateration principle from the distances between the two ultrasonic sensors and the object and the distance and positional relationship between the two ultrasonic sensors US, which are included in the US distance data. In addition, the sensor processing unit 12 detects the object in the image using an image recognition algorithm. In this embodiment, two objects, OB1 and OB2, are detected as shown in FIG. 3.

[0031] The sensor processing unit 12 uses an algorithm to determine whether the object whose position has been calculated and the object detected in the image are the same object. If it is determined that they are the same object, the sensor processing unit integrates the object whose position has been calculated and the object detected in the image using an algorithm for integrating the object whose position has been calculated and the object in the image. In this way, the sensor processing unit 12 generates sensor information, which is information on the position of the object. The sensor information is based on a predetermined position.

[0032] In this embodiment, the sensor processing unit 12 further generates information on the height, width, and type of the object, and the reliability of each piece of information. The information on the height and width of the object is generated using an algorithm that calculates the height and width of the object. The type of the object is determined by an algorithm that determines the type of the object. In this embodiment, the type of the object based on OB1 in FIG. 3 is determined to be "vehicle", and the type of the object based on OB2 in FIG. 3 is determined to be "pedestrian". The reliability is generated by an algorithm that calculates the reliability of the generated information. The information generated by the sensor processing unit 12, including the sensor information, is linked to the number of the synchronization group to which it belongs.

[0033] The information storage unit 13 shown in FIG. 1 stores LiDAR information and sensor information. In this embodiment, the information storage unit 13 further stores the height, width, type information, and reliability of the group generated by the LiDAR processing unit 11. The information storage unit 13 also stores the height, width, type information, and reliability of the object generated by the sensor processing unit 12. In this embodiment, the information storage unit 13 stores information including LiDAR information generated based on the LS point cloud, US distance data, and image belonging to the same synchronization group, and information including sensor information in association with each other.

[0034] The learning unit 14 is a trained model that has executed supervised machine learning with LiDAR information as an input value and sensor information as a correct value. The correct value is also called teacher data. In this embodiment, the learning unit 14 has executed supervised deep neural network learning. The learning unit 14 learns so that the output value approaches the sensor information that is the correct value. In this embodiment, the learning unit 14 has executed learning with sensor information that belongs to the same synchronization group as the LiDAR information of the input value as the correct value. The information that is the input value, the output value that is output, and the information that is the correct value are the same type of information. The same type of information means that, for example, when the input value is information that represents a type, the correct value is also information that represents a type.

[0035] In this embodiment, the learning unit 14 has already learned by inputting an input value and a correct answer value by an operator. LiDAR information and sensor information about the same object are input to the learning unit 14. For example, the LiDAR information of group 1 in FIG. 2 belonging to synchronization group 1 in FIG. 4 is input to the learning unit 14 as an input value, and sensor information of the object based on OB1 in FIG. 3 is input as a correct answer value. In this embodiment, the learning unit 14 has already learned by inputting information generated by the LiDAR processing unit 11 and information generated by the sensor processing unit 12 about the same object.

[0036] The learning unit 14 that has already been trained uses virtual environment information as an input value and outputs virtual sensor information that is information on the position of an object in a virtual image. The virtual environment information and the virtual image will be explained later. In addition, in Fig. 1, for ease of understanding, the learning unit 14 that is currently learning is represented by a dashed frame, and the learning unit 14 that has already been trained is represented by a solid frame.

[0037] The virtual generation unit 15 generates a virtual image representing a virtual environment. The virtual image represents a virtual environment based on a vehicle. The vehicle referred to here is a vehicle controlled by a vehicle control unit VC shown in FIG. 1, which is an object to be evaluated by the simulation system 1 of the present disclosure. The vehicle control unit VC executes control of the vehicle. Specifically, the vehicle control unit VC controls steering, driving, and braking of the vehicle based on input information including virtual sensor information to be described later.

[0038] The virtual image includes one or more objects, including a road on which the vehicle can travel, a parking lot, other vehicles, and pedestrians. The virtual generation unit 15 first creates a scenario that assumes the position and movement of the object. The virtual generation unit 15 creates a three-dimensional shape of the object based on the scenario, and generates a virtual image in which the depth of the object is set. In this embodiment, a plurality of virtual images are prepared. In this embodiment, the plurality of virtual images includes an image in which parking in a parking lot is assumed, as shown in FIG. 5. The virtual image in which parking in a parking lot is assumed includes a parking lot where a plurality of other vehicles are parked, and pedestrians moving through the parking lot.

[0039] The virtual generation unit 15 generates virtual environment information, which is information on the position of an object present in a virtual image based on the vehicle. The virtual environment information is generated by an algorithm that generates position information from the three-dimensional shape of the object generated by the virtual generation unit 15. In this embodiment, the virtual generation unit 15 further generates information on the height, width, and type of the object present in the virtual image. The information generated by the virtual generation unit 15 has the same format as the information generated by the LiDAR processing unit 11. Specifically, the virtual environment information is a position expressed in orthogonal coordinates based on the vehicle. In addition, the type of the object present in the virtual image is set to correspond to the information generated by the LiDAR processing unit 11, such as "vehicle" for another vehicle and "pedestrian" for a person. The information generated by the virtual generation unit 15 is input to the trained learning unit 14. The trained learning unit 14 outputs the above-mentioned virtual sensor information.

[0040] In this embodiment, an image including a parking lot and pedestrians moving through the parking lot shown in Fig. 5 is configured so that the distance to the pedestrian relative to the vehicle, which is included in the virtual sensor information, is within a range of 5 cm to 10 m. Also, in this embodiment, information generated based on virtual environment information including virtual sensor information is generated assuming three disturbances: wind at or above a predetermined speed per second, precipitation at or above a predetermined value, and a road surface that is at least either uneven or inclined.

[0041] The simulation unit 16 shown in Fig. 1 performs a simulation for controlling the vehicle using information output by the learning unit 14, including the virtual sensor information. The simulation unit 16 inputs the information output by the learning unit 14, including the virtual sensor information, to the vehicle control unit VC. Then, the vehicle control unit VC generates a signal for controlling the vehicle based on the input information. The signal is, for example, a signal for stopping the vehicle, a signal for rotating the steering device 60° to the right, etc.

[0042] The evaluation unit 17 evaluates the vehicle control unit VC based on the result of the control simulation based on the information output by the learning unit 14. The result of the control simulation is a signal generated by the vehicle control unit VC. As described above, in this embodiment, the vehicle control unit VC performs steering control, driving control, and braking control of the vehicle based on the input information, so the evaluation unit 17 controls the vehicle control unit VC based on the results of the steering control, driving control, and braking control of the vehicle.

[0043] An example of evaluation by the evaluation unit 17 will be described. As a premise, a signal serving as an evaluation criterion is input in advance by an operator to the evaluation unit 17 for each piece of information output by the learning unit 14. The evaluation unit 17 evaluates the signal based on the evaluation criterion. For example, a control signal output by the vehicle control unit VC to which virtual sensor information that the vehicle is 1 m in the X-axis direction, 50 cm in the Y-axis direction, and 1 m in the Z-axis direction is input, and information that the type of object is "pedestrian", is compared with the signal serving as the evaluation criterion.

[0044] The storage device 18 stores virtual environment information, virtual sensor information, the results of control by the vehicle control unit VC, and an evaluation of the results.

[0045] A2.Simulation method: 6, a plurality of LS point cloud data, a plurality of US distance data, and a plurality of image data captured by the camera CM are input to the synchronization processing unit 10 by an operator in the simulation system 1. In step S20, the synchronization processing unit 10 links the LS point cloud data, the US distance data, and the image data. The synchronization processing unit 10 creates a plurality of synchronization groups.

[0046] In step S30, the LiDAR processing unit 11 generates information including LiDAR information, which is information on the position of a group based on a predetermined position, based on the LiDAR point cloud. In step S40, the sensor processing unit 12 generates information including US distance data and sensor information, which is information on the position of an object based on a predetermined position, based on the image. Note that the process of step S40 may be performed before the process of step S30.

[0047] In step S50, the information storage unit 13 stores the LiDAR information and the sensor information in association with the synchronization group to which they belong. In step S60, the pre-learning learning unit 14 executes supervised deep neural network learning using information including LiDAR information belonging to the same synchronization group as an input value and information including sensor information as a correct answer value. In step S70, the virtual generation unit 15 generates information including virtual environment information. Note that the process of step S70 may be executed before the process of step S60.

[0048] In step S80, the virtual generation unit 15 inputs information including virtual environment information to the learning unit 14 that has already performed supervised deep neural network learning. In step S90, the simulation unit 16 inputs information including the virtual sensor information output from the learning unit 14 to the vehicle control unit VC of the vehicle to be evaluated in the simulation system 1, thereby performing a simulation for controlling the vehicle. In step S100, the evaluation unit 17 evaluates the vehicle control unit VC based on the result of the simulation. In step S110, the storage device 18 stores the virtual sensor information, the result of control of the vehicle control unit VC, and the evaluation. Thereafter, the process ends.

[0049] It is considered that the sensor information generated based on at least one of the ultrasonic sensor US and the camera CM is more likely to have errors in recognition of the position and type of the object, or to be undetected, compared to the LiDAR information generated based on the LiDAR sensor LS that emits laser light. According to the simulation system 1 of this embodiment, the learning unit 14 learns by using LiDAR information that is considered to be close to the information of the actual object as an input value, and sensor information that is considered to have a large error in recognition from the information of the actual object as a correct answer value. When virtual environment information, which is information on a virtual environment, is input to the learned learning unit 14, the virtual environment information is converted into virtual sensor information that is information close to information on an object generated based on at least one of the ultrasonic sensor US and the camera CM. In other words, it is possible to convert information on an artificially generated object into information that includes errors in recognition and undetection. Furthermore, a simulation for controlling the vehicle can be performed using the generated virtual sensor information. In addition, it is possible to evaluate the vehicle control unit VC on the assumption of errors in recognition and undetection of information on the position of the object derived from at least one of the ultrasonic sensor US and the camera CM.

[0050] Furthermore, compared to a mode in which information on the height, width, and type of object based on a predetermined position of the object, and the reliability of each piece of information are not generated, the amount of LiDAR information and the amount of sensor information as correct answer values ​​input to the learning unit 14 are large, and therefore the simulation is executed using the learning unit 14 with a large amount of learning. Also, by storing one or more of the virtual sensor information, the control result, and the evaluation, the results of the simulation can be accumulated.

[0051] In a parking lot where many vehicles move, it may be difficult to see the movements of pedestrians. According to the simulation system 1 of this embodiment, in a situation where a pedestrian is present within a close range of 30 cm or more and 1 m from the vehicle, it is possible to carry out an experiment on vehicle control under the assumption of erroneous detection of the pedestrian's position or non-detection of the pedestrian. In addition, compared to a situation where no disturbance is assumed, the sensor information is more likely to include errors or missing input, making it possible to perform a simulation in such a situation.

[0052] B. Other embodiments: B1. Alternative embodiment 1: (1) In the above embodiment, the sensor processing unit 12 generates sensor information based on both the reflected waves of ultrasonic waves reflected by the object and the image. The sensor processing unit may generate sensor information based on at least one of the reflected waves of ultrasonic waves reflected by the object and the image. For example, the sensor processing unit may generate sensor information based only on the reflected waves of ultrasonic waves reflected by the object. In this aspect, the synchronization processing unit links the data of the LS point cloud with the US distance data generated at the time closest to the time when the LS point cloud was generated. The sensor processing unit may also generate sensor information based only on the image. In this aspect, the synchronization processing unit links the data of the LS point cloud with the data of the image generated at the time closest to the time when the LS point cloud was generated.

[0053] Generating sensor information based on at least one of the reflected waves of ultrasonic waves reflected by an object and an image includes cases where either one of the US distance data and the image is not generated, and the sensor information is generated based on the other. For example, in a mode in which the US distance data does not include an object detected in the image, in the US distance data and the image belonging to a synchronization group, the sensor processing unit generates sensor information based only on the image. In a mode in which the image captured by the camera does not include an object included in the US distance data, the sensor processing unit generates sensor information based only on the ultrasonic point cloud.

[0054] For example, the moving body on which the LiDAR sensor is mounted may not be equipped with a camera. In this embodiment, the sensor processing unit generates sensor information based on a reflected wave of an ultrasonic wave emitted from an ultrasonic sensor that emits ultrasonic waves so that at least a part of the detection range of the LiDAR sensor overlaps with the target. Also, in an embodiment in which the ultrasonic sensor is not mounted on the moving body on which the LiDAR sensor is mounted, the sensor processing unit generates sensor information based on an image captured by a camera that captures an image in a range where at least a part of the detection range of the LiDAR sensor overlaps with at least a part of the detection range of the LiDAR sensor, including the target.

[0055] (2) In the above embodiment, each of the multiple LS point clouds is linked to the time when it was generated, and each of the multiple US distance data is linked to the time when it was generated. Note that, for example, each of the multiple LS point clouds may be linked to the time when the LiDAR sensor received the reflected light, and each of the multiple US distance data may be linked to the time when the ultrasonic sensor received the reflected wave.

[0056] (3) In the above embodiment, the simulation environment is one in which the LiDAR sensor LS, the ultrasonic sensor US, and the camera CM are mounted on a vehicle, which is a moving body. Note that, for example, the LiDAR sensor, the ultrasonic sensor, and the camera may be mounted on a moving body other than a vehicle, such as a motorcycle or a light vehicle. Also, for example, the LiDAR sensor, the ultrasonic sensor, and the camera may be installed on a sidewalk.

[0057] (4) In the above embodiment, the vehicle, which is a moving body, is equipped with one LiDAR sensor LS and one camera CM, and the vehicle is equipped with two ultrasonic sensors US. For example, a vehicle may be equipped with a plurality of LiDAR sensors, ultrasonic sensors, and cameras. For example, a LiDAR sensor may be mounted on each of the front and rear of the vehicle, two ultrasonic sensors may be mounted on each of the front, rear, left and right of the vehicle, and a camera may be mounted on each of the front and rear of the vehicle. In this embodiment, the two ultrasonic sensors emit ultrasonic waves so that at least a part of the detection range of at least one LiDAR sensor overlaps with at least a part of the detection range of at least one LiDAR sensor, and the camera captures an image in a range where at least a part of the detection range of at least one LiDAR sensor overlaps with at least a part of the detection range of at least one LiDAR sensor. The synchronization processing unit may link data generated or captured close to each other among the data of the LiDAR point cloud generated by all LiDAR sensors, the data of the ultrasonic point cloud generated by all ultrasonic sensors, and the data of the image captured by all cameras.

[0058] (5) For example, in a simulation environment in which LiDAR information and sensor information are generated by a plurality of LiDAR sensors, a plurality of ultrasonic sensors, and a plurality of cameras, data from a LiDAR sensor, an ultrasonic sensor, and a camera that are disposed in close proximity to each other may be linked together. For example, data from a LiDAR sensor, an ultrasonic sensor, and a camera that are disposed in the front of a vehicle are linked together, and data from a LiDAR sensor, an ultrasonic sensor, and a camera that are disposed in the rear of the vehicle are linked together.

[0059] (6) In the above embodiment, the predetermined position is the center of one of the rear wheels of the vehicle. Note that, for example, the predetermined position may be the position of a LIDAR sensor, an ultrasonic sensor, or a camera, or may be any position of a moving body on which a LiDAR sensor or the like is mounted. The LiDAR information and the sensor information are generated based on the predetermined position.

[0060] (7) In the above embodiment, the LIDAR sensor, the ultrasonic sensor, and the camera all detect the same object. For example, in a mode in which a group of other vehicles, a group of pedestrians, and a group of brick buildings are generated based on the LS point cloud, a group of buildings may not be generated based on US distance data, and buildings may not be included in the objects detected based on the image. In this mode, the learning unit may receive information about the building, such as its position and type, as input values, and may receive "none" as the correct value of the sensor information. In general, it is known that ultrasonic waves are not reflected by many substances compared to laser light. For example, when the camera does not capture an object and the ultrasonic sensor cannot detect it, the learning unit learns that the correct value is "none". When virtual environment information is input, the learning unit outputs virtual sensor information as none, making it possible to perform a simulation assuming that the object is not detected.

[0061] (8) In the above embodiment, the ultrasonic sensor US emits ultrasonic waves so that the detection range overlaps with the detection range of the LiDAR sensor LS. The ultrasonic sensor may emit ultrasonic waves so that the detection range overlaps with a part of the detection range of the LiDAR sensor. In the above embodiment, the camera CM captures an image so that the detection range overlaps with the detection range of the LiDAR sensor LS, and further captures an image so that the detection range overlaps with the detection range of the ultrasonic sensor US. The camera may capture an image of an object so that the detection range of the LiDAR sensor overlaps with a part of the detection range of the ultrasonic sensor. The camera may emit ultrasonic waves so that the entire detection range overlaps with the detection range of the LiDAR sensor, and capture an image of an object so that the detection range of the ultrasonic sensor US overlaps with a part of the detection range. The camera may be configured so that the detection range of the LiDAR sensor overlaps with a part of the detection range, and the detection range of the object overlaps with a part of the detection range of the ultrasonic sensor US. The camera may be configured so that the detection range of the LiDAR sensor overlaps with a part of the detection range, and the detection range of the object overlaps with a part of the detection range of the ultrasonic sensor US.

[0062] (9) In the above embodiment, the LiDAR information and the sensor information are expressed in Cartesian coordinates with a predetermined position as a reference. Note that the position of the point may be expressed in polar coordinates with a predetermined position as a reference.

[0063] (10) In the above embodiment, the LiDAR information may be information on the position of the representative point. For example, the LiDAR information may be information on a point that is the smallest distance from a predetermined position among the points that constitute the group.

[0064] (11) For example, in a case where the sensor information is generated only from an image, the sensor processing unit may generate information on the position of an object in the image using an algorithm that estimates the position of a detected object.

[0065] (12) In the above embodiment, the learning unit 14 has been trained by inputting an input value and a correct answer value by an operator. For example, the learning unit may have been trained by inputting information by an input unit that reads the same type of information, LiDAR information and sensor information, belonging to the same synchronization group stored in the information storage unit from the information storage unit. More specifically, the input unit selects the most correlated group and object from among a plurality of groups based on the LS point cloud and a plurality of objects based on the US distance data and images, belonging to the same synchronization group, and sets the LiDAR information of the group as an input value and the sensor information of the object as a correct answer value.

[0066] (13) In the above embodiment, the virtual image includes a parking lot where multiple other vehicles are parked and pedestrians moving in the parking lot, and is configured so that the distance to the pedestrians based on the vehicle included in the virtual sensor information generated by the virtual generation unit 15 is within a range of 5 cm or more and 10 m or less. Note that the virtual image may be an image that does not include the parking lot where multiple other vehicles are parked and the pedestrians moving in the parking lot. For example, the virtual image may be an image that includes the parking lot where multiple other vehicles are parked and does not include pedestrians, or may be an image that includes pedestrians walking on a crosswalk. Also, for example, the virtual image may be configured so that the distance to the pedestrians based on the vehicle included in the virtual sensor information generated by the virtual generation unit is 10 m or more.

[0067] (14) The vehicle controlled by the vehicle control unit may be a computer program that generates signals for controlling the steering, driving, braking, etc. of the vehicle.

[0068] (15) In the above embodiment, in step S80 of Fig. 6, the virtual generation unit 15 inputs the virtual environment information to the learning unit 14 that has been trained in supervised deep neural network learning. Note that the virtual environment information may be input by, for example, an operator to the learning unit 14 that has been trained in supervised deep neural network learning.

[0069] B2. Alternative embodiment 2: In the above embodiment, the supervised machine learning is supervised deep neural network learning. For example, the supervised machine learning may be other than supervised deep neural network learning, such as supervised convolutional neural network learning or supervised recurrent neural network learning.

[0070] B3. Alternative embodiment 3: (1) In the above embodiment, the evaluation unit 17 evaluates the vehicle control unit VC based on the result of a simulation of the control of the vehicle control unit VC based on the virtual sensor information. Note that, for example, in a mode in which an operator evaluates the vehicle control unit based on the result of the simulation, the simulation system does not need to include an evaluation unit.

[0071] (2) The evaluation unit may perform the evaluation using a score. For example, the evaluation using a score may be performed according to the time required for the vehicle control unit to generate a signal. Furthermore, the evaluation by the evaluation unit may be performed only using a score.

[0072] B4. Alternative embodiment 4: In the above embodiment, the evaluation unit 17 evaluates the vehicle control unit VC based on the results of the vehicle steering control, drive control, and braking control of the vehicle by the vehicle control unit VC. The evaluation unit may evaluate the vehicle control unit based on the results of one or two of the vehicle steering control, drive control, and braking control.

[0073] B5. Alternative embodiment 5: (1) In the above embodiment, the LiDAR processing unit 11 and the sensor processing unit 12 generate information on the height, width, and type of the object, and the reliability of each piece of information. The LiDAR processing unit and the sensor processing unit may generate one or more pieces of information on the height, width, and type of the object, and the reliability of each piece of information. The LiDAR processing unit and the sensor processing unit may further generate information on the depth, speed, and direction of movement of the object.

[0074] (2) For example, the LiDAR processing unit and the sensor processing unit do not need to generate information on the height, width, and type of the object and the reliability of each piece of information.

[0075] B6. Alternative embodiment 6: (1) In the above embodiment, the storage device 18 stores virtual sensor information, a result of control by the vehicle control unit VC, and an evaluation. The storage device may store one or two of the virtual sensor information, the result of control by the vehicle control unit VC, and the evaluation.

[0076] (2) For example, the simulation system does not need to include a storage device. In this aspect, an operator may store, in an external memory, one or more of the virtual sensor information, the result of the control by the vehicle control unit, and the evaluation.

[0077] B7. Alternative Embodiment 7: In the above embodiment, the virtual sensor information is generated assuming the following disturbances: wind at a speed equal to or faster than a predetermined speed per second, precipitation equal to or faster than a predetermined value, and a road surface having at least one of unevenness and inclination. The virtual sensor information may be generated assuming one or two disturbances among the following: wind at a speed equal to or faster than a predetermined speed per second, precipitation equal to or faster than a predetermined value, and a road surface having at least one of unevenness and inclination. The virtual sensor information may also be generated without assuming any disturbance.

[0078] B8. Alternative Embodiment 8: In the above embodiment, the simulation system executes a simulation for controlling the vehicle, but the simulation may be executed by an information generation program stored in the computer. This information generation program causes the computer to function as a LiDAR processing unit that generates LiDAR information, which is information on the position of an object based on a predetermined position, based on a point cloud generated from reflected light generated by the reflection of laser light irradiated by the LiDAR sensor on the object, a sensor processing unit that generates sensor information, which is information on the position of an object based on a predetermined position, based on at least one of a reflected wave of ultrasonic waves oscillated from an ultrasonic sensor and reflected on the object and an image captured by a camera of a range including the object, a learning unit that performs supervised machine learning using the LiDAR information as an input value and the sensor information as a correct answer value, a virtual generation unit that generates virtual environment information, which is information on the position of an object present in a virtual image based on the vehicle, and a simulation unit that performs a simulation for controlling the vehicle using virtual sensor information, which is information on the position of an object output from the learning unit using the virtual environment information as an input value.

[0079] The present disclosure is not limited to the above-mentioned embodiments and modifications, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments and modifications corresponding to the technical features in each aspect described in the Summary of the Invention column can be appropriately replaced or combined in order to solve some or all of the above-mentioned problems or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted. [Explanation of symbols]

[0080] 1...simulation system, 11...LiDAR processing unit, 12...sensor processing unit, 14...learning unit, 15...virtual generation unit, 16...simulation unit, CM...camera, LS...LiDAR sensor, US...ultrasonic sensor

Claims

1. A simulation system (1), comprising: A LiDAR processing unit (11) that generates LiDAR information, which is information on the position of an object based on a predetermined position, based on a point cloud generated from reflected light generated when a laser light irradiated by a LiDAR sensor (LS) is reflected by the object; a sensor processing unit (12) that generates sensor information, which is information on the position of the object based on the predetermined position, based on at least one of: a reflected wave of ultrasonic waves reflected from the object by an ultrasonic sensor (US) that emits ultrasonic waves so that at least a portion of the detection range of the LiDAR sensor overlaps with the detection range of the LiDAR sensor; and an image of a range including the object captured by a camera (Cm) that captures an image so that at least a portion of the detection range of the LiDAR sensor overlaps with the detection range of the LiDAR sensor; A learning unit (14) that has performed supervised machine learning using the LiDAR information as an input value and the sensor information as a correct answer value; a virtual generation unit (15) that generates virtual environment information, which is information on the positions of objects present in a virtual image relative to a vehicle; a simulation unit (16) that performs a simulation for controlling the vehicle using virtual sensor information, which is information on the position of the object output from the learning unit, using the virtual environment information as an input value.

2. 2. The simulation system according to claim 1, The simulation system, wherein the supervised machine learning is supervised deep neural network learning.

3. 2. The simulation system according to claim 1, A simulation system that evaluates a vehicle control unit that executes control of the vehicle based on a result of a simulation of the control based on the virtual sensor information.

4. The simulation system according to claim 3, A simulation system that evaluates the vehicle control unit based on one or more results of the vehicle's steering control, driving control, and braking control.

5. 2. The simulation system according to claim 1, The LiDAR processing unit and the sensor processing unit further generate one or more of the following information: the height, width, and type of the object, and the reliability of each piece of information.

6. The simulation system according to claim 3 or 4, A simulation system comprising: a storage device (18) that stores at least one of the virtual sensor information, the result of the control of the vehicle control unit, and the evaluation.

7. 2. The simulation system according to claim 1, The virtual sensor information includes: A simulation system generated by assuming one or more of the following disturbances: wind speeds equal to or greater than a predetermined speed per second, precipitation equal to or greater than a predetermined value, and a road surface having at least one of unevenness or a slope.

8. An information generation program, comprising: A LiDAR processing unit that generates LiDAR information, which is information on the position of the object based on a predetermined position, based on a point cloud generated from reflected light generated when a laser light irradiated by a LiDAR sensor is reflected by the object; a sensor processing unit that generates sensor information, which is information on the position of the object based on the predetermined position, based on at least one of a reflected wave of an ultrasonic wave emitted from an ultrasonic sensor and reflected by the object and an image captured by a camera of an area including the object; and A learning unit that performs supervised machine learning using the LiDAR information as an input value and the sensor information as a correct answer value; a virtual generation unit that generates virtual environment information, which is information on the positions of objects present in a virtual image relative to a vehicle; An information generation program that functions as a simulation unit that performs a simulation for controlling the vehicle using virtual sensor information, which is information on the position of the object output from the learning unit, as an input value for the virtual environment information.

9. 1. A simulation method, comprising: Generate LiDAR information, which is information on the position of the object based on a predetermined position, based on a point cloud generated from reflected light generated when the laser light irradiated by the LiDAR sensor is reflected by the object; Generate sensor information, which is information on the position of the object based on the predetermined position, based on at least one of: a reflected wave of ultrasonic waves emitted from an ultrasonic sensor that emits ultrasonic waves so that at least a portion of the detection range of the LiDAR sensor overlaps with the detection range of the LiDAR sensor, and a camera that captures an image of a range including the object so that at least a portion of the detection range of the LiDAR sensor overlaps with the detection range of the LiDAR sensor; Executing supervised machine learning using the LiDAR information as an input value and the sensor information as a correct answer value; Generate virtual environment information, which is information on the positions of objects present in a virtual image relative to the vehicle; inputting the virtual environment information into a learning unit that has already performed the supervised machine learning; A simulation method comprising: performing a simulation for controlling the vehicle using virtual sensor information, which is information on the position of the object output from the learning unit.

Images