Information processing device, information processing method, and program

Abstract

The present invention realizes a device and a method for performing analysis by synchronizing a plurality of sensor detection data items having different frame rates and inputting the same to an analysis part. A data processing part: inputs a camera frame that is an image generated by a camera at a frame rate P (fps) and a point cloud frame that has point cloud data indicating an object position generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps); compares a time stamp of a frame unit in the camera frame with a time stamp of a component unit in the point cloud frame; synthesizes a plurality of point cloud frames that are continuously inputted; and generates a composite point cloud frame that is synchronized with the frame rate of the camera frame.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program, and more particularly to an information processing device, an information processing method, and a program that perform analysis and synthesis (fusion processing) of information detected by multiple different sensors, such as a camera and a laser sensor mounted on a vehicle.

[0002] Recently, technological development related to automated driving has progressed rapidly, such as the so-called automated driving system (ADS) and advanced driver assistance system (ADAS).

[0003] In autonomous driving, cameras and laser-based sensors installed in vehicles (automobiles) are used to detect obstacles and to achieve safe driving using the detected information. Cameras and other sensors detect various objects around the vehicle, such as oncoming vehicles, pedestrians, and guardrails, and the vehicle's autonomous driving execution unit performs autonomous driving while calculating a safe driving route that avoids collisions with these objects.

[0004] An autonomous vehicle is equipped with a plurality of different sensors, such as cameras and LiDAR (Laser Imaging Detection and Ranging), and analyzes the situation around the vehicle using information obtained by analyzing the information detected by these different sensors and by combining (fusion) the information detected by the plurality of sensors. LiDAR is a sensor that uses laser light to analyze the distance and three-dimensional position of various objects around the vehicle, such as other vehicles, such as oncoming vehicles, pedestrians, and other structures.

[0005] The LiDAR sequentially generates a point cloud frame consisting of point cloud information that expresses, for example, object distance and three-dimensional position using pixel-by-pixel shading information, at each LiDAR detection timing.

[0006] However, the frame rate of the camera frame, which is the image frame captured by the camera, and the frame rate of the point cloud frame generated by the LiDAR may be set differently. In this case, the sensor detection information analysis unit must perform analysis processing of the camera frame and point cloud frame input from these two sensors, i.e., the camera and LiDAR, at different times, which increases the processing load. Furthermore, it becomes difficult to perform synthesis processing (fusion processing), such as matching processing of the detection information from the camera and LiDAR.

[0007] As a result, processing delays and errors are more likely to occur, which could reduce the safety of autonomous driving.

[0008] In addition, Patent Document 1 (JP 2021-515254 A) discloses a configuration in which a frame-by-frame timestamp is set for each camera frame and point cloud frame input from a camera and LiDAR, and camera frames and point cloud frames with similar timestamps are selected to perform matching processing.

[0009] However, this document only discloses a matching process that uses frame-by-frame timestamps. Even if the configuration described in this document is applied, the input frames from the camera and LiDAR are asynchronous, so the analysis process for each frame must be performed at different times. In other words, it does not solve the problem of increased analysis load of sensor detection information.

[0010] Special Publication No. 2021-515254

[0011] The present disclosure aims to solve, for example, the above-mentioned problems, and to provide an information processing device, an information processing method, and a program that reduce the processing load of analyzing sensor detection information from different sensors attached to a vehicle, for example, sensor detection information with different frame rates input from a camera and LiDAR, and that are capable of efficiently and accurately performing synthesis processing (fusion processing) when performing matching processing of each sensor detection information.

[0012] A first aspect of the present disclosure resides in an information processing device having a data processing unit that inputs first sensor frames generated by a first sensor at a frame rate P (fps) and second sensor frames generated by a second sensor at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple consecutively input second sensor frames to generate a synthesized second sensor frame synchronized with the frame rate of the first sensor frames.

[0013] Furthermore, a second aspect of the present disclosure is an information processing method executed in an information processing device, in which a data processing unit inputs camera frames, which are images generated by a camera at a frame rate P (fps), and point cloud frames having point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple point cloud frames input consecutively to generate a synthesized point cloud frame synchronized with the frame rate of the camera frames.

[0014] Furthermore, a third aspect of the present disclosure is a program for executing information processing in an information processing device, which program inputs, into a data processing unit, camera frames which are images generated by a camera at a frame rate P (fps) and point cloud frames having point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple point cloud frames input consecutively to generate a synthesized point cloud frame synchronized with the frame rate of the camera frames.

[0015] The program of the present disclosure is, for example, a program that can be provided in a computer-readable format via a storage medium or a communication medium to an information processing device or a computer system capable of executing various program codes. By providing such a program in a computer-readable format, processing according to the program is realized on the information processing device or the computer system.

[0016] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description of the embodiments of the present disclosure and the accompanying drawings. Note that in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices that are located within the same housing.

[0017] According to one embodiment of the present disclosure, a device and method are realized in which multiple sensor detection data with different frame rates are synchronized and input to an analysis unit for analysis. Specifically, for example, a data processing unit inputs camera frames, which are images generated by a camera at a frame rate P (fps), and point cloud frames containing point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and compares frame-by-frame timestamps of the camera frames with timestamps of component parts within the point cloud frames to synthesize the multiple consecutively input point cloud frames and generate a composite point cloud frame synchronized with the frame rate of the camera frames. This configuration realizes a device and method in which multiple sensor detection data with different frame rates are synchronized and input to an analysis unit for analysis. Note that the effects described in this specification are merely exemplary and not limiting, and additional effects may also be present.

[0018] 1 is a diagram illustrating an example of sensor mounting on a vehicle equipped with an information processing device of the present disclosure. FIG. 1 is a diagram illustrating a specific example in which the frame rate of a camera frame differs from the frame rate of a point cloud frame generated by LiDAR. FIG. 2 is a diagram illustrating a specific example in which the frame rate of a camera frame differs from the frame rate of a point cloud frame generated by LiDAR. FIG. 3 is a diagram illustrating a problem when the frame rate of a camera frame differs from the frame rate of a point cloud frame generated by LiDAR. FIG. 4 is a diagram illustrating a configuration example of an information processing device of the present disclosure. FIG. 5 is a diagram illustrating a specific example of synchronization processing of sensor detection information performed by a sensor detection information synchronization processing unit. FIG. 6 is a diagram illustrating an overview of synchronization processing performed by a sensor detection information synchronization processing unit. FIG. 7 is a diagram illustrating synchronization processing using timestamps performed by a sensor detection information synchronization processing unit and processing for generating a composite point cloud frame. FIG. 8 is a diagram illustrating a point cloud data range extracted from point cloud frames L and L+1 used when generating a composite point cloud frame M. FIG. 9 is a diagram illustrating a specific example of point cloud data selection and extraction processing with reference to timestamps for each point cloud frame constituent part set in a point cloud frame. FIG. 10 is a diagram illustrating a specific example of point cloud data selection and extraction processing with reference to timestamps for each point cloud frame constituent part set in a point cloud frame. 10 is a diagram illustrating a specific example of a process for selectively extracting partial data from point cloud frame L and partial data from point cloud frame L+1 to generate one composite point cloud frame M, i.e., one composite point cloud frame M synchronized with a camera frame. FIG. 11 is a diagram illustrating an example of setting a timestamp on a pixel basis for a point cloud frame. FIG. 12 is a diagram illustrating an example of setting a timestamp on a multiple-pixel basis for constituent pixels of a point cloud frame. FIG. 13 is a diagram illustrating an example of setting a timestamp on a multiple-line basis for a point cloud frame. FIG. 14 is a diagram illustrating a flowchart illustrating a synchronization process sequence for sensor detection information executed by an information processing device of the present disclosure. FIG. 15 is a diagram illustrating a specific example of positional deviation of the same subject (object) in point cloud frame L and point cloud frame L+1 generated at different times by LiDAR. FIG. 16 is a diagram illustrating a configuration example of an information processing device of Example 2.27 is a diagram illustrating an example of a subject position shift that occurs when multiple camera frames captured by a general camera are combined. FIG. 28 is a diagram illustrating a specific example of a subject shift amount (dx) between multiple point cloud frames generated by LiDAR and a calculation process. FIG. 29 is a diagram illustrating a specific example of a composite point cloud frame correction process performed by a composite point cloud frame correction unit. FIG. 29 is a diagram illustrating an example of a composite point cloud frame correction process when both the LiDAR-equipped vehicle and the subject move. FIG. 29 is a diagram illustrating an example of a composite point cloud frame correction process when both the LiDAR-equipped vehicle and the subject move. FIG. 29 is a diagram illustrating an example of a composite point cloud frame correction process when both the LiDAR-equipped vehicle and the subject move. FIG. 29 is a diagram illustrating a flowchart illustrating a sequence of generation and correction processes of a composite point cloud frame when both the LiDAR and the subject move. FIG. 29 is a diagram illustrating an example of a hardware configuration of an information processing device of the present disclosure. FIG. 29 is a block diagram illustrating an example of a configuration of a vehicle control system for a vehicle equipped with an information processing device of the present disclosure. FIG. 29 is a diagram illustrating an example of a sensing area of ​​an external recognition sensor of the vehicle control system of FIG.

[0019] The information processing device, information processing method, and program of the present disclosure will be described in detail below with reference to the drawings. The description will be made according to the following items: 1. Example of sensor mounting on a vehicle 2. Configuration of the information processing device of the present disclosure and processing executed 3. Details of synchronization processing of multiple different sensor detection data 4. Synchronization processing sequence of sensor detection information executed by the information processing device of the present disclosure 5. Example of execution of correction processing of composite point cloud frame 6. Correction processing sequence of composite point cloud frame executed by the information processing device of the present disclosure 7. Example of hardware configuration of the information processing device of the present disclosure 8. Example of overall configuration of a vehicle equipped with the information processing device of the present disclosure 9. Summary of configuration of the present disclosure

[0020] [1. Examples of Sensor Mounting on Vehicle] First, examples of sensor mounting on a vehicle will be described.

[0021] 1 is a diagram showing an example of sensors mounted on a vehicle equipped with an information processing device according to the present disclosure, for example, a vehicle 10 that performs autonomous driving. The vehicle 10 is, for example, a vehicle that travels by autonomous driving.

[0022] In autonomous driving, obstacles are detected using cameras, laser-based sensors, and the like provided on the vehicle 10, and safe driving is achieved using the detected information. Sensors such as cameras detect various objects around the vehicle, such as oncoming vehicles, pedestrians, and guardrails, and the control unit of the vehicle 10 detects a safe driving route that avoids collisions with these objects and drives the vehicle autonomously.

[0023] 1 is a vehicle equipped with a device that analyzes detection information from sensors attached to the vehicle 10, i.e., an information processing device of the present disclosure. The information processing device of the present disclosure analyzes detection information from sensors such as a camera and a LiDAR (Laser Imaging Detection and Ranging) provided on the vehicle 10, outputs the analysis results to a vehicle control unit, and performs autonomous driving.

[0024] The LiDAR is an object detection sensor that uses laser light and analyzes the positions (distance, three-dimensional position, etc.) of various objects such as other vehicles, such as oncoming vehicles, pedestrians, and other structures around the vehicle 10. The LiDAR outputs a point cloud frame consisting of point cloud information that expresses, for example, object distance and three-dimensional position as pixel shading information and color information at each LiDAR detection timing.

[0025] The information processing device of the present disclosure analyzes the situation around the vehicle 10 using images captured by a camera, point cloud information detected by LiDAR, and the like.

[0026] Fig. 1 shows an example of mounting a camera 31 and a LiDAR 32 on a vehicle 10. The following four cameras 31 are mounted on the vehicle 10 shown in Fig. 1: a forward camera 31FC, a right camera 31RC, a left camera 31LC, and a rear camera 31BC. These four cameras continuously capture images (moving images) in their respective directions, and input the captured images (camera frames) to a data processing unit of the information processing device of the present disclosure.

[0027] 1 is also equipped with four LiDARs (lidars), which are object detection sensors that use laser light: forward LiDAR 32FL, right LiDAR 32RL, left LiDAR 32LL, and rear LiDAR 32BL. These four LiDARs (lidars) acquire the distance or three-dimensional position of objects (other vehicles, people, structures, etc.) in their respective directions, and input the acquired information as point cloud information consisting of pixel-by-pixel grayscale data and color data into a point cloud frame, which is then input to the data processing unit of the information processing device disclosed herein.

[0028] 1 is merely an example, and a configuration in which sensors different from those shown in this example are mounted is also possible. Also, a configuration in which more cameras are mounted, a configuration in which sensors other than LiDAR are mounted, etc. may be adopted.

[0029] The camera 31 mounted on the vehicle 10 shown in Fig. 1 captures images at a predetermined frame rate and inputs the captured camera frames to a data processing unit of the information processing device of the present disclosure. Similarly, the LiDAR 32 mounted on the vehicle 10 shown in Fig. 1 also generates a point cloud frame that shows distance information and the like of surrounding objects as, for example, a point cloud with different shading, and inputs the point cloud frame to the data processing unit of the information processing device of the present disclosure.

[0030] However, as mentioned above, the frame rate of the camera frames captured by the camera 31 and the frame rate of the point cloud frames generated by the LiDAR 32 are often set to different rates.

[0031] In this case, the analysis processing unit for the sensor detection information needs to analyze the camera frames and point cloud frames input from the camera and LiDAR at different times, which increases the processing load. Furthermore, it also makes it difficult to perform detection information synthesis processing (fusion processing), which includes matching processing such as object matching processing within the detection information from the camera and LiDAR. As a result, processing delays and processing errors are more likely to occur, which may reduce the safety of autonomous driving.

[0032] A specific example in which the frame rate of the camera frames captured by the camera 31 differs from the frame rate of the point cloud frames generated by the LiDAR 32 will be described with reference to Figures 2 and subsequent figures.

[0033] Figure 2 is a diagram showing an example of the output sequence of sensor detection data from the forward camera 31FC that captures images in front of the vehicle 10 and the forward LiDAR 32FL that detects the object distance (or three-dimensional position) in front of the vehicle 10.

[0034] The frame rate of the camera frames captured by the camera 31 is P (fps), and the frame rate of the point cloud frames generated by the LiDAR 32 is Q (fps). The frame rates of these sensors are different, i.e., P≠Q. Note that fps stands for frames per second, and refers to the number of output frames per second.

[0035] Specifically, for example, the settings are P = 30, Q = 15, i.e., the frame rate of the camera frame is 30 (fps), and the frame rate of the point cloud frame generated by LiDAR 32 is 15 (fps), etc.

[0036] Figure 2 shows two camera frames N and N+1, which are images captured consecutively by the camera 31, and two point cloud frames L and L+1 generated by the LiDAR 32.

[0037] Camera frames N and N+1 each capture an image of the front of the vehicle 10. As the vehicle 10 travels forward, the captured images change sequentially. Camera frame N is a camera frame composed of images whose capture began at time t2, and camera frame N+1 is a camera frame composed of images whose capture began at a subsequent time t4.

[0038] The frame rate of the camera frames is P (fps), and as shown in the figure, the time interval (t4-t2) between camera frame N and camera frame N+1 is camera frame interval (t4-t2)=1 / P (sec).

[0039] On the other hand, each of the point cloud frames L and L+1 is a point cloud frame configured by object position detection data consisting of point cloud information that indicates the distance (or three-dimensional position) of an object ahead of the vehicle, for example, using a point cloud with different shading. As the vehicle 10 travels forward, the point cloud frames also change sequentially.

[0040] Point cloud frame L is a point cloud frame composed of data for which detection by LiDAR began at time t1, and point cloud frame L+1 is a point cloud frame composed of data for which detection by LiDAR began at a subsequent time t3.

[0041] The frame rate of the point cloud frame is Q (fps), and as shown in the figure, the time interval (t3-t1) between point cloud frame L and point cloud frame L+1 is point cloud frame interval (t3-t1) = 1 / Q (sec).

[0042] Figure 3 is a diagram showing an example of the output sequence of sensor detection data from the right-direction camera 31RC that captures images to the right of the vehicle 10 and the right-direction LiDAR 32RL that detects the object distance (or three-dimensional position) to the right of the vehicle 10.

[0043] Each of camera frames N and N+1 captures an image to the right of the vehicle. Each of point cloud frames L and L+1 contains object position detection data consisting of point cloud information that indicates the distance (or three-dimensional position) of an object to the right of the vehicle using point clouds with different shading.

[0044] In this case, as explained with reference to Figure 2, the frame rate of the camera frame is P (fps) and the frame rate of the point cloud frame is Q (fps), and the image frame and point cloud frame are generated at different times.

[0045] 4 , the detection data from the camera 31 and the LiDAR 32, i.e., the camera frames and point cloud frames, are sequentially input to the sensor detection information analysis unit 40, which then executes analysis processing. The sensor detection information analysis unit 40 performs image analysis processing using the camera frames and object distance analysis processing using the point cloud frames to analyze the types and distances of objects (other vehicles, pedestrians, buildings, etc.) around the vehicle 10.

[0046] The results of the analysis processing in the sensor detection information analysis unit 40 are output to the vehicle control unit 50, and the vehicle control unit 50 controls the vehicle 10's driving speed and direction of travel based on the results of the analysis processing in the sensor detection information analysis unit 40, thereby allowing the vehicle 10 to drive safely.

[0047] However, as shown in Figure 4, camera frames, which are images captured by camera 31, are input sequentially at P (fps) to the sensor detection information analysis unit 40, and point cloud frames generated by LiDAR 32 are input sequentially at Q (fps).

[0048] In this case, the sensor detection information analysis unit 40 needs to perform analysis processing of the camera frames and point cloud frames input from these two sensors, i.e., the camera 31 and the LiDAR 32, at different times.

[0049] As shown in Figure 4, for camera frames input at P (fps), image analysis processing must be performed at intervals of (1 / P) seconds, and for point cloud frames input at Q (fps), object position (distance, etc.) analysis processing must be performed at intervals of (1 / Q) seconds.

[0050] As a result, the processing load on the sensor detection information analysis unit 40 increases. Furthermore, it becomes difficult to perform a fusion process of the analysis information from each sensor, which includes a matching process between an object in a camera frame and an object in a point cloud frame.

[0051] As a result, processing delays and processing errors are more likely to occur, which may reduce the safety of autonomous driving. The information processing device of the present disclosure solves these problems.

[0052] 2. Configuration of and Processes Executed by an Information Processing Apparatus According to the Present Disclosure Next, the configuration of and processes executed by an information processing apparatus according to the present disclosure will be described.

[0053] 5 is a diagram showing an example of the configuration of the information processing device 100 of the present disclosure. As shown in FIG. 5, the information processing device 100 of the present disclosure includes a sensor unit 110, a CAN 120, a data processing unit 150, and a vehicle control unit 160.

[0054] The sensor unit 110 includes a camera 111, a LiDAR 112, a radar 113, a GPS 114, and an IMU 115. The data processing unit 150 includes a timestamp setting unit 151, a sensor detection information synchronization processing unit 152, and a sensor detection information analysis unit 153.

[0055] In this embodiment, the sensor detection information synchronization processing unit 152 functions as a camera-LiDAR synchronization processing unit that performs synchronization processing between the camera frames generated sequentially by the camera 111 according to a frame rate of P (fps) and the point cloud frames generated sequentially by the LiDAR 112 according to a frame rate of Q (fps).

[0056] The camera 111 of the sensor unit 110 captures images of the surroundings of the vehicle 10. The camera 111 is configured by a plurality of cameras capturing images in each direction of the vehicle 10, for example, as described above with reference to FIG. 1 . Each camera captures images in each direction at a predetermined frame rate P (fps) and sequentially inputs the camera frames, which are the captured images, to the data processing unit 150. For example, if P=30, the camera 111 sequentially inputs camera frames (captured images) at a frame rate of 30 (fps) to the data processing unit 150.

[0057] As described above, the LiDAR 112 is an object detection sensor that uses laser light and analyzes the distance and three-dimensional position of various objects, such as other vehicles, such as oncoming vehicles, pedestrians, and other structures, around the vehicle 10. The LiDAR 112 is also composed of multiple LiDARs that detect object positions (distance, etc.) in each direction of the vehicle 10, as described above with reference to FIG.

[0058] Each LiDAR generates point cloud frames consisting of point cloud information that expresses object position information in each direction as pixel shading information at a predetermined frame rate Q (fps), and sequentially inputs the generated point cloud frames to the data processing unit 150. For example, if Q=15, the LiDAR 112 sequentially inputs point cloud frames (point cloud information such as distance images) at a frame rate of 15 (fps) to the data processing unit 150.

[0059] The radar 113 is a sensor that emits radio waves to detect the distance and direction of an object, and is composed of a pulse radar, continuous wave radar, etc. The GPS 114 is a sensor that estimates the vehicle's own position using GPS signals received from GPS satellites. The IMU 115 is an inertial measurement unit (IMU) that is a motion sensor that simultaneously measures acceleration in three axes (x, y, z), angular velocity around three axes (x, y, z), etc. Each of these sensors also generates sensor detection data at predetermined time intervals and sequentially inputs the generated sensor detection data to the data processing unit 150.

[0060] The CAN 120 is a standardized in-vehicle network. The CAN (Controller Area Network) is a communication network connected to each component of the vehicle 10, and each component of the vehicle 10 is controlled by transmitting and receiving CAN messages conforming to the CAN protocol.

[0061] Next, a description will be given of the configuration of the data processing unit 150. The data processing unit 150 includes a timestamp setting unit 151, a sensor detection information synchronization processing unit 152, and a sensor detection information analysis unit 153.

[0062] The timestamp setting unit 151 sets timestamps for the sensor detection information generated by each sensor in the sensor unit 110 and for the data transmitted and received via the CAN 120. That is, the timestamp setting unit 151 executes a process of setting a timestamp indicating the generation timing of the sensor detection information. The timestamp setting unit 151 also executes a process of setting a timestamp indicating the generation timing of the data transmitted and received via the CAN 120.

[0063] The sensor detection information synchronization processing unit 152 performs synchronization processing of the sensor detection information input from a plurality of different sensors. The sensor detection information synchronization processing unit 152 of this embodiment performs synchronization processing of the sensor detection information input from each sensor of the camera 111 and the LiDAR 112.

[0064] That is, synchronization processing is performed between the camera frames generated sequentially by the camera 111 at a frame rate of P (fps) and the point cloud frames generated sequentially by the LiDAR 112 at a frame rate of Q (fps).

[0065] 6, a specific example of the synchronization processing of the sensor detection information executed by the sensor detection information synchronization processing unit 152 will be described. As shown in Fig. 6, the sensor detection information synchronization processing unit 152 sequentially inputs camera frames sequentially generated by the camera 111 at a frame rate of P (fps) and point cloud frames sequentially generated by the LiDAR 112 at a frame rate of Q (fps), and executes synchronization processing of each of these frames.

[0066] The input camera frames are camera frames in which a timestamp is set for each camera frame. The input point cloud frames are point cloud frames in which a timestamp is set for each component of the point cloud frame, such as a pixel or line, rather than for each point cloud frame. The sensor detection information synchronization processor 152 uses these timestamps to perform synchronization processing. Details of this synchronization processing will be described later.

[0067] The sensor detection information synchronization processing unit 152 performs frame rate synchronization processing to convert the point cloud frame with a frame rate of Q (fps) input from the LiDAR 112 to a frame rate of P (fps), the same as the frame rate of the camera frame (captured image) input from the camera 111, and inputs it to the sensor detection information analysis unit 153.

[0068] Through this processing, the sensor detection information analysis unit 153 inputs the camera frame (captured image), which is the image captured by the camera 111, and the point cloud frame, which is the object position information generated by the LiDAR 112, as data with the same frame rate of P (fps), making it possible to process two different sensor detection data at the same time.

[0069] This reduces the processing load on the sensor detection information analysis unit 153. Furthermore, it becomes possible to easily and quickly execute a synthesis process (fusion process) of the analysis information of each sensor, which includes a matching process such as a comparison process between an object in a camera frame and an object in a point cloud frame.

[0070] As a result, processing delays and the occurrence of processing errors are reduced, making it possible to improve the safety of autonomous driving.

[0071] Details of the synchronization process between the camera frame and the point cloud frame executed by the sensor detection information synchronization processing unit 152 will be described later.

[0072] 5 , the description of the configuration of the information processing device 100 of the present disclosure will continue. The sensor detection information analysis unit 153 of the data processing unit 150 analyzes the sensor detection information input from each sensor of the sensor unit 110. That is, it performs processing such as analyzing the object types around the vehicle 10, such as oncoming vehicles, pedestrians, and buildings, and analyzing the object distance.

[0073] In addition, the sensor detection information analysis unit 153 inputs the camera frame (captured image), which is the image captured by the camera 111, and the point cloud frame, which is the object position information generated by the LiDAR 112, at the same frame rate of P (fps) via the sensor detection information synchronization processing unit 152, making it possible to perform analysis processing on these two different sensor detection data at the same time.

[0074] The analysis result of the sensor detection information analysis unit 153 is output to the vehicle control unit 160. Based on the analysis result of the sensor detection information analysis unit 153, the vehicle control unit 160 executes vehicle control, such as speed control and traveling direction control, to allow the vehicle 10 to travel safely.

[0075] [3. Details of Synchronization Processing of Data Detected by a Plurality of Different Sensors] Next, details of synchronization processing of data detected by a plurality of different sensors will be described.

[0076] As described above, the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100 of the present disclosure performs frame rate synchronization processing to convert point cloud frames having a frame rate of Q (fps) input from the LiDAR 112 into a frame rate of P (fps), which is the same as the frame rate of the camera frames (captured images) input from the camera 111. Details of this synchronization processing will be described below.

[0077] First, an overview of the synchronization process executed by the sensor detection information synchronization processing unit 152 will be described with reference to Fig. 7. Fig. 7 shows a time axis (t) in the center, with time elapsed from left to right.

[0078] Above the time axis (t), camera frame C and camera frame C+1 are shown, which are two consecutive frames input from the camera 111. Camera frame C shown on the left is the preceding camera frame as the preceding captured image, and camera frame C+1 shown on the right is the following camera frame as the following captured image.

[0079] Camera frame C shown on the left is a camera frame whose shooting started at time (t2), and camera frame C+1 shown on the right is a camera frame whose shooting started at a later time (t6).

[0080] If the camera frame rate is P (fps), the time difference (t6-t2) between the start time of shooting of camera frame C (t2) and the start time of shooting of the subsequent camera frame C+1 (t6) is: (t6-t2) = 1 / P (sec) If P (fps) = 30 (fps), then: (t6-t2) = 1 / P (sec) = (1 / 30) (sec).

[0081] Also, below the time axis (t) are shown point cloud frame L and point cloud frame L+1, which are two consecutive frames input from the LiDAR 112. Point cloud frame L shown on the left is the preceding point cloud frame, and point cloud frame L+1 shown on the right is the succeeding point cloud frame.

[0082] The point cloud frame L shown on the left is a point cloud frame for which the point cloud frame generation process by LiDAR112 started at time (t1), and the point cloud frame L+1 shown on the right is a point cloud frame for which the point cloud frame generation process by LiDAR112 started at a subsequent time (t4).

[0083] If the point cloud frame rate is Q (fps), the time difference (t4-t1) between the generation start time (t1) of point cloud frame L and the generation start time (t4) of the subsequent point cloud frame L+1 is: (t4-t1) = 1 / Q (sec) If Q (fps) = 15 (fps), then: (t4-t1) = 1 / Q (sec) = (1 / 15) (sec).

[0084] The sensor detection information synchronization processing unit 152 performs frame rate synchronization processing to convert point cloud frames with a frame rate of Q (fps) input from LiDAR 112 into a frame rate of P (fps), the same as the frame rate of the camera frame (captured image) input from camera 111.

[0085] In this synchronization process, the sensor detection information synchronization processing unit 152 generates a new composite point cloud frame M using two consecutive frames, point cloud frame L and point cloud frame L+1, input from the LiDAR 112 shown in Figure 7.

[0086] The new composite point cloud frame M generated by the sensor detection information synchronization processing unit 152 has a frame rate P (fps) that is the same as the frame rate of the camera frames (captured images) input from the camera 111 .

[0087] The sensor detection information synchronization processing unit 152 outputs to the sensor detection information analysis unit 153, together with the camera frame (captured image) input from the camera 111, a new composite point cloud frame M generated using two consecutive point cloud frames.

[0088] The sensor detection information synchronization processing unit 152 references the timestamps set in the camera frames and the point cloud frames to generate a composite point cloud frame M shown in FIG.

[0089] As described above, a timestamp is set for each camera frame in a camera frame, and a timestamp is set for each component of the point cloud frame, for example, for each pixel or line, in a point cloud frame. These timestamps are set by the timestamp setting unit 151 of the data processing unit 150 of the information processing device 100 shown in FIG. 5 .

[0090] Referring to FIG. 8, the synchronization process using timestamps executed by the sensor detection information synchronization processor 152, that is, the process of generating the composite point cloud frame M, will be described.

[0091] Similar to FIG. 7, FIG. 8 shows two consecutive camera frames C and C+1 above the time axis (t) and two consecutive point cloud frames L and L+1 below the time axis (t).

[0092] A timestamp is set for each camera frame in camera frames C and C+1 generated by camera 111. The timestamp indicates the generation start time of each camera frame (= the start time of shooting).

[0093] The timestamp of the preceding camera frame C is C(ts), where C(ts) = t2. The timestamp of the subsequent camera frame C+1 is C+1(ts), where C+1(ts) = t6.

[0094] When the camera frame rate is P (fps), the time difference (t6-t2) between the start time of shooting of camera frame C (t2) and the start time of shooting of the subsequent camera frame C+1 (t6) is (t6-t2)=1 / P (sec).

[0095] On the other hand, each of the point cloud frames L and L+1 generated by LiDAR 112 has a timestamp set for each component part of the point cloud frame, such as a pixel unit or a line unit.

[0096] A plurality of timestamps, (L)(ts)(s) to (L)(ts)(e), are set as timestamps for the point cloud frame L shown in the figure. That is, timestamps are set for each component of the point cloud frame L, for example, for each pixel or line.

[0097] The first timestamp (L)(ts)(s) of point cloud frame L corresponds to the generation start time (t1) of point cloud frame L, and (L)(ts)(s)=t1. The last timestamp (L)(ts)(e) of point cloud frame L corresponds to the generation end time (t3) of point cloud frame L, and (L)(ts)(e)=t3.

[0098] Furthermore, a plurality of timestamps, (L+1)(ts)(s) to (L+1)(ts)(e), are set as timestamps for the point cloud frame L+1 generated by the LiDAR 112 following the point cloud frame L. In other words, timestamps are set for each component of the point cloud frame L+1, for example, for each pixel or line.

[0099] The first timestamp (L+1)(ts)(s) of point cloud frame L+1 corresponds to the generation start time (t4) of point cloud frame L+1, and (L+1)(ts)(s) = t4. The last timestamp (L+1)(ts)(e) of point cloud frame L+1 corresponds to the generation end time (t7) of point cloud frame L, and (L+1)(ts)(e) = t7.

[0100] The composite point cloud frame M shown in Figure 8, i.e., the composite point cloud frame M having a frame rate that matches the frame rate P (fps) of the camera frame, is generated by combining a portion of the data of point cloud frame L and a portion of the data of point cloud frame L+1.

[0101] As shown in Figure 8, the sensor detection information synchronization processing unit 152 generates a composite point cloud frame M by combining the point cloud data of the latter half of point cloud frame L, i.e., the point cloud data in the range where timestamps (L)ts(u) to (L)ts(e) are set, with the point cloud data of the first half of point cloud frame L+1, i.e., the point cloud data in the range where timestamps (L+1)ts(s) to (L+1)ts(u).

[0102] Referring to FIG. 9, the data range of point cloud data extracted from point cloud frames L and L+1 used when generating a composite point cloud frame M will be described.

[0103] The point cloud data of point cloud frames L and L+1 used by the sensor detection information synchronization processing unit 152 when generating a composite point cloud frame M having a frame rate matching the camera frame frame rate P (fps) are as follows: (a) Point cloud data in point cloud frame L, in which timestamps in the range of (L)ts(u) to (L)ts(e) are set; (b) Point cloud data in point cloud frame L+1, in which timestamps in the range of (L+1)ts(s) to (L+1)ts(v) are set.

[0104] Here, the timestamp (L) ts(u) in the point cloud frame L shown in (a) above is the timestamp (C(ts)) for each camera frame of the camera frame C, i.e., the time t2 corresponding to the generation start time of the camera frame C (= the shooting start time of the camera frame C). Also, the timestamp (L) ts(e) is the time t3 corresponding to the generation end time of the point cloud frame L.

[0105] In other words, the point cloud data of point cloud frame L used when generating composite point cloud frame M is point cloud data whose timestamps corresponding to the constituent data of point cloud frame L (constituent data of pixels, lines, etc.) are in the following range: Timestamp = (L)ts(u) to (L)ts(e) = (C)ts to (L)ts(e) = t2 to t3.

[0106] In this way, from the point cloud frame L, the area of ​​point cloud data from the point cloud data portion set with a timestamp that matches or is closest to the camera frame unit timestamp (C(ts)) of camera frame C to the end position of the point cloud frame is selected and extracted when generating the composite point cloud frame M.

[0107] On the other hand, from the subsequent point cloud frame L+1, as described above, (b) in point cloud frame L+1, point cloud data in which timestamps are set in the range of (L+1)ts(s) to (L+1)ts(v) (= t4 to t5) is selected and extracted.

[0108] This corresponds to performing a process of selecting from point cloud frame L+1 an area that corresponds to an area (timestamp=t1 to t2) that was not selected from the preceding point cloud frame L.

[0109] The timestamp (L+1)ts(s) in point cloud frame L+1 shown in (b) above is time t4, which corresponds to the start time of generation of point cloud frame L+1. Note that, when the frame rate Q (fps) of the point cloud frame is used, (L+1)ts(s) can be expressed as follows, since the frame interval between point cloud frame L and frame L+1 is (1 / Q) sec: (L+1)ts(s) = t4 = (L)ts(s) + (1 / Q)

[0110] Furthermore, the timestamp (L+1)ts(v) can be expressed as follows: (L+1)ts(v) = t5 = (L+1)ts(s) + (t2 - t1) = (L)ts(s) + (1 / Q) + (t2 - t1) = (C)ts + (1 / Q)

[0111] In other words, the point cloud data of point cloud frame L+1 used when generating composite point cloud frame M is point cloud data in the range in which timestamps set corresponding to the constituent data of point cloud frame L+1 (constituent data of pixels, lines, etc.) are set as follows: Timestamp = (L+1)ts(s) to (L+1)ts(v) = t4 to t5 = (L+1)ts(s) to (C)ts+(1 / Q).

[0112] In this way, for point cloud frame L+1, the timestamp setting area ranging from the generation start position of point cloud frame L+1 (timestamp = t4) to the generation start position of camera frame C to be synchronized with the preceding point cloud frame L (timestamp = t2 (= C(ts)) plus the frame interval of the point cloud frames (1 / Q) ((C)ts + (1 / Q)) (timestamp = t5) becomes the point cloud data area used to generate the composite point cloud frame M.

[0113] When generating a composite point cloud frame M synchronized with one camera frame C in this way, the sensor detection information synchronization processing unit 152 selects a point cloud frame L including a timestamp that matches or is closest to the timestamp (C)ts of the camera frame C, and the subsequent point cloud frame L+1, and generates a composite point cloud frame M by synthesizing the point cloud data of (a) in point cloud frame L, (L)ts(u) to (L)ts(e), i.e., point cloud data set with timestamps in the range of (C)ts to (L)ts(e), and (b) in point cloud frame L+1, (L+1)ts(s) to (L+1)ts(v), i.e., point cloud data set with timestamps in the range of (L+1)ts(s) to (C)ts+(1 / Q).

[0114] As mentioned above, a point cloud frame has a timestamp set for each component part of the point cloud frame, such as a pixel unit or a line unit, and the sensor detection information synchronization processing unit 152 refers to the timestamp for each component part of the point cloud frame to determine the range to be selected and extracted from multiple point cloud frames.

[0115] A specific example of the point cloud data selection and extraction process that references the timestamps of the point cloud frame constituent parts set in the point cloud frame will be described with reference to FIG. 10 and subsequent figures.

[0116] FIG. 10 shows an example of a point cloud frame L that includes a timestamp that matches or is closest to the timestamp (C)ts of camera frame C. The point cloud frame L has timestamps set for each line (lines 1 to Y). The LiDAR 112 performs a process of scanning horizontally along one horizontal line (detecting object positions), and after scanning one line, scans the line below it. This process is repeated to scan multiple lines from the top line to the bottom line of the point cloud frame (the number of lines = Y in the example shown), generating one point cloud frame.

[0117] The timestamp of the top line of the point cloud frame L is the first timestamp corresponding to the generation start position of the point cloud frame L described with reference to Fig. 9, and is timestamp (L) ts(s). In the example shown in Fig. 10, timestamp (L) ts(s) = line 1 (L) ts = 2023-07-27, 07:25:22.000000.

[0118] The timestamp of the bottom line of the point cloud frame L is the first timestamp corresponding to the end position of generation of the point cloud frame L described with reference to Fig. 9, and is timestamp (L)ts(e). In the example shown in Fig. 10, timestamp (L)ts(e) = line Y(L)ts = 2023-07-27, 07:25:22.066600.

[0119] Furthermore, the line having the timestamp that matches or is closest to the timestamp C(ts) in the camera frame unit of the camera frame C that is the starting point of the area to be used in the composite point cloud frame described with reference to Figure 9, that is, the line near the middle of the point cloud frame L, as shown in Figure 10.

[0120] As shown in FIG. 10, the constituent data of the point cloud frame L used to generate the composite point cloud frame M is the point cloud data within the set range of timestamps (L)ts(u) (=C(ts)) to (L)ts(e).

[0121] That is, the point cloud data of the substantially lower half area in the point cloud frame L including the tree as the subject shown in FIG.

[0122] 11 is an example of a point cloud frame L+1 generated by the LiDAR 112 following the point cloud frame L. The point cloud frame L+1 also has timestamps set for each line (lines 1 to Y).

[0123] The timestamp of the top line of point cloud frame L+1 is the first timestamp corresponding to the generation start position of point cloud frame L+1 described with reference to Fig. 9, and is timestamp (L+1)ts(s). In the example shown in Fig. 11, timestamp (L+1)ts(s) = line 1 (L+1)ts = 2023-07-27, 07:25:22.066666.

[0124] The timestamp of the lowest line of point cloud frame L+1 is the first timestamp corresponding to the end position of generation of point cloud frame L+1 described with reference to Fig. 9, and is timestamp (L+1)ts(e). In the example shown in Fig. 11, timestamp (L+1)ts(e) = line Y(L+1)ts = 2023-07-27, 07:25:22.1333306.

[0125] Furthermore, the line having the timestamp that matches or is closest to the timestamp (L+1)ts(v) in the point cloud frame described with reference to Figure 9, i.e., the end point of the area to be used in the composite point cloud frame and the value obtained by adding (1 / Q) to the timestamp C(ts) per camera frame of the camera frame C that is the target of synchronization processing, is the line near the middle of the point cloud frame L+1, as shown in Figure 11.

[0126] As shown in FIG. 11, the constituent data of the point cloud frame L+1 used to generate the composite point cloud frame M is the point cloud data in the set range of timestamps (L+1)ts(s) to (L+1)ts(v) (=C(ts)+(1 / Q)).

[0127] That is, the point cloud data in the approximately upper half region of the point cloud frame L+1 containing trees as subjects shown in FIG.

[0128] Figure 12 shows a specific example of a process for selectively extracting some data from point cloud frame L and some data from point cloud frame L+1 to generate one composite point cloud frame M, i.e., one composite point cloud frame M synchronized with the camera frame.

[0129] As shown in Figure 12, the sensor detection information synchronization processing unit 152 combines point cloud data selected from the lower area of ​​point cloud frame L and point cloud data selected from the upper area of ​​point cloud frame L+1 to generate a single composite point cloud frame M synchronized with the camera frame.

[0130] That is, the sensor detection information synchronization processing unit 152 first selects the point cloud frame L having the timestamp closest to the timestamp of the camera frame C to be synchronized, and its subsequent point cloud frame L+1. Next, (a) from point cloud frame L, it selects timestamp setting areas (timestamp setting line areas) within the ranges of timestamp (L)ts(u) (=C(ts)) to (L)ts(e), and (b) from point cloud frame L+1, it selects timestamp setting areas (timestamp setting line areas) within the ranges of timestamp (L+1)ts(s) to (L+1)ts(v) (=C(ts) + (1 / Q)). Finally, it combines the point cloud data selected from the two point cloud frames L and L+1. The sensor detection information synchronization processing unit 152 executes these processes to generate a combined point cloud frame M synchronized with camera frame C.

[0131] The composite point cloud frame M generated by the sensor detection information synchronization processing unit 152 is output together with the camera frame C to the sensor detection information analysis unit 153 .

[0132] Through this processing, the sensor detection information analysis unit 153 inputs the camera frame (captured image), which is the image captured by the camera 111, and the point cloud frame, which is the object position information generated by the LiDAR 112, both as data with the same frame rate of P (fps), making it possible to process two different sensor detection data at the same time.

[0133] This reduces the processing load on the sensor detection information analysis unit 153. Furthermore, it becomes possible to easily and quickly execute a synthesis process (fusion process) of the analysis information of each sensor, which includes a matching process such as a comparison process between an object in a camera frame and an object in a point cloud frame.

[0134] Note that the example described with reference to Figures 7 to 12 is an example in which two consecutive point cloud frames L and L+1 are used to generate one composite point cloud frame M that is synchronized with the camera frame, but this is just one example of how to generate a composite point cloud frame M.

[0135] In reality, the difference between the generation start times of the camera frame P (fps) and the point cloud frame Q (fps) varies widely. For example, at a certain timing, the start times of the camera frame and the point cloud frame may almost coincide. In such a case, a process is performed in which one point cloud frame with a start time almost coinciding with the camera frame is used to generate a composite point cloud frame M synchronized with the camera frame.

[0136] Furthermore, the examples described with reference to Figures 10 to 12 are examples in which a timestamp is set on a line-by-line basis for a point cloud frame, and a point cloud data area is selected to generate a composite point cloud frame M that is synchronized with the camera frame on a line-by-line basis. However, the way in which the timestamp is set for a point cloud frame is not limited to line-by-line, and various settings are possible, such as pixel-by-pixel, multiple-pixel, multiple-line, etc.

[0137] In a configuration in which the timestamp is set in units of pixels, multiple pixels, multiple lines, etc., the selection process for the point cloud data area that generates the composite point cloud frame M synchronized with the camera frame is also performed in units of pixels, multiple pixels, multiple lines, etc.

[0138] An example of setting pixel-by-pixel timestamps for a point cloud frame will be described with reference to Fig. 13. An example of point cloud data for point cloud frame L including a tree as a detected object is shown on the left side of Fig. 13. Above that, similar to the example described with reference to Fig. 10, multiple timestamp setting examples (timestamps (L) ts(s) to (L) ts(e)) from the point cloud frame generation start position to the generation end position are shown.

[0139] 13 shows an enlarged view of point cloud frame L, illustrating an example of setting timestamps in pixel units for point cloud frame L. As shown in this enlarged view of point cloud frame L, point cloud frame L has horizontal lines 1 to Y and vertical lines 1 to X, and is composed of a total number of pixels = XY.

[0140] As explained above, the LiDAR 112 executes a scanning process (=object position detection process) in the horizontal direction starting from the top line 1, and sequentially scans lines 1 to Y while shifting the scan line position in the horizontal direction. The example explained above with reference to FIG. 10 is an example in which a timestamp is set for each horizontal line. The example shown in FIG. 13 is an example in which a timestamp is set for each pixel. The timestamp setting process is executed by the timestamp setting unit 151 of the data processing unit 150 explained above with reference to FIG. 5.

[0141] The timestamp of the upper left pixel (x, y) = (1, 1) of the constituent pixels of the point cloud frame is the timestamp of the start of generation of the point cloud frame L, and is the timestamp (L) ts(s). Also, the timestamp of the lower right pixel (x, y) = (X, Y) of the constituent pixels of the point cloud frame is the timestamp of the end of generation of the point cloud frame L, and is the timestamp (L) ts(e).

[0142] The example shown in FIG. 13 is an example in which a different timestamp is set for each pixel of the constituent pixels of the point cloud frame, and an individual timestamp is set for each constituent pixel (x, y) of the point cloud frame.

[0143] Even in a configuration in which pixel-level timestamps are set in this manner, the process of generating the composite point cloud frame M described above with reference to Figures 9 to 12, i.e., the composite point cloud frame M synchronized with the camera frame, can be performed using pixel-level timestamps.

[0144] Fig. 14 shows an example in which timestamps are set in units of multiple pixels constituting a point cloud frame. Fig. 14 also shows an example in which timestamps are set in units of four pixels. The timestamp (L)ts(1) of the four pixels at the top left of the point cloud frame is the timestamp at the start of generation of point cloud frame L, and timestamp (L)ts(1) = timestamp (L)ts(s).

[0145] Furthermore, the timestamp (L) ts(n) of the four pixels at the bottom right of the constituent pixels of the point cloud frame is the timestamp at the end of generation of the point cloud frame L, and timestamp (L) ts(n) = timestamp (L) ts(e).

[0146] In this way, the example shown in FIG. 14 is an example of setting timestamps in units of multiple constituent pixels of a point cloud frame, and is an example of setting individual timestamps in units of four pixels of the point cloud frame.

[0147] Even in a configuration in which timestamps are set in units of multiple pixels in this manner, it is possible to perform the process of generating the composite point cloud frame M described above with reference to Figures 9 to 12, i.e., the composite point cloud frame M synchronized with the camera frame, using timestamps in units of multiple pixels.

[0148] Fig. 15 shows an example in which timestamps are set in units of multiple lines of a point cloud frame. Fig. 15 also shows an example in which timestamps are set in units of two lines. The timestamps (L)ts(1) of the first two lines of pixels constituting the point cloud frame are the timestamps at the start of generation of the point cloud frame L, and timestamp (L)ts(1) = timestamp (L)ts(s).

[0149] Furthermore, the timestamp (L) ts(Y / 2) of the bottom two lines of the point cloud frame is the timestamp at the end of generation of the point cloud frame L, and timestamp (L) ts(Y / 2) = timestamp (L) ts(e).

[0150] In this way, the example shown in FIG. 15 is an example of setting timestamps in units of multiple lines of a point cloud frame, and is an example of setting individual timestamps in units of two lines of a point cloud frame.

[0151] Even in a configuration in which timestamps are set in units of multiple lines in this manner, it is possible to perform the process of generating the composite point cloud frame M described above with reference to Figures 9 to 12, i.e., the composite point cloud frame M synchronized with the camera frame, using timestamps in units of multiple lines.

[0152] 4. Regarding the Sensor Detection Information Synchronization Processing Sequence Executed by the Information Processing Device of the Present Disclosure] Next, the sensor detection information synchronization processing sequence executed by the information processing device of the present disclosure will be described.

[0153] The flowchart shown in FIG. 16 is a flowchart illustrating a synchronization process sequence of sensor detection information executed by the information processing device 100 of the present disclosure shown in FIG.

[0154] Note that the processing according to the flowchart shown in Fig. 16 can be executed according to a program stored in a storage unit of the information processing device 100 of the present disclosure, although this program is not shown in Fig. 5. The information processing device 100 of the present disclosure has a processor such as a CPU having a program execution function, and can perform processing according to the flow shown in Fig. 16 by executing the program by the processor. The processing of each step of the flowchart shown in Fig. 16 will be described below in order.

[0155] (Step S101) First, in step S101, the information processing device of the present disclosure executes an image capturing process (Pfps) by the camera 111 of the sensor unit 110 and an object position (distance, etc.) detection process (scanning process) (Qfps) by the LiDAR 112.

[0156] The camera 111 captures images at a frame rate of P (fps), and the LiDAR 112 performs object position (distance, etc.) detection processing (scanning processing) at a frame rate of Q (fps). Specifically, for example, P (fps) = 30 (fps) and Q (fps) = 15 (fps), i.e., P ≠ Q, and the camera frames and point cloud frames are generated at different frame rates.

[0157] (Step S102) Next, in step S102, the information processing device according to the present disclosure executes the following processes: (a) setting a camera frame unit timestamp (a timestamp indicating the start time of camera frame capture) in a camera frame (Pfps) that is an image captured by a camera, and (b) setting a point cloud frame component unit timestamp (a timestamp for each pixel, line, etc.) in a point cloud frame (Qfps) generated by the LiDAR.

[0158] This process is executed by the timestamp setting unit 151 of the data processing unit 150 of the information processing device of the present disclosure. The timestamp setting unit 151 sets a camera frame unit timestamp, i.e., a timestamp indicating the start time of camera frame capture, for the camera frame, which is an image captured by the camera 111. Furthermore, for the point cloud frame generated by the LiDAR 112, a point cloud frame component unit timestamp (a timestamp for each pixel, each line, etc.) is set.

[0159] The processing according to the flow shown in Figure 16 is a processing that is repeatedly executed, and the timestamp setting unit 151 repeatedly executes a camera frame unit timestamp setting processing for each camera frame that is continuously generated by the camera 111, and a point cloud frame component part unit timestamp setting processing for the point cloud frames that are continuously generated by the LiDAR 112, for the camera frames and point cloud frames that are generated sequentially.

[0160] (Step S103) Next, in step S103, the information processing device of the present disclosure acquires a camera frame unit timestamp from one camera frame.

[0161] This process is executed by the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100 shown in FIG.

[0162] 16 is a process that is repeatedly executed, and the sensor detection information synchronization processing unit 152 sequentially executes the camera frame unit time stamp acquisition process for each camera frame input to the data processing unit 150. The processes from step S103 onwards are a process that is repeatedly executed for each camera frame input to the data processing unit 150.

[0163] (Step S104) Next, in step S104, the information processing apparatus of the present disclosure selects, from the point cloud frame, a point cloud frame constituent part unit timestamp that is closest to the camera frame unit timestamp acquired in step S103.

[0164] This process is also executed by the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100 shown in FIG.

[0165] A specific example of this process will be described with reference to the example shown in Fig. 9 described above. Assume that the camera frame unit timestamp acquired in step S103 is the timestamp (C)ts of camera frame C shown in Fig. 9. In this case, in step S104, a process is executed to select the point cloud frame constituent part unit timestamp (L)ts(u) from point cloud frame L shown in Fig. 9 as the point cloud frame constituent part unit timestamp closest to the timestamp (C)ts of camera frame C.

[0166] (Step S105) Next, the information processing device of the present disclosure executes the following process in step S105.

[0167] The point cloud data after the point cloud data for which the point cloud frame constituent part unit timestamp selected in step S104 is set is obtained from one or more point cloud frames and synthesized to generate a new synthesized point cloud frame (=synthesized point cloud frame synchronized with the camera image frame).

[0168] This process is also executed by the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100 shown in FIG.

[0169] A specific example of this process will be described with reference to the example shown in Fig. 9 described above. Assume that the point cloud frame constituent part unit timestamp selected in step S104 is the point cloud frame constituent part unit timestamp (L)ts(u) of point cloud frame L shown in Fig. 9. In this case, in step S105, point cloud data subsequent to the point cloud data for which the point cloud frame constituent part unit timestamp (L)ts(u) of point cloud frame L shown in Fig. 9 is set is obtained from point cloud frame L and the subsequent point cloud frame L+1 and combined to generate a new combined point cloud frame (= combined point cloud frame synchronized with the camera image frame).

[0170] That is, in the example shown in FIG. 9 , the sensor detection information synchronization processing unit 152 selects a point cloud frame L including a timestamp that matches or is closest to the timestamp (C)ts of camera frame C, and the subsequent point cloud frame L+1, and generates a composite point cloud frame M by combining the point cloud data (a) in point cloud frame L, (L)ts(u) to (L)ts(e), i.e., point cloud data set with timestamps in the range of (C)ts to (L)ts(e), and (b) in point cloud frame L+1, (L+1)ts(s) to (L+1)ts(v), i.e., point cloud data set with timestamps in the range of (L+1)ts(s) to (C)ts+(1 / Q).

[0171] The composite point cloud frame M generated here is a point cloud frame synchronized with the camera frame C. That is, in step S105, a process is executed to sequentially generate composite point cloud frames at a frame rate P (fps) synchronized with the frame rate P (fps) of the camera frame.

[0172] (Step S106) Next, the information processing device of the present disclosure executes the following process in step S106.

[0173] The sensor detection information synchronization processing unit 152 outputs the generated composite point cloud frame to the sensor detection information analysis unit 153 together with the camera frame.

[0174] As described above, the composite point cloud frame M generated in step S105 is a point cloud frame synchronized with the camera frame C, and in step S106, the composite point cloud frame having the same frame rate = P (fps) is output from the sensor detection information synchronization processing unit 152 to the sensor detection information analysis unit 153 together with the camera frame having the same frame rate = P (fps).

[0175] (Step S107) Next, the information processing device of the present disclosure executes the following process in step S107.

[0176] The sensor detection information analysis unit 153 performs analysis processing on the camera frame and the composite point cloud frame input from the sensor detection information synchronization processing unit 152 to generate analysis information, and outputs the generated analysis information to the vehicle control unit.

[0177] As described above, the sensor detection information analysis unit 153 can input, from the sensor detection information synchronization processing unit 152, camera frames with a frame rate of P (fps) as well as composite point cloud frames with the same frame rate of P (fps).

[0178] This allows the sensor detection information synchronization processing unit 152 to simultaneously perform image analysis and point cloud analysis at the same timing in a cycle according to the frame rate of the camera frame. As a result, the processing load on the sensor detection information analysis unit 153 is reduced. Furthermore, it also becomes possible to easily and quickly perform synthesis processing (fusion processing) of the analysis information of each sensor, including matching processing between objects in the camera frame and objects in the point cloud frame.

[0179] (Step S108) Next, the information processing device of the present disclosure executes the following process in step S108.

[0180] The vehicle control unit 160 of the information processing device executes vehicle control using the analysis information input from the sensor detection information analysis unit 153 of the data processing unit 150 .

[0181] As described above, the sensor detection information analysis unit 153 performs both image analysis and point cloud analysis at the same timing in a cycle according to the frame rate of the camera frame. The sensor detection information analysis unit 153 outputs the analysis results to the vehicle control unit 160. The vehicle control unit 160 uses the analysis information input from the sensor detection information analysis unit 153 to perform vehicle control.

[0182] As described above, the sensor detection information synchronization processing unit 152 can perform image analysis and point cloud analysis simultaneously at the same timing in a cycle that conforms to the frame rate of the camera frame. As a result, processing delays and the occurrence of processing errors are reduced, and the vehicle control unit 160 is able to perform safer autonomous driving control.

[0183] In the above-described embodiment, the sensor detection data to be synchronized is a combination of a camera frame generated by a camera and a point cloud frame generated by a LiDAR, but this is just one example. The synchronization process using a timestamp according to the process of the present disclosure is not limited to a combination of a camera and a LiDAR, and can also be applied to combinations of other sensors.

[0184] 5. Example of Executing Correction Processing of Composite Point Cloud Frame Next, an example of executing correction processing of a composite point cloud frame will be described as a second example of an information processing device of the present disclosure.

[0185] As described above, the information processing device 100 of the present disclosure performs a process of selectively extracting some data from point cloud frame L and some data from point cloud frame L+1 to generate one composite point cloud frame M, i.e., one composite point cloud frame M synchronized with the camera frame.

[0186] The sensor detection information synchronization processing unit 152 of the information processing device 100 of the present disclosure combines point cloud data selected from the lines in the lower region of point cloud frame L and point cloud data selected from the lines in the upper region of point cloud frame L+1, as shown in, for example, previously described Figure 12, to generate one composite point cloud frame M synchronized with the camera frame.

[0187] That is, the sensor detection information synchronization processing unit 152 first selects the point cloud frame L having the timestamp closest to the timestamp of the camera frame C to be synchronized, and its subsequent point cloud frame L+1. Next, (a) from point cloud frame L, it selects timestamp setting areas within the ranges of timestamp (L)ts(u) (=C(ts)) to (L)ts(e), and (b) from point cloud frame L+1, it selects timestamp setting areas within the ranges of timestamp (L+1)ts(s) to (L+1)ts(v) (=C(ts) + (1 / Q)). Finally, it combines the point cloud data selected from the two point cloud frames L and L+1. The sensor detection information synchronization processing unit 152 executes these processes to generate a combined point cloud frame M synchronized with camera frame C.

[0188] However, the two point cloud frames L and L+1 used to generate the composite point cloud frame M are point cloud frames generated at different times. Therefore, for example, when the vehicle 10 equipped with the LiDAR 112 is traveling or when the object being the subject is a moving object such as an oncoming vehicle, a discrepancy occurs between the positions of the same subject (object) in the point cloud frame L and the point cloud frame L+1 generated by the LiDAR 112 at different times.

[0189] Referring to Figure 17, a specific example of positional deviation occurring for the same subject (object) in point cloud frame L and point cloud frame L+1 generated by LiDAR 112 at different times is described.

[0190] As shown on the left side of Fig. 17, the vehicle 10 is traveling leftward at a speed V. There is a subject (tree) on the right side of the vehicle 10. At this time, the right-direction LiDAR32RL mounted on the vehicle 10 continuously generates two point cloud frames, L and L+1, each of which records point cloud data indicating the position (distance, etc.) of the subject (tree).

[0191] 17, however, the positions of the point cloud data of the subject (tree) recorded in point cloud frame L and point cloud frame L+1 are different. That is, the position of the point cloud data of the subject (tree) in point cloud frame L is on the left side of the frame, while the position of the point cloud data of the subject (tree) in point cloud frame L+1 is on the right side of the frame.

[0192] This is because the timing of generating the point cloud frame L and the point cloud frame L+1 is different. That is, after generating the point cloud frame L, the vehicle 10 moves, and the position of the LiDAR 32 also moves as the vehicle moves.

[0193] As a result, as shown in the lower part of Figure 17, a large deviation occurs in the subject (tree) in the composite point cloud frame M generated by combining point cloud data selected from two point cloud frames L and point cloud frame L+1 at the junction of the two point cloud frames.

[0194] 5 generates a composite point cloud frame M in which such a subject position shift has occurred and outputs it to the sensor detection information analysis unit 153. This prevents the sensor detection information analysis unit 153 from accurately detecting the object position (distance, etc.). This also reduces the accuracy of the synthesis process (fusion process) of analysis information from multiple sensors, such as the matching process of objects in camera-captured images.

[0195] An embodiment that solves this problem, that is, a second embodiment that executes a correction process for a composite image, will be described below.

[0196] 18 shows an example of the configuration of an information processing device 100b according to the second embodiment. The difference between the information processing device 100b according to the second embodiment and the information processing device 100 described above with reference to FIG. 5 is the configuration of the sensor detection information synchronization processing unit 152.

[0197] The sensor detection information synchronization processing unit 152 in this embodiment includes a composite point cloud frame generation unit 181 and a composite point cloud frame correction unit 182 .

[0198] The composite point cloud frame generation unit 181 performs the processing performed by the sensor detection information synchronization processing unit 152 of the information processing device 100 described earlier with reference to Figure 5, i.e., the processing of synthesizing point cloud data selected and extracted from multiple point cloud frames to generate a composite point cloud frame synchronized with the camera frame.

[0199] If a positional shift of the subject occurs in the composite point cloud frame generated by the composite point cloud frame generation unit 181, the composite point cloud frame correction unit 182 performs an image correction process to eliminate the positional shift, and outputs the corrected composite point cloud frame without the positional shift of the subject to the sensor detection information analysis unit 153 together with the camera frame to be synchronized.

[0200] In this way, the sensor detection information synchronization processing unit 152 of this embodiment has a composite point cloud frame generation unit 181 and a composite point cloud frame correction unit 182, and if there is a positional shift of the subject in the composite point cloud frame generated by the composite point cloud frame generation unit 181, the composite point cloud frame correction unit 182 performs a correction process to eliminate the positional shift of the subject, and generates and outputs a corrected composite point cloud frame without the positional shift of the subject.

[0201] The correction process executed by the composite point cloud frame correction unit 182 to eliminate the subject position shift will be described in detail below.

[0202] Before explaining the correction process for the subject position shift that occurs in the point cloud frame synthesis process generated by LiDAR, we will first explain an example of the subject position shift that occurs when multiple camera frames captured by a general camera are synthesized.

[0203] The diagram on the left side of Fig. 19 is a diagram showing "(a) a specific example of subject deviation." This diagram explains an example in which a camera 111 mounted on a vehicle 10 traveling at a certain speed captures an image of a subject (a tree). The camera 111 captures the scenery to the right of the vehicle 10. For example, image capture is performed at a frame rate of P (fps). The vehicle 10 is traveling at a constant speed (V) to the left of the diagram.

[0204] FIG. 19 shows the vehicle positions at time t1 and the subsequent time t2, that is, vehicle (camera) position @t1, vehicle (camera) position @t2.

[0205] Furthermore, below each vehicle position, a captured image @t1, which is a camera frame captured by camera 111 at vehicle (camera) position @t1 at time t1, and a captured image @t2, which is a camera frame captured by camera 111 at vehicle (camera) position @t2 at a subsequent time t2, are shown.

[0206] The subject (tree) in the image @t1 taken at time t1 is located on the left side of the camera frame, but the subject (tree) in the image @t2 taken at a later time t2 is located on the right side of the camera frame. In this way, the positions of the same subject, a tree, are different in the images taken at different times t1 and t2.

[0207] This is because the position of the vehicle 10, i.e., the position of the camera 111, has moved during the time lapse between times t1 and t2, and the relative position of the subject (tree) with respect to the camera 111 has changed.

[0208] The positional deviation of the subject (tree) in the camera frame at times t1 and t2 (the number of pixels in the camera frame) is dx shown at the bottom of FIG. 19(a).

[0209] This subject displacement amount dx can be calculated according to the following (Equation 1), as shown in "(b) Example of calculation of subject displacement amount dx within camera frame" on the right side of FIG.

[0210] dx = (x x w) / (2 x L x tan(θ / 2)) (Equation 1) Note that the parameters in the above (Equation 1) are as follows: x: horizontal movement distance of the camera (or subject) (unit: meters) w: number of horizontal pixels in the image captured by the camera (unit: pixels) L: distance from the camera to the subject (unit: meters) (distance in the direction of the camera's optical axis) θ: angle of view of the camera lens (unit: rad)

[0211] Among these parameters, x (= horizontal movement distance of the camera (or subject)) can utilize the results of analysis of vehicle speed information and steering information input from the vehicle control unit 160 via CAN (Controller Area Network) 120, which is a communication network connected to each component of the vehicle 10, and input information from the camera 111 and other sensor units 110. L (= distance from the camera to the subject) can also utilize the results of analysis of input information from the sensor units 110.

[0212] For w (= the number of horizontal pixels in the image captured by the camera) and θ (= the angle of view of the camera lens), information stored in advance in the memory of the information processing device 100 or information stored in the memory of the camera 111 can be used.

[0213] Subject deviation in a point cloud frame indicating an object position (distance, etc.) detected by LiDAR also occurs in a manner similar to that of the camera-captured image shown in Fig. 19. A specific example of the subject deviation amount (dx) between multiple point cloud frames generated by LiDAR and the calculation process will be described with reference to Fig. 20.

[0214] The diagram on the left side of Figure 20 is a diagram showing "(a) Specific example of subject deviation" similar to Figure 19(a). This diagram explains an example in which the LiDAR 112 mounted on a vehicle 10 traveling at a certain speed performs a scan process to detect the position (distance, etc.) of a subject (tree). The LiDAR 112 performs a scan process to detect the object position in the area to the right of the vehicle 10. For example, point cloud frames are generated at a frame rate of Q (fps). The vehicle 10 is traveling at a constant speed (V) to the left of the diagram.

[0215] FIG. 20 shows the vehicle positions at time t1 and the subsequent time t2, i.e., Vehicle (LiDAR) position @t1, Vehicle (LiDAR) position @t2, and these vehicle positions.

[0216] Furthermore, below each vehicle position, a "point cloud frame @t1" is shown, which is the point cloud frame generated by LiDAR 112 at the vehicle (LiDAR) position @t1 at time t1, and a "point cloud frame @t2" is shown, which is the point cloud frame generated by LiDAR 112 at the vehicle (LiDAR) position @t2 at a subsequent time t2.

[0217] The position of the subject (tree) in the "point cloud frame @t1" generated at time t1 is located on the left side of the point cloud frame, but the position of the subject (tree) in the "point cloud frame @t2" generated at a later time t2 is located on the right side of the point cloud frame. In this way, the positions of the same subject, a tree, are different in the point cloud frames generated at different times t1 and t2.

[0218] This is because the position of the vehicle 10, i.e., the position of the LiDAR 112, moved during the time lapse between times t1 and t2, and the relative position of the subject (tree) with respect to the LiDAR 112 changed.

[0219] The positional deviation of the subject (tree) in the point cloud frame at times t1 and t2 (the number of pixels in the point cloud frame) is dx shown at the bottom of FIG. 20(a).

[0220] This subject displacement amount dx can be calculated according to the following (Equation 2), as shown in "(b) Example of calculation of subject displacement amount dx in point cloud frame" on the right side of FIG.

[0221] dx = (v x t x w) / (2 x L x tan(θ / 2)) (Equation 2) The parameters in the above (Equation 2) are as follows: v: vehicle (or subject) moving speed (unit: (m / s)) t: time interval between point cloud frames continuously generated by the LiDAR (unit: (sec)) w: number of horizontal pixels in the LiDAR-generated point cloud frame (unit: pixels) L: distance from the LiDAR to the subject (unit: m) (vertical distance of the LiDAR) θ: LiDAR lens angle of view (unit: rad) Note that (v x t) = x: horizontal moving distance of the LiDAR (or subject) (unit: m).

[0222] Among these parameters, v (= the moving speed of the vehicle (or subject)) can utilize the results of analyzing vehicle speed information and steering information input from the vehicle control unit 160 via CAN (Controller Area Network) 120, which is a communication network connected to each component of the vehicle 10, and input information from the camera 111 and other sensor units 110. L (= the distance from the LiDAR to the subject) can also utilize the results of analyzing input information from the sensor unit 110.

[0223] For t (= the time interval between point cloud frames continuously generated by LiDAR), w (= the number of horizontal pixels in the LiDAR-generated point cloud frame), and θ (the lens angle of view of the LiDAR), information stored in advance in the memory of the information processing device 100 or information stored in the memory of LiDAR 112 can be used.

[0224] The composite point cloud frame correction unit 182 configured within the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100b of this embodiment shown in Figure 18 calculates the subject shift amount dx that occurs in the composite point cloud frame according to the above (Equation 2), and performs correction processing of the composite point cloud frame using the calculated subject shift amount dx.

[0225] A specific example of the composite point cloud frame correction process executed by the composite point cloud frame correction unit 182 will be described with reference to FIG.

[0226] The upper part of Figure 21 shows two point cloud frames, L and L+1, successively generated by the LiDAR 112. Each point cloud frame contains point cloud data of the same subject (tree). However, as described with reference to Figure 20, point cloud frames L and L+1 were generated while the vehicle 10 equipped with the LiDAR 112 was traveling at a predetermined speed (V), and therefore the positions of the subjects (trees) in each frame are shifted.

[0227] The composite point cloud frame generation unit 181 configured within the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100b shown in Figure 18 selects a portion of point cloud data from each of the two point cloud frames L and L+1 shown in the upper part of Figure 21, and generates a composite point cloud frame M shown in the lower left of Figure 21.

[0228] The composite point cloud frame M shown in the lower left of Fig. 21 is a composite point cloud frame in which the upper point cloud data is point cloud data acquired from point cloud frame L+1, and the lower point cloud data is point cloud data acquired from point cloud frame L. As is clear from the figure, the positions of the subjects (trees) are shifted vertically.

[0229] The composite point cloud frame corrector 182 in the sensor detection information synchronization processor 152 performs the correction process for this subject position shift. As previously described with reference to Fig. 9 etc., the point cloud frame L is a point cloud frame having a timestamp that matches or is closest to the timestamp (C)ts of the camera frame C, which is the shooting start timing of the camera frame C that is the synchronization process target.

[0230] In order to perform fusion processing (combining processing) including matching processing such as subject matching processing between camera frame C and composite point cloud frame, it is preferable to use camera frames and point cloud frames with closer time stamps. Therefore, the composite point cloud frame correction unit 182 sets point cloud frame L+1 as the correction target, using point cloud frame L, which has a time stamp closer to the time stamp (C)ts of camera frame C, which is the synchronization processing target, as the reference. In other words, correction processing is performed to match the subject position in point cloud frame L+1 with the subject position in point cloud frame L.

[0231] During this subject position correction process, the composite point cloud frame correction unit 182 calculates the subject shift amount (dx) described with reference to Figure 20 in accordance with the above-mentioned (Equation 2), and uses the calculated subject shift amount (dx) to perform correction processing to move the pixel position of point cloud frame L+1 within the composite point cloud frame M.

[0232] The corrected combined point cloud frame M shown in the lower right of FIG. 21 is the “corrected combined point cloud frame M” generated as a result of correction by the combined point cloud frame correction unit 182 .

[0233] The "corrected composite point cloud frame M" is a composite point cloud frame generated by performing a correction process to move the subject position in the upper point cloud frame L+1 to the left by a number of pixels equivalent to the subject displacement amount (dx) calculated according to the above-mentioned (Equation 2). Through this correction process, the subject in the corrected composite point cloud frame becomes a subject in which the displacement between the two point cloud frames to be composited has been eliminated.

[0234] The sensor detection information synchronization processing unit 152 of this embodiment shown in Figure 18 outputs this corrected composite point cloud frame M generated by the composite point cloud frame correction unit 182 and the camera frame C to be synchronized to the sensor detection information analysis unit 153.

[0235] As described above, the corrected composite point cloud frame M is a composite point cloud frame generated using the point cloud frame L as a reference frame, which has a timestamp that matches or is close to the frame-by-frame timestamp (C)ts of the camera frame C to be synchronized. In other words, the camera frame C and the corrected composite point cloud frame M are frames composed of subject data observed at approximately the same time and from approximately the same direction, and the subject positions in each frame are approximately the same.

[0236] As a result, the sensor detection information analysis unit 153 can perform analysis processing of the camera frame C and the corrected composite point cloud frame M, as well as fusion processing (combination processing) including subject matching, more efficiently and with higher accuracy.

[0237] Note that the subject deviation correction process for the composite point cloud frame described with reference to Figure 21 is an example of a case where a vehicle equipped with LiDAR is moving, but similar processing is also possible when the vehicle equipped with LiDAR stops and the subject moves.

[0238] However, when the LiDAR-equipped vehicle and the subject move, processing that takes into account the amount of movement of both is required. An example of correction processing for a composite point cloud frame when both the LiDAR-equipped vehicle and the subject move will be described with reference to Figure 22 and subsequent figures.

[0239] The upper part of Fig. 22 shows a vehicle 10 equipped with a LiDAR 112 and an oncoming vehicle 20. The vehicle 10 equipped with a LiDAR 112 is traveling to the left of the figure at a speed Vx (e.g., +10 m / s), and the oncoming vehicle 20 is traveling to the right of the figure at a speed Vy (e.g., -15 m / s). Note that the speed to the left is represented as (+) and the speed to the right as (-).

[0240] In this situation, the LiDAR 112 of the vehicle 10 performs a scan process of the subject to the right of the vehicle and continuously generates point cloud frames at a predetermined frame rate. Fig. 22 shows a point cloud frame L (point cloud frame @t1) generated by the LiDAR 112 of the vehicle 10 at time t1 and a point cloud frame L+1 (point cloud frame @t2) generated at a subsequent time t2.

[0241] Point cloud frame L (point cloud frame @t1) and point cloud frame L+1 (point cloud frame @t2) contain point cloud data of the oncoming vehicle 20, which is the subject. However, the point cloud data positions of the oncoming vehicle 20 in the two point cloud frames are different. This is because the vehicle 10 and the oncoming vehicle 20 are traveling in opposite directions, and the relative positions of the LiDAR 112 of the vehicle 10 and the oncoming vehicle 20 have changed at times t1 and t2.

[0242] If a composite point cloud frame M is generated using these two point cloud frames L and L+1, the point cloud data of the oncoming vehicle 20, which is the subject in the composite point cloud frame, will be significantly shifted at the boundary between the point cloud frames L and L+1, as shown in Figure 23.

[0243] In such a case, the composite point cloud frame corrector 182 of the information processing device 100b according to this embodiment executes a process for correcting this subject shift. A specific example of the composite point cloud frame correction process executed by the composite point cloud frame corrector 182 will be described with reference to FIG.

[0244] The upper part of Figure 24 shows two point cloud frames, L and L+1, successively generated by the LiDAR 112. Each point cloud frame contains point cloud data of the same subject (the oncoming vehicle 20). However, as described with reference to Figure 22, point cloud frames L and L+1 were generated while the vehicle 10 equipped with the LiDAR 112 was traveling at a predetermined speed (Vx) and the oncoming vehicle 20 was traveling in the opposite direction at a predetermined speed (Vy), and therefore the positions of the subject (the oncoming vehicle 20) in each frame are shifted.

[0245] The composite point cloud frame generation unit 181 configured within the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100b shown in Figure 18 selects a portion of point cloud data from each of the two point cloud frames L and L+1 shown in the upper part of Figure 24, and generates a composite point cloud frame M shown in the lower left of Figure 24.

[0246] 24 is a composite point cloud frame in which the upper point cloud data is point cloud data acquired from point cloud frame L+1 and the lower point cloud data is point cloud data acquired from point cloud frame L. As is clear from the figure, the position of the subject (oncoming vehicle 20) is shifted vertically.

[0247] The composite point cloud frame corrector 182 in the sensor detection information synchronization processor 152 performs the correction process for this subject position shift. As previously described with reference to Fig. 9 etc., the point cloud frame L is a point cloud frame having a timestamp that matches or is closest to the timestamp (C)ts of the camera frame C, which is the shooting start timing of the camera frame C that is the synchronization process target.

[0248] In order to perform a synthesis process (fusion process) including subject matching between the camera frame C and the synthesized point cloud frame, it is preferable to use a camera frame and a point cloud frame with closer time stamps. Therefore, the synthesized point cloud frame correction unit 182 sets the point cloud frame L+1 as the correction target, using the point cloud frame L, which has a time stamp closer to the time stamp (C)ts of the camera frame C, which is the synchronization process target, as the reference. In other words, a correction process is performed to match the subject position in the point cloud frame L+1 with the subject position in the point cloud frame L.

[0249] During this subject position correction process, the composite point cloud frame correction unit 182 calculates the subject shift amount (dx) between point cloud frame L and point cloud frame L+1, and uses the calculated subject shift amount (dx) to perform correction processing to move the pixel position of point cloud frame L+1 within the composite point cloud frame M.

[0250] The formula for calculating the subject shift amount (dx) previously explained with reference to Figure 20 is: dx = (v x t x w) / (2 x L x tan(θ / 2)) (Equation 2) The above (Equation 2) is a formula for calculating the subject shift amount (dx) when only the LiDAR or the subject is moving at a speed (v).

[0251] In this example, as previously described with reference to Figure 22, both the vehicle 10 equipped with LiDAR 112 and the oncoming vehicle 20, which is the subject, are traveling at speeds Vx (+10 m / s) and Vy (-15 m / s), respectively.

[0252] In this case, the parameter v (v: vehicle movement speed (unit: (m / s))) in the above (Equation 2) is set to the relative speed between the vehicle 10 equipped with the LiDAR 112 and the oncoming vehicle 20, which is the subject. That is, the subject deviation amount (dx) is calculated using the following (Equation 3), in which the parameter v in the above (Equation 2) is changed to v = Vx - Vy.

[0253] dx=((Vx-Vy)×t×w) / (2×L×tan(θ / 2)) (Formula 3)

[0254] The parameters in the above (Equation 3) are as follows: Vx: moving speed of the LiDAR (LiDAR-equipped vehicle) (unit: (m / s)) Vy: moving speed of the subject (unit: (m / s)) t: time interval between point cloud frames continuously generated by the LiDAR (unit: (sec)) w: number of horizontal pixels of the LiDAR-generated point cloud frame (unit: pixels) L: distance from the LiDAR to the subject (unit: m) (distance in the vertical direction of the LiDAR) θ: angle of view of the LiDAR lens (unit: rad)

[0255] Among these parameters, Vx (= the movement speed of the LiDAR (LiDAR-equipped vehicle)) can utilize the results of analysis of vehicle speed information and steering information input from the vehicle control unit 160 via CAN (Controller Area Network) 120, which is a communication network connected to each component of the vehicle 10, as well as input information from the camera 111 and other sensor units 110. Vy (= the movement speed of the subject) and L (= the distance from the LiDAR to the subject) can utilize the results of analysis of input information from the sensor unit 110.

[0256] For t (= the time interval between point cloud frames continuously generated by LiDAR), w (= the number of horizontal pixels in the LiDAR-generated point cloud frame), and θ (the lens angle of view of the LiDAR), information stored in advance in the memory of the information processing device 100 or information stored in the memory of LiDAR 112 can be used.

[0257] 22, for example, Vx = +10 m / s, Vy = -15 m / s, so (Vx - Vy) = 10 - (-15) = 25. That is, 25 m / s, which is the relative speed between the vehicle 10 equipped with the LiDAR 112 and the oncoming vehicle 20 that is the subject, is substituted for (Vx - Vy) in the above (Equation 3).

[0258] The composite point cloud frame correction unit 182 calculates the subject shift amount (dx) in the composite point cloud frame according to the above (Equation 3), and performs correction processing to move the pixel position of point cloud frame L+1 within the composite point cloud frame M using the calculated subject shift amount (dx).

[0259] The corrected combined point cloud frame M shown in the lower right of FIG. 24 is the “corrected combined point cloud frame M” generated as a result of correction by the combined point cloud frame correction unit 182 .

[0260] The "corrected composite point cloud frame M" is a composite point cloud frame generated by performing a correction process to move the subject position in the upper point cloud frame L+1 to the left by the number of pixels corresponding to the subject displacement (dx) calculated according to the above (Equation 3). Through this correction process, the subject in the corrected composite point cloud frame becomes a subject in which the displacement between the two point cloud frames to be composited has been eliminated.

[0261] The sensor detection information synchronization processing unit 152 of this embodiment shown in Figure 18 outputs this corrected composite point cloud frame M generated by the composite point cloud frame correction unit 182 and the camera frame C to be synchronized to the sensor detection information analysis unit 153.

[0262] As described above, the corrected composite point cloud frame M is a composite point cloud frame generated using the point cloud frame L as a reference frame, which has a timestamp that matches or is close to the frame-by-frame timestamp (C)ts of the camera frame C to be synchronized. In other words, the camera frame C and the corrected composite point cloud frame M are frames composed of subject data observed at approximately the same time and from approximately the same direction, and the subject positions in each frame are approximately the same.

[0263] As a result, the sensor detection information analysis unit 153 can perform analysis processing of the camera frame C and the corrected composite point cloud frame M, as well as fusion processing (combination processing) including subject matching, more efficiently and with higher accuracy.

[0264] 6. Composite Point Cloud Frame Correction Processing Sequence Executed by the Information Processing Device of the Present Disclosure Next, a composite point cloud frame correction processing sequence executed by the information processing device of the present disclosure will be described.

[0265] In the information processing device 100b of this second embodiment having the configuration described with reference to Figure 18, as described above, the composite point cloud frame correction unit 182 in the sensor detection information synchronization processing unit 152 shown in Figure 18 performs image correction processing to eliminate positional shifts when a positional shift of the subject occurs in the composite point cloud frame generated by the composite point cloud frame generation unit 181.

[0266] The sensor detection information synchronization processing unit 152 shown in FIG. 18 outputs the corrected composite point cloud frame without subject position shift to the sensor detection information analysis unit 153 together with the camera frame to be synchronized.

[0267] The overall sequence of processing executed by the information processing device 100b of the second embodiment is the processing according to the flowchart previously shown in Fig. 16. However, in the processing of step S105 in the flow shown in Fig. 16, that is, in the processing of generating a composite point cloud frame (= a composite point cloud frame synchronized with a camera image frame), the sequence is one in which a correction process for a subject position shift is executed.

[0268] The process executed by the information processing device 100b of the second embodiment will be described with reference to Fig. 25. In step S105 in the flow shown in Fig. 16, the information processing device 100b of the second embodiment executes two process steps shown in Fig. 25, namely, a process of step S105a and a process of step S105b. These processes will be described below.

[0269] (Step S105a) In step S105a, the information processing device executes the following process.

[0270] The point cloud data after the point cloud data for which the point cloud frame constituent part unit timestamp selected in step S104 is set is obtained from multiple consecutive point cloud frames and synthesized to generate a new synthesized point cloud frame (= synthesized point cloud frame synchronized with the camera image frame).

[0271] This process is executed by the composite point cloud frame generation unit 181 in the sensor detection information synchronization processing unit 152 of the data processing unit 150 of the information processing device 100b shown in FIG.

[0272] A specific example of this process will be described with reference to the example shown in Fig. 21 described above. Assume that the point cloud frame constituent part unit timestamp selected in step S104 is the point cloud frame constituent part unit timestamp (L)ts(u) of point cloud frame L shown in Fig. 21. In this case, in step S105a, point cloud data subsequent to the point cloud data for which the point cloud frame constituent part unit timestamp (L)ts(u) of point cloud frame L shown in Fig. 21 is set is obtained from point cloud frame L and the subsequent point cloud frame L+1 and combined to generate a new combined point cloud frame (= combined point cloud frame synchronized with the camera image frame).

[0273] That is, in the example shown in FIG. 21 , the composite point cloud frame generation unit 181 of the sensor detection information synchronization processing unit 152 selects a point cloud frame L including a timestamp that matches or is closest to the timestamp (C)ts of camera frame C, and the subsequent point cloud frame L+1, and generates a composite point cloud frame M by combining the point cloud data (a) in point cloud frame L, (L)ts(u) to (L)ts(e), i.e., point cloud data set with timestamps in the range of (C)ts to (L)ts(e), and (b) in point cloud frame L+1, (L+1)ts(s) to (L+1)ts(v), i.e., point cloud data set with timestamps in the range of (L+1)ts(s) to (C)ts+(1 / Q).

[0274] The composite point cloud frame M generated here is a point cloud frame synchronized with the camera frame C. That is, in step S105, a process is executed to sequentially generate composite point cloud frames at a frame rate P (fps) synchronized with the frame rate P (fps) of the camera frame.

[0275] (Step S105b) Next, the information processing device executes the following process in step S105b.

[0276] The composite point cloud frame generation unit 181 in the sensor detection information synchronization processing unit 152 corrects the positional shift of the subject in the composite point cloud frame generated in step S105a and generates a composite point cloud frame (= a composite point cloud frame synchronized with the camera frame).

[0277] This process is executed by the composite point cloud frame correction unit 182 in the sensor detection information synchronization processing unit 152.

[0278] A specific example of this process will be described with reference to Fig. 21. It is assumed that the composite point cloud frame generated by the composite point cloud frame generation unit 181 in the sensor detection information synchronization processing unit 152 in step S105a is composite point cloud frame M shown in the lower left of Fig. 21.

[0279] The composite point cloud frame M shown in the lower left of Figure 21 is a composite point cloud frame in which the upper point cloud data is point cloud data acquired from point cloud frame L+1 and the lower point cloud data is point cloud data acquired from point cloud frame L. As is clear from the figure, the positions of the subjects (trees) are shifted vertically.

[0280] In step S105b, the composite point cloud frame correction unit 182 in the sensor detection information synchronization processing unit 152 executes a correction process for this subject position shift.

[0281] The composite point cloud frame correction unit 182 corrects the point cloud frame L+1 using the point cloud frame L having a timestamp close to the timestamp (C)ts of the camera frame C to be synchronized as a reference. In other words, correction processing is performed to match the subject position in the point cloud frame L+1 with the subject position in the point cloud frame L.

[0282] In this subject position correction process, the composite point cloud frame corrector 182 calculates the subject displacement amount (dx) according to (Equation 2) or (Equation 3) previously described with reference to FIG. 20. Next, using the calculated subject displacement amount (dx), correction processing is performed to move the pixel position of point cloud frame L+1 within composite point cloud frame M. As a result, a corrected composite point cloud frame M shown in the lower right of FIG. 21 is generated, for example.

[0283] The "corrected composite point cloud frame M" is a composite point cloud frame generated by performing a correction process to move the subject position in the upper point cloud frame L+1 to the left by a number of pixels equivalent to the subject displacement amount (dx) calculated according to the above-mentioned (Equation 2). Through this correction process, the subject in the corrected composite point cloud frame becomes a subject in which the displacement between the two point cloud frames to be composited has been eliminated.

[0284] In the second embodiment, the processes of steps S105a and S105b are executed by the composite point cloud frame generation unit 181 and the composite point cloud frame correction unit 182 in the sensor detection information synchronization processing unit 152 shown in Fig. 18. Then, the process proceeds to step S106 described above in the flowchart of the first embodiment. That is, the sensor detection information synchronization processing unit 152 of the present embodiment shown in Fig. 18 outputs the corrected composite point cloud frame M generated by the composite point cloud frame correction unit 182 and the camera frame C to be synchronized to the sensor detection information analysis unit 153.

[0285] As described above, the corrected composite point cloud frame M is a composite point cloud frame generated using the point cloud frame L as a reference frame, which has a timestamp that matches or is close to the frame-by-frame timestamp (C)ts of the camera frame C to be synchronized. In other words, the camera frame C and the corrected composite point cloud frame M are frames composed of subject data observed at approximately the same time and from approximately the same direction, and the subject positions in each frame are approximately the same.

[0286] As a result, the sensor detection information analysis unit 153 can perform analysis processing of the camera frame C and the corrected composite point cloud frame M, as well as fusion processing (combination processing) including subject matching, more efficiently and with higher accuracy.

[0287] 7. Hardware Configuration Example of Information Processing Device Disclosed Hereinafter, a hardware configuration example of an information processing device disclosed herein will be described with reference to Fig. 26 . The information processing device is mounted inside a vehicle 10. The hardware configuration shown in Fig. 26 is an example of the hardware configuration of an information processing device inside the vehicle 10. The hardware configuration shown in Fig. 26 will be described.

[0288] The CPU (Central Processing Unit) 301 functions as a data processing unit that executes various processes according to programs stored in the ROM (Read Only Memory) 302 or the storage unit 308. For example, it executes processes according to the sequences described in the above-mentioned embodiments. The RAM (Random Access Memory) 303 stores programs and data executed by the CPU 301. The CPU 301, ROM 302, and RAM 303 are interconnected by a bus 304.

[0289] The CPU 301 is connected to an input / output interface 305 via a bus 304, and the input / output interface 305 is connected to an input unit 306 including various switches, a touch panel, a microphone, and further a user input unit and a unit for acquiring status data from various sensors 321 such as a camera and LiDAR, and an output unit 307 including a display, a speaker, etc. The output unit 307 also outputs drive information to a drive unit 322 of the vehicle.

[0290] The CPU 301 receives commands and status data from an input unit 306, executes various processes, and outputs the processing results to, for example, an output unit 307. A storage unit 308 connected to the input / output interface 305 is formed of, for example, a hard disk, and stores programs executed by the CPU 301 and various data. A communication unit 309 functions as a transmitter / receiver for data communication via a network such as the Internet or a local area network, and communicates with external devices. In addition to the CPU, the system may also include a GPU (Graphics Processing Unit) as a dedicated processing unit for image information input from a sensor such as a camera.

[0291] A drive 310 connected to the input / output interface 305 drives removable media 311 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory such as a memory card, and executes recording or reading of data.

[0292] 8. Example of the overall configuration of a vehicle equipped with an information processing device according to the present disclosure Next, an example of the overall configuration of a vehicle equipped with an information processing device according to the present disclosure will be described.

[0293] 27 is a diagram showing an example of the overall configuration of a vehicle equipped with an information processing device according to the present disclosure. A vehicle 500 is equipped with a vehicle control system 511 that performs the processes executed by the information processing device according to the present disclosure and other processes.

[0294] The vehicle control system 511 is provided in the vehicle 500 and performs processing related to automated driving of the vehicle 500. This automated driving includes levels 1 to 5 of automated driving, as well as remote driving and / or remote assistance of the vehicle 500 by a remote driver. The level of automated driving may refer to the Society of Automotive Engineers (SAE) J3016™ APL2021 Levels of Driving Automation, where SAE Level 0 denotes the lowest level of automated driving and SAE Level 5 denotes the highest level of automated driving. For example, SAE Level 1 automated driving may be composed of driver assistance functions that provide steering or braking / acceleration support to the driver, and SAE Level 5 automated driving may be composed of automated driving functions that can drive the vehicle under all conditions.

[0295] The vehicle control system 511 includes a vehicle control ECU (Electronic Control Unit) 521, a communication unit 522, a map information storage unit 523, a location information acquisition unit 524, an external recognition sensor 525, an in-vehicle sensor 526, a vehicle sensor 527, a memory unit 528, a driving automation control unit 529, a DMS (Driver Monitoring System) 530, an HMI (Human Machine Interface) 531, and a vehicle control unit 532.

[0296] Two or more (or in some cases, all) of the vehicle control ECU 521, communication unit 522, map information storage unit 523, position information acquisition unit 524, external recognition sensor 525, in-vehicle sensor 526, vehicle sensor 527, memory unit 528, driving automation control unit 529, DMS 530, HMI 531, and vehicle control unit 532 are connected to each other so as to be able to communicate with each other via a communication network 541. The communication network 541 is configured by an in-vehicle communication network or bus that complies with a digital two-way communication standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay (registered trademark), or Ethernet (registered trademark), for example. In some embodiments, the communication network 541 may include two or more types of communication networks, and different types of communication networks may be used depending on the type of data being transmitted. For example, a CAN may be used for data related to vehicle control, and an Ethernet may be used for large-capacity data. In some embodiments, two or more (or in some cases, all) units of the vehicle control system 511 may be directly connected using wireless communication (e.g., relatively short-range communication) without using the communication network 541. In some embodiments, the wireless communication may use a short-range wireless communication technology. Non-limiting examples of short-range wireless communication technologies include near-field communication (NFC) and Bluetooth (registered trademark). In some embodiments, two or more (or in some cases, all) units of the vehicle control system 511 may be connected using the communication network 541 and wireless communication technology (e.g., short-range wireless communication technology).

[0297] Hereinafter, in an embodiment in which two or more units of the vehicle control system 511 communicate with each other via the communication network 541, the description of the communication network 541 will be omitted. For example, in an embodiment in which the vehicle control ECU 521 and the communication unit 522 communicate with each other via the communication network 541, it will simply be described that the vehicle control ECU 521 and the communication unit 522 communicate with each other.

[0298] The vehicle control ECU 521 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 521 controls the entire or part of the functions of the vehicle control system 511.

[0299] The communication unit 522 communicates with various devices inside the vehicle 500 (hereinafter referred to as in-vehicle devices), various devices outside the vehicle 500 (hereinafter referred to as out-vehicle devices), other vehicles, base stations, etc., and transmits and receives various types of data. In some embodiments, the communication unit 522 may communicate using multiple communication technologies.

[0300] A non-limiting example of communication between the communication unit 522 and an external device will now be briefly described. In some embodiments, the communication unit 522 may communicate with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using wireless communication technology. Non-limiting examples of wireless communication technology include 5G (fifth generation mobile communication system), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc. Examples of external networks with which the communication unit 522 can communicate include the Internet, a cloud network, or a network specific to a carrier. The communication technology used by the communication unit 522 to communicate with the external network is not particularly limited as long as it is a wireless communication technology that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0301] In some embodiments, the communication unit 522 may use P2P (Peer To Peer) technology to communicate with a terminal located near the vehicle. The terminal located near the vehicle may be, for example, a terminal worn by a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location, such as a store, and / or a Machine Type Communication (MTC) terminal. In some embodiments, the communication unit 522 may perform Vehicle to Everything (V2X) communication. V2X communication generally refers to communication between the vehicle and another entity. Non-limiting examples of V2X communication include vehicle-to-vehicle (V2X) communication with other vehicles, vehicle-to-infrastructure (V2X) communication with roadside devices or the like, vehicle-to-home (V2X) communication with a home, and vehicle-to-pedestrian (V2X) communication with a terminal or the like carried or worn by a pedestrian.

[0302] In some embodiments, the communication unit 522 may receive a program for updating software that controls the operation of the vehicle control system 511 from outside the vehicle 500 (e.g., over the air). In some embodiments, the communication unit 522 may receive map information, traffic information, information about the surroundings of the vehicle 500, etc. from outside the vehicle 500. In some embodiments, the communication unit 522 may transmit information about the vehicle 500, information about the surroundings of the vehicle 500, etc. to an external device or an external network. Non-limiting examples of information about the vehicle 500 that the communication unit 522 transmits to an external device or an external network include data indicating the status of the vehicle 500, recognition results by the recognition unit 573, etc. In some embodiments, the communication unit 522 may communicate with a vehicle emergency notification system. Non-limiting examples of a vehicle emergency notification system include eCall, etc.

[0303] In some embodiments, the communication unit 522 may receive electromagnetic waves transmitted by a road traffic information communication system. In some embodiments, the electromagnetic waves may be transmitted using a radio beacon, an optical beacon, FM multiplex broadcasting, or the like.

[0304] A non-limiting example of communication with the in-vehicle devices that the communication unit 522 can perform will be briefly described below. In some embodiments, the communication unit 522 may communicate with the in-vehicle devices using wireless communication. For example, in some embodiments, the communication unit 522 may communicate with the in-vehicle devices using wireless communication technology that enables bidirectional digital communication at a predetermined communication speed or higher. Non-limiting examples of wireless communication technology include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). Without being limited thereto, the communication unit 522 may also communicate with the in-vehicle devices using wired communication (in addition to or as an alternative to wireless communication). For example, in some embodiments, the communication unit 522 may communicate with the in-vehicle devices using wired communication via a cable connected to a connection terminal (not shown). In some embodiments, the communication unit 522 may communicate with the in-vehicle devices using wired communication technology that enables bidirectional digital communication at a predetermined communication speed or higher. Non-limiting examples of wired communication technologies include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI)®, and Mobile High-Definition Link (MHL).

[0305] Here, the in-vehicle devices refer to, for example, devices inside the vehicle 500 that are not connected to the communication network 541. The in-vehicle devices are divided into devices that constitute the vehicle control system 511 and devices that do not constitute the vehicle control system 511. Non-limiting examples of in-vehicle devices that do not constitute the vehicle control system 511 include mobile devices and wearable devices carried by users of the vehicle 500 (for example, the driver or passengers), and information devices that are temporarily installed inside the vehicle 500. These devices can, for example, be moved outside the vehicle 500 and become external devices.

[0306] The map information storage unit 523 stores maps acquired from an external device or an external network and / or maps created by the vehicle 500. For example, the map information storage unit 523 may store a three-dimensional high-precision map, a global map that is less accurate than a high-precision map and covers a wide area, or the like.

[0307] The high-precision map may be, for example, a dynamic map, a point cloud map, a vector map, etc. The dynamic map may be, for example, a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and may be provided to the vehicle 500 from an external server or the like. The point cloud map may be a map composed of a point cloud (point cloud data). The vector map may be, for example, a map adapted to automated driving by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.

[0308] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created by the vehicle 500 as a map for matching with a local map (described later) based on sensing results from the camera 551, radar 552, LiDAR 553, etc., and stored in the map information storage unit 523. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters related to the planned route along which the vehicle 500 will travel may be acquired from the external server or the like in order to reduce communication capacity.

[0309] The position information acquisition unit 524 acquires position information of the vehicle 500. The acquired position information may be provided to the driving automation control unit 529. In some embodiments, the position information acquisition unit 524 may receive GNSS (Global Navigation Satellite System) signals from GNSS satellites. In some embodiments, the position information acquisition unit 524 may receive signals from beacons or the like.

[0310] The external recognition sensor 525 includes various sensors used to recognize the situation outside the vehicle 500, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 511. The type and number of sensors included in the external recognition sensor 525 are arbitrary.

[0311] In some embodiments, the external recognition sensor 525 may include a camera 551, a radar 552, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 553, and an ultrasonic sensor 554. Without being limited to this, the external recognition sensor 525 may be configured to include one or more types of sensors selected from the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554. The number of cameras 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554 is not particularly limited as long as the number is a number that can be realistically installed in the vehicle 500. Furthermore, the types of sensors included in the external recognition sensor 525 are not limited to this example, and the external recognition sensor 525 may include other types of sensors. Examples of sensing areas of each sensor included in the external recognition sensor 525 will be described later.

[0312] The camera 551 may use any suitable imaging method. In some embodiments, the camera 551 may use an imaging method capable of distance measurement. Non-limiting examples of cameras using imaging methods capable of distance measurement include a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. Alternatively, the camera 551 may simply acquire an image without distance measurement.

[0313] In some embodiments, the external recognition sensor 525 may include an environmental sensor for detecting characteristics of the environment around the vehicle 500. Non-limiting examples of environmental characteristics that may be detected include weather, climate, brightness, etc. In some embodiments, the environmental sensor may include various sensors such as a rain sensor, a fog sensor, a sunlight sensor, a snow sensor, an illuminance sensor, etc.

[0314] In some embodiments, the external recognition sensor 525 may include a microphone used to detect sounds around the vehicle 500 and the location of sound sources.

[0315] The interior sensor 526 includes various sensors for detecting information about the interior of the vehicle 500, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 511. The types and number of the various sensors included in the interior sensor 526 are not particularly limited as long as they are of the types and number that can be realistically installed in the vehicle 500.

[0316] In some embodiments, the interior sensor 526 may include one or more sensors selected from the group consisting of a camera, radar, a seat sensor, a microphone, and a biometric sensor. In some embodiments, the camera included in the interior sensor 526 may use an imaging method capable of measuring distances. Non-limiting examples of cameras using imaging methods capable of measuring distances include a Time of Flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. Without being limited thereto, the camera included in the interior sensor 526 may simply acquire captured images without measuring distances. The biometric sensor included in the interior sensor 526 may be provided, for example, on a seat or a steering wheel, and may detect various types of biometric information of the user.

[0317] The vehicle sensor 527 includes various sensors for detecting the state of the vehicle 500, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 511. The types and number of the various sensors included in the vehicle sensor 527 are not particularly limited as long as they are of the types and number that can be realistically installed on the vehicle 500.

[0318] In some embodiments, the vehicle sensors 527 may include a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and / or an inertial measurement unit (IMU) that integrates these. In some embodiments, the vehicle sensors 527 may include a steering angle sensor that detects the steering angle of a steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation (e.g., pedal force, pedal stroke), and / or a brake sensor that detects the amount of brake pedal operation (e.g., pedal force, pedal stroke). In some embodiments, the vehicle sensors 527 may include a rotation sensor that detects the number of rotations of an engine or a motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and / or a wheel speed sensor that detects the rotation speed of a wheel. In some embodiments, the vehicle sensors 527 may include a battery sensor that detects the remaining battery level and temperature, and / or an impact sensor that can detect an external impact.

[0319] The storage unit 528 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Non-limiting examples of storage media include an EEPROM (Electrically Erasable Programmable Read Only Memory), a RAM (Random Access Memory), and / or a magnetic storage device such as a HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 528 stores various programs and data used by one or more (or in some cases, all) units of the vehicle control system 511. In some embodiments, the memory unit 528 may include an Event Data Recorder (EDR) or a Data Storage System for Automated Driving (DSSAD) and may store information about the vehicle 500 before and after an event such as an accident, or information acquired by the in-vehicle sensors 526.

[0320] The driving automation control unit 529 controls the driving automation function of the vehicle 500. In some embodiments, the driving automation control unit 529 may include an analysis unit 561, an action planning unit 562, and an operation control unit 563.

[0321] The analysis unit 561 performs an analysis process of the vehicle 500 and / or the surrounding situation. The analysis unit 561 includes a self-position estimation unit 571, a sensor fusion unit 572, and a recognition unit 573.

[0322] In some embodiments, the self-position estimation unit 571 may estimate the self-position of the vehicle 500 based on sensor data from the external recognition sensor 525 and a high-precision map stored in the map information storage unit 523. For example, the self-position estimation unit 571 may generate a local map based on the sensor data from the external recognition sensor 525 and estimate the self-position of the vehicle 500 by matching the local map with the high-precision map. The position of the vehicle 500 may be based on, for example, the center of the rear wheel pair axle.

[0323] In some embodiments, the local map may be a three-dimensional high-precision map, an occupancy grid map, or the like created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map may be, for example, the point cloud map described above. The occupancy grid map may be a map obtained by dividing a three-dimensional or two-dimensional space around the vehicle 500 into grids of a predetermined size and indicating the occupancy status of objects on a grid-by-grid basis. The occupancy status of an object may be indicated, for example, by the presence or absence or probability of the object. In some embodiments, the local map may also be used, for example, in detection processing and / or recognition processing of the situation outside the vehicle 500 by the recognition unit 573.

[0324] In some embodiments, the self-position estimation unit 571 may estimate the self-position of the vehicle 500 based on the position information acquired by the position information acquisition unit 524 and / or sensor data from the vehicle sensor 527.

[0325] The sensor fusion unit 572 performs sensor fusion processing to obtain information by combining multiple different types of sensor data (for example, image data supplied from the camera 551 and sensor data supplied from the radar 552). Methods for combining different types of sensor data include, but are not limited to, compounding, integration, fusion, and association.

[0326] The recognition unit 573 executes a detection process for detecting the situation outside the vehicle 500 and / or a recognition process for recognizing the situation outside the vehicle 500 .

[0327] For example, the recognition unit 573 may perform detection processing and / or recognition processing of the external situation of the vehicle 500 based on information from the external recognition sensor 525, information from the self-position estimation unit 571, information from the sensor fusion unit 572, etc.

[0328] Specifically, for example, the recognition unit 573 may perform detection processing and / or recognition processing of objects around the vehicle 500. The object detection processing may be, for example, processing to detect the presence or absence, size, shape, position, movement, etc. of an object. The object recognition processing may be, for example, processing to recognize attributes such as the type of object, or to identify a specific object. The detection processing and the recognition processing are not necessarily clearly separated, and there may be at least a partial overlap.

[0329] In some embodiments, the recognition unit 573 may detect objects around the vehicle 500 by performing clustering to classify a point cloud based on sensor data from the radar 552 and / or the LiDAR 553, etc. into clusters of points. This makes it possible to detect the presence, size, shape, and position of objects around the vehicle 500.

[0330] In some embodiments, the recognition unit 573 may detect the movement of objects around the vehicle 500 by tracking the movement of clusters of the point cloud classified by clustering. This makes it possible to detect the speed and / or the traveling direction (movement vector) of the objects around the vehicle 500.

[0331] In some embodiments, the recognition unit 573 may detect and / or recognize vehicles (including bicycles), people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data provided by the camera 551. In some embodiments, the recognition unit 573 may recognize the type of object around the vehicle 500 by performing recognition processing such as semantic segmentation.

[0332] In some embodiments, the recognition unit 573 may perform a recognition process of traffic rules around the vehicle 500 based on the map stored in the map information storage unit 523, the result of estimation of the self-position by the self-position estimation unit 571, and / or the result of recognition of objects around the vehicle 500 by the recognition unit 573. Through this process, the recognition unit 573 may recognize the position and / or state of traffic lights, the contents of traffic signs and / or road markings, the contents of traffic regulations, and / or lanes that can be traveled, etc.

[0333] In some embodiments, the recognition unit 573 may perform recognition processing of the environment around the vehicle 500. In some embodiments, the recognition unit 573 may recognize weather characteristics (temperature, humidity, brightness) and / or road surface conditions, etc.

[0334] The behavior planning unit 562 creates a behavior plan for the vehicle 500. For example, the behavior planning unit 562 may create a behavior plan by performing route planning and route tracking.

[0335] In some embodiments, path planning may include global path planning and local path planning. Global path planning may include a process of planning a rough route from a start to a goal. Local path planning, also referred to as trajectory planning, may include generating a trajectory that allows the vehicle 500 to proceed safely and smoothly along a planned route in the vicinity of the vehicle 500, taking into account the motion characteristics of the vehicle 500, the presence of any obstacles, and the like.

[0336] In some embodiments, path following may be a planning of an operation for safely and accurately traveling along a route planned by the route plan within a planned time. The action planning unit 562 may, for example, calculate a target speed and / or a target angular velocity of the vehicle 500 based on the result of the path following process.

[0337] The operation control unit 563 controls the operation of the vehicle 500 in order to realize the action plan created by the action planning unit 562 .

[0338] For example, in some embodiments, the operation control unit 563 may control a steering control unit 581, a brake control unit 582, and / or a drive control unit 583 included in a vehicle control unit 532 (described later) to perform lateral vehicle motion control and / or longitudinal vehicle motion control so that the vehicle 500 proceeds along the trajectory calculated by the trajectory planning. For example, the operation control unit 563 may perform control (e.g., lateral vehicle motion control, longitudinal vehicle motion control) for one or more driver assistance functions and / or driving automation. Non-limiting examples of driver assistance functions include collision avoidance or impact mitigation, following distance control (e.g., control to maintain a specific distance from a vehicle traveling in front of the vehicle 500), vehicle speed control (e.g., control to maintain a specific speed), vehicle collision warning, and lane departure warning. Non-limiting examples of driving automation include driving without operation by a driver or a remote driver.

[0339] In some embodiments, the DMS 530 may perform a driver authentication process and / or a driver state recognition process based on sensor data from the in-vehicle sensors 526 and / or input data input to the HMI 531 (described later), etc. Non-limiting examples of the driver state that may be recognized include physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, posture, etc.

[0340] In some embodiments, DMS 530 may perform authentication processing of a user other than the driver (e.g., a passenger) and / or recognition processing of the state of the user. In some embodiments, DMS 530 may perform recognition processing of the interior situation of vehicle 500 based on sensor data from interior sensors 526. Non-limiting examples of characteristics of the interior situation of vehicle 500 that may be recognized include temperature, humidity, brightness, odor, etc.

[0341] The HMI 531 receives various data, instructions, etc. as input, and presents the various data to the user.

[0342] The input of data to the HMI 531 will now be briefly described. The HMI 531 includes an input device through which a person inputs data, instructions, etc. The HMI 531 generates an input signal based on the data, instructions, etc. input via the input device and supplies the signal to one or more (or in some cases, all) units of the vehicle control system 511. In some embodiments, the HMI 531 may include a touch panel, buttons, switches, and / or levers as input devices. Without being limited thereto, the HMI 531 may also include an input device that allows information to be input by means other than manual operation, such as voice or gestures. In some embodiments, the HMI 531 may include an input device such as an infrared and / or radio remote control device or an externally connected device that can operate the vehicle control system 511. Non-limiting examples of externally connected devices include mobile devices (e.g., smartphones) and wearable devices (e.g., smart watches).

[0343] The presentation of data by the HMI 531 will be briefly described. The HMI 531 generates visual information, auditory information, and / or tactile information for the user and / or a person outside the vehicle 500. The HMI 531 may also perform output control, controlling the output, output content, output timing, and / or output method of each piece of generated information. Non-limiting examples of visual information that can be generated and output by the HMI 531 include information displayed by images or lights, such as an operation screen, a status display of the vehicle 500, a warning display, and a monitor image showing the situation around the vehicle 500. Non-limiting examples of auditory information that can be generated and output by the HMI 531 include voice guidance, warning sounds, warning messages, etc. Non-limiting examples of tactile information that can be generated and output by the HMI 531 include information imparted to the user's sense of touch by force, vibration, movement, etc.

[0344] In some embodiments, the HMI 531 may include, as an output device capable of outputting visual information, a display device that presents visual information by displaying an image itself or a projector device that presents visual information by projecting an image. In some embodiments, the display device may be, in addition to or instead of a typical display device, a device that displays visual information within the user's field of view, such as a head-up display, a see-through display, or a wearable device with an augmented reality (AR) function. In some embodiments, the HMI 531 may include, as an output device capable of outputting visual information, a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 500.

[0345] In some embodiments, the HMI 531 may include an audio speaker, headphones, or earphones as output devices capable of outputting auditory information.

[0346] In some embodiments, the HMI 531 may include a haptic element using haptic technology as an output device capable of outputting tactile information. The haptic element may be provided on a part of the vehicle 500 that the user comes into contact with, such as a steering wheel or a seat.

[0347] The vehicle control unit 532 controls one or more (or in some cases, all) units of the vehicle 500. The vehicle control unit 532 includes a steering control unit 581, a brake control unit 582, a drive control unit 583, a body system control unit 584, a light control unit 585, and a horn control unit 586.

[0348] The steering control unit 581 detects and / or controls the state of the steering system of the vehicle 500. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 581 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0349] The brake control unit 582 detects and / or controls the state of the brake system of the vehicle 500. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 582 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0350] The drive control unit 583 detects and / or controls the state of the drive system of the vehicle 500. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 583 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0351] The body system control unit 584 detects and / or controls the states of the body system systems of the vehicle 500. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 584 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0352] The light control unit 585 detects and / or controls the states of various lights of the vehicle 500. Non-limiting examples of lights that can be controlled by the light control unit 585 include headlights, backlights, fog lights, turn signals, brake lights, projector lights, bumper indicators, etc. The light control unit 585 includes a light ECU that controls the lights, an actuator that drives the lights, etc.

[0353] Horn control unit 586 detects and / or controls the state of the car horn of vehicle 500. Horn control unit 586 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0354] Fig. 28 is a diagram showing an example of a sensing area by the camera 551, radar 552, LiDAR 553, ultrasonic sensor 554, etc. of the external recognition sensor 525 in Fig. 27. Fig. 28 schematically shows a state of the vehicle 500 as seen from above.

[0355] Sensing area 601F and sensing area 601B show examples of sensing areas of ultrasonic sensors 554. Sensing area 601F (e.g., sensing area of ​​multiple ultrasonic sensors 554) covers the periphery of the front end of vehicle 500. Sensing area 601B (e.g., sensing area of ​​multiple ultrasonic sensors 554) covers the periphery of the rear end of vehicle 500.

[0356] The sensing results in the sensing area 601F and / or the sensing area 601B may be used, for example, for parking assistance for the vehicle 500.

[0357] Sensing area 602F, sensing area 602B, sensing area 602L, and sensing area 602R show examples of sensing areas of the short-range or medium-range radar 552. Sensing area 602F covers a position farther in front of the vehicle 500 than sensing area 601F. Sensing area 602B covers a position farther behind the vehicle 500 than sensing area 601B. Sensing area 602L covers the periphery behind the left side of the vehicle 500. Sensing area 602R covers the periphery behind the right side of the vehicle 500.

[0358] The sensing results in the sensing area 602F may be used, for example, to detect vehicles, pedestrians, and the like present in front of the vehicle 500. The sensing results in the sensing area 602B may be used, for example, for a collision prevention function behind the vehicle 500. The sensing results in the sensing area 602L and / or the sensing area 602R may be used, for example, to detect one or more objects in blind spots on the left and / or right sides of the vehicle 500.

[0359] Sensing area 603F, sensing area 603B, sensing area 603L, and sensing area 603R show examples of sensing areas sensed by camera 551. Sensing area 603F covers a position farther in front of vehicle 500 than sensing area 602F. Sensing area 603B covers a position farther in the rear of vehicle 500 than sensing area 602B. Sensing area 603L covers the periphery on the left side of vehicle 500. Sensing area 603R covers the periphery on the right side of vehicle 500.

[0360] The sensing results in the sensing area 603F may be used, for example, for recognizing traffic lights and traffic signs, a lane departure prevention assist system, or an automatic headlight control system. The sensing results in the sensing area 603B may be used, for example, for parking assistance and / or a surround view system. The sensing results in the sensing area 603L and / or the sensing area 603R may be used, for example, for a surround view system.

[0361] Sensing area 604 shows an example of the sensing area of ​​LiDAR 553. Sensing area 604 covers a position farther ahead of vehicle 500 than sensing area 603F. On the other hand, sensing area 604 has a narrower range in the left-right direction of vehicle 500 than sensing area 603F.

[0362] The sensing results in the sensing area 604 may be used to detect objects such as surrounding vehicles, for example.

[0363] A sensing area 605 shows an example of the sensing area of ​​the long-range radar 552. The sensing area 605 covers a position further ahead of the vehicle 500 than the sensing area 604. On the other hand, the sensing area 605 has a narrower range in the left-right direction of the vehicle 500 than the sensing area 604.

[0364] The sensing results in the sensing area 605 may be used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.

[0365] In some embodiments, the sensing area of ​​each sensor of the external recognition sensor 525 (for example, the camera 551, the radar 552, the LiDAR 553, and the ultrasonic sensor 554) may have various configurations other than the configuration shown in Fig. 28. Specifically, in some embodiments, the ultrasonic sensor 554 may also sense the sides of the vehicle 500, and the LiDAR 553 may sense the rear of the vehicle 500. Furthermore, the installation position of each sensor is not limited to the above-mentioned examples. Furthermore, the number of each sensor may be one or more.

[0366] [9. Summary of the Configuration of the Present Disclosure] The embodiments of the present disclosure have been described above in detail with reference to specific examples. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of examples and should not be interpreted as being limited. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0367] The technology disclosed in this specification can be configured as follows: (1) An information processing device having a data processing unit that receives as input first sensor frames generated by a first sensor at a frame rate P (fps) and second sensor frames generated by a second sensor at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the consecutively input second sensor frames to generate a synthesized second sensor frame synchronized with the frame rate of the first sensor frames.

[0368] (2) The information processing device described in (1) above, wherein the data processing unit performs a comparison process between a second sensor frame component unit timestamp, which is a timestamp set for each component unit of the second sensor frame, and a first sensor frame unit timestamp, which is a timestamp set for each frame unit of the first sensor frame, and generates the composite second sensor frame by combining the component data of a second sensor frame L+1 that follows the second sensor frame L and the second sensor frame L having the second sensor frame component unit timestamp closest to the first sensor frame unit timestamp.

[0369] (3) The information processing device according to (1) or (2), wherein the data processing unit has: a sensor detection information synchronization processing unit that synthesizes the plurality of second sensor frames to generate a synthesized second sensor frame synchronized with the frame rate of the first sensor frame; and outputs the first sensor frame and the synthesized second sensor frame synchronized with the first sensor frame to a sensor detection information analysis unit; and a sensor detection information analysis unit that inputs the first sensor frame and the synthesized second sensor frame from the sensor detection information synchronization processing unit and performs analysis processing on each input sensor frame.

[0370] (4) The information processing device according to any one of (1) to (3), wherein the first sensor is a camera, the second sensor is a LiDAR, and the data processing unit inputs a camera frame, which is an image generated by the camera at a frame rate P (fps), and a point cloud frame having point cloud data indicating an object position generated by the LiDAR at a frame rate Q (fps), and synthesizes the multiple point cloud frames input continuously to generate a synthesized point cloud frame synchronized with the frame rate of the camera frame.

[0371] (5) The information processing device described in (4) above, wherein the data processing unit performs a matching process between a point cloud frame component part unit timestamp, which is a timestamp set for each component part of the point cloud frame, and a camera frame unit timestamp, which is a timestamp set for each frame of the camera frame, and generates the composite point cloud frame by combining point cloud frame L having a point cloud frame component part unit timestamp closest to the camera frame unit timestamp and point cloud data of point cloud frame L+1 that follows point cloud frame L.

[0372] (6) The information processing device according to (5), wherein the point cloud frame constituent part unit timestamp is any one of a line unit timestamp, a pixel unit timestamp, a multiple line unit timestamp, or a multiple pixel unit timestamp of the point cloud frame.

[0373] (7) The information processing device according to (5) or (6), wherein the data processing unit extracts from the point cloud frame L a point cloud data area set with a timestamp after the point cloud frame component part unit timestamp closest to the camera frame unit timestamp, and extracts from the point cloud frame L+1 a point cloud data area from the earliest point cloud frame component part unit timestamp set area to the point cloud frame component part unit timestamp set area closest to the point cloud frame component part unit timestamp closest to the camera frame unit timestamp in the point cloud frame L plus time (1 / Q), and generates the composite point cloud frame by combining the point cloud data extracted from the point cloud frame L and the point cloud data extracted from the point cloud frame L+1.

[0374] (8) The information processing device described in any one of (4) to (7), wherein the data processing unit has: a sensor detection information synchronization processing unit that synthesizes the multiple point cloud frames to generate a composite point cloud frame synchronized with the frame rate of the camera frames; and outputs the camera frames and the composite point cloud frame synchronized with the camera frames to a sensor detection information analysis unit; and a sensor detection information analysis unit that inputs the camera frames and the composite point cloud frame from the sensor detection information synchronization processing unit and performs analysis processing on each input frame.

[0375] (9) The information processing device according to (8), wherein the sensor detection information analysis unit executes the analysis process of the camera frame and the analysis process of the composite point cloud frame synchronized with the camera frame in the same cycle.

[0376] (10) The information processing device according to (8) or (9), wherein the sensor detection information analysis unit executes a matching process between an object in the camera frame and an object in the composite point cloud frame synchronized with the camera frame.

[0377] (11) An information processing device according to any one of (8) to (10), wherein the first sensor and the second sensor are sensors attached to the inside of a vehicle, and the sensor detection information analysis unit outputs the analysis results of the camera frame and the composite point cloud frame to a vehicle control unit that executes driving control of the vehicle.

[0378] (12) The information processing device described in any one of (5) to (11), wherein the data processing unit has a timestamp setting unit that executes a process of setting a camera frame unit timestamp, which is a timestamp for each frame, for the camera frame, and a process of setting a point cloud frame constituent part unit timestamp, which is a timestamp for each constituent part of the point cloud frame, for the point cloud frame.

[0379] (13) The information processing device according to any one of (8) to (12), wherein the sensor detection information synchronization processing unit has a composite point cloud frame correction unit that corrects a subject shift that occurs in a composite point cloud frame generated by combining multiple point cloud frames.

[0380] (14) The information processing device according to (13), wherein the composite point cloud frame correction unit calculates a shift amount of a subject shift that has occurred in the composite point cloud frame, and executes a correction process of moving pixel positions so as to cancel the calculated shift amount.

[0381] (15) The information processing device according to (13) or (14), wherein the composite point cloud frame correction unit executes the calculation process of the amount of deviation based on the amount of movement of the LiDAR or the amount of movement of the subject within the generation time of the plurality of point cloud frames used to generate the composite point cloud frame.

[0382] (16) The information processing device according to (15), wherein the LiDAR is a LiDAR mounted on a vehicle, and the composite point cloud frame correction unit receives a movement amount of the LiDAR from a vehicle control unit.

[0383] (17) An information processing method executed in an information processing device, in which a data processing unit inputs camera frames, which are images generated by a camera at a frame rate P (fps), and point cloud frames having point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple point cloud frames input consecutively to generate a synthesized point cloud frame synchronized with the frame rate of the camera frames.

[0384] (18) A program for executing information processing in an information processing device, the program inputting, into a data processing unit, camera frames which are images generated by a camera at a frame rate P (fps) and point cloud frames having point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and synthesizing the multiple point cloud frames input continuously to generate a synthesized point cloud frame synchronized with the frame rate of the camera frames.

[0385] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence can be installed and executed in the memory of a computer incorporated in dedicated hardware, or the program can be installed and executed on a general-purpose computer capable of executing various processes. For example, the program can be pre-recorded on a recording medium. In addition to installing the program from the recording medium to the computer, the program can also be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as an internal hard disk.

[0386] Furthermore, the various processes described in this specification may not only be executed in chronological order as described, but may also be executed in parallel or individually depending on the processing capabilities of the devices executing the processes or as needed. Furthermore, in this specification, a system refers to a logical collective configuration of multiple devices, and is not limited to devices in the same housing.

[0387] As described above, according to the configuration of one embodiment of the present disclosure, an apparatus and method are realized in which multiple sensor detection data with different frame rates are synchronized and input to an analysis unit for analysis. Specifically, for example, a data processing unit inputs camera frames, which are images generated by a camera at a frame rate P (fps), and point cloud frames containing point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and compares frame-by-frame timestamps of the camera frames with timestamps of component parts within the point cloud frames to synthesize the multiple consecutively input point cloud frames and generate a composite point cloud frame synchronized with the frame rate of the camera frames. This configuration realizes an apparatus and method in which multiple sensor detection data with different frame rates are synchronized and input to an analysis unit for analysis.

[0388] 10 Vehicle 20 Oncoming vehicle 31 Camera 32 LiDAR 100 Information processing device 110 Sensor unit 111 Camera 112 LiDAR 113 Radar 114 GPS 115 IMU 120 CAN 150 Data processing unit 151 Timestamp setting unit 152 Sensor detection information synchronization processing unit 153 Sensor detection information analysis unit 160 Vehicle control unit 181 Composite point cloud frame generation unit 182 Composite point cloud frame correction unit 301 CPU 302 ROM 303 RAM 304 Bus 305 Input / output interface 306 Input unit 307 Output unit 308 Storage unit 309 Communication unit 310 Drive 311 Removable media 321 Sensor 322 Drive unit

Claims

1. An information processing device having a data processing unit that inputs first sensor frames generated by a first sensor at a frame rate P (fps) and second sensor frames generated by a second sensor at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple consecutively input second sensor frames to generate a synthesized second sensor frame that is synchronized with the frame rate of the first sensor frames.

2. The information processing device described in claim 1, wherein the data processing unit performs a comparison process between a second sensor frame component unit timestamp, which is a timestamp set for each component unit of the second sensor frame, and a first sensor frame unit timestamp, which is a timestamp set for each frame unit of the first sensor frame, and generates the composite second sensor frame by combining the component data of a second sensor frame L+1 that follows the second sensor frame L and the second sensor frame L having the second sensor frame component unit timestamp closest to the first sensor frame unit timestamp.

3. The information processing device according to claim 1, wherein the data processing unit has: a sensor detection information synchronization processing unit that synthesizes the plurality of second sensor frames to generate a synthesized second sensor frame synchronized with the frame rate of the first sensor frame; and outputs the first sensor frame and the synthesized second sensor frame synchronized with the first sensor frame to a sensor detection information analysis unit; and a sensor detection information analysis unit that inputs the first sensor frame and the synthesized second sensor frame from the sensor detection information synchronization processing unit and performs analysis processing on each input sensor frame.

4. The information processing device of claim 1, wherein the first sensor is a camera, the second sensor is a LiDAR, and the data processing unit inputs a camera frame which is an image generated by the camera at a frame rate P (fps) and a point cloud frame having point cloud data indicating object positions generated by the LiDAR at a frame rate Q (fps), and synthesizes the multiple point cloud frames input continuously to generate a synthesized point cloud frame synchronized with the frame rate of the camera frame.

5. The information processing device according to claim 4, wherein the data processing unit performs a comparison process between a point cloud frame constituent part unit timestamp, which is a timestamp set for each constituent part of the point cloud frame, and a camera frame unit timestamp, which is a timestamp set for each frame of the camera frame, and generates the composite point cloud frame by combining point cloud data of a point cloud frame L having a point cloud frame constituent part unit timestamp closest to the camera frame unit timestamp and point cloud frame L+1 that follows point cloud frame L.

6. An information processing device according to claim 5, wherein the timestamp per unit of a point cloud frame constituent part is any one of: a timestamp per line of the point cloud frame, a timestamp per pixel, a timestamp per multiple lines, or a timestamp per multiple pixels.

7. The information processing device according to claim 5, wherein the data processing unit extracts from the point cloud frame L a point cloud data area set with a timestamp subsequent to the point cloud frame component part unit timestamp closest to the camera frame unit timestamp, and extracts from the point cloud frame L+1 a point cloud data area from the earliest point cloud frame component part unit timestamp set area to the point cloud frame component part unit timestamp set area closest to the point cloud frame component part unit timestamp closest to the camera frame unit timestamp in the point cloud frame L plus time (1 / Q), and generates the composite point cloud frame by combining the point cloud data extracted from the point cloud frame L and the point cloud data extracted from the point cloud frame L+1.

8. The information processing device according to claim 4, wherein the data processing unit has: a sensor detection information synchronization processing unit that synthesizes the multiple point cloud frames to generate a composite point cloud frame synchronized with the frame rate of the camera frames; and outputs the camera frames and the composite point cloud frame synchronized with the camera frames to a sensor detection information analysis unit; and a sensor detection information analysis unit that inputs the camera frames and the composite point cloud frame from the sensor detection information synchronization processing unit and performs analysis processing on each input frame.

9. The information processing device according to claim 8, wherein the sensor detection information analysis unit executes an analysis process of the camera frame and an analysis process of the composite point cloud frame synchronized with the camera frame in the same cycle.

10. The information processing device according to claim 8, wherein the sensor detection information analysis unit performs a matching process between an object in the camera frame and an object in the composite point cloud frame synchronized with the camera frame.

11. The information processing device described in claim 8, wherein the first sensor and the second sensor are sensors attached to the inside of a vehicle, and the sensor detection information analysis unit outputs the analysis results of the camera frame and the composite point cloud frame to a vehicle control unit that performs driving control of the vehicle.

12. The information processing device according to claim 5, wherein the data processing unit has a timestamp setting unit that executes a process of setting a camera frame unit timestamp, which is a timestamp for each frame, for the camera frame, and a process of setting a point cloud frame constituent part unit timestamp, which is a timestamp for each constituent part of the point cloud frame, for the point cloud frame.

13. The information processing device according to claim 8, wherein the sensor detection information synchronization processing unit has a composite point cloud frame correction unit that corrects subject shift that occurs in a composite point cloud frame generated by combining multiple point cloud frames.

14. An information processing device according to claim 13, wherein the composite point cloud frame correction unit calculates the amount of subject shift that has occurred in the composite point cloud frame, and executes a correction process that moves pixel positions so as to eliminate the calculated amount of shift.

15. The information processing device described in claim 13, wherein the composite point cloud frame correction unit performs the calculation process of the amount of deviation based on the amount of movement of the LiDAR or the amount of movement of the subject within the generation time of the multiple point cloud frames used to generate the composite point cloud frame.

16. The information processing device according to claim 15, wherein the LiDAR is a LiDAR mounted on a vehicle, and the composite point cloud frame correction unit receives the amount of movement of the LiDAR as input from a vehicle control unit.

17. An information processing method executed in an information processing device, in which a data processing unit inputs camera frames, which are images generated by a camera at a frame rate P (fps), and point cloud frames having point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple point cloud frames input consecutively to generate a synthesized point cloud frame synchronized with the frame rate of the camera frames.

18. A program for executing information processing in an information processing device, which program inputs into a data processing unit camera frames, which are images generated by a camera at a frame rate P (fps), and point cloud frames containing point cloud data indicating object positions generated by a LiDAR at a frame rate Q (fps) different from the frame rate P (fps), and synthesizes the multiple point cloud frames input consecutively to generate a synthesized point cloud frame synchronized with the frame rate of the camera frames.

Images