Sensor simulation system and method, storage medium, product, equipment and vehicle

Abstract

The invention relates to a sensor simulation system and method, a storage medium, a product, equipment and a vehicle. The sensor simulation system comprises a simulation engine and a data processing module, wherein the simulation engine is used for generating simulation data of a plurality of monocular cameras and timestamps of the simulation data; the data processing module is used for adding simulation data and a timestamp of the monocular camera into a message queue of the monocular camera; according to the timestamps in the plurality of message queues, acquiring a plurality of simulation data with time synchronization from the plurality of message queues; and generating multi-view simulation data according to the multiple pieces of simulation data subjected to time synchronization. According to the embodiment of the invention, the simulation of the multi-view camera is realized through the simulation of the plurality of monocular cameras, and the multi-view simulation data is generated according to the simulation data of the plurality of monocular cameras with time synchronization, so that the accuracy of the multi-view simulation data is ensured, and the simulation effect of the multi-view camera is improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a sensor simulation system, method, storage medium, product, device and vehicle. Background Art

[0002] Vehicles are often equipped with multiple cameras to capture scene and environmental information. For vehicles with intelligent driving capabilities, the data collected by these cameras is used to implement functions such as obstacle detection, distance detection, and environmental perception. Therefore, the development and verification of multi-cameras are crucial for the visual perception and positioning functions of intelligent driving. Related technologies propose simulation testing of intelligent driving systems. As key sensors in intelligent driving systems, multi-cameras also need to be fully developed and verified in a simulation environment. The realization of multi-camera simulation is a technical problem that currently needs to be solved. Summary of the Invention

[0003] The embodiments of the present application provide a sensor simulation system, method, storage medium, product, equipment and vehicle, which realize the simulation of multi-cameras, ensure the accuracy of multi-camera simulation data, and improve the simulation effect of multi-cameras, so as to at least partially solve the above-mentioned technical problems.

[0004] In order to achieve the above-mentioned purpose, according to the first aspect of the present application, a sensor simulation system is provided, comprising: a simulation engine and a data processing module; wherein the simulation engine is used to: generate simulation data of multiple monocular cameras and timestamps of the simulation data; the data processing module is used to: add the simulation data and the timestamp of the monocular camera to the message queue of the monocular camera; obtain multiple time-synchronized simulation data from the multiple message queues according to the timestamps in the multiple message queues; and generate multi-camera simulation data based on the multiple time-synchronized simulation data.

[0005] Optionally, the first communication middleware between the simulation engine and the data processing module includes any one of the following: ROS, CyberRT.

[0006] Optionally, the simulation engine is further configured to: add a plurality of the monocular cameras to the simulator according to camera parameters of the multi-camera.

[0007] Optionally, the camera parameters of the multi-camera include at least one of the following: camera intrinsic parameters, camera extrinsic parameters, camera spacing, and installation position.

[0008] Optionally, the sensor simulation system further includes: an intelligent driving module; wherein the data processing module is further used to: send the multi-eye simulation data and simulation parameters of the multi-eye simulation data to the intelligent driving module.

[0009] Optionally, the second communication middleware between the data processing module and the intelligent driving module includes any one of the following: ROS, DDS, UDP, TCP.

[0010] Optionally, the simulation parameters of the multi-target simulation data include at least one of the following: the timestamp, size data, and data type.

[0011] Optionally, the intelligent driving module is used to perform intelligent driving simulation based on the multi-eye simulation data and the simulation parameters of the multi-eye simulation data; wherein the intelligent driving simulation includes at least one of the following: obstacle detection, distance detection, and environmental perception.

[0012] According to a second aspect of the present application, a sensor simulation method is provided, comprising: adding simulation data of a monocular camera and a timestamp of the simulation data to a message queue of the monocular camera; obtaining a plurality of the simulation data synchronized in time from the plurality of message queues based on the timestamps in the plurality of the message queues; and generating multi-camera simulation data based on the plurality of the simulation data synchronized in time.

[0013] Optionally, obtaining multiple simulation data synchronized with time from multiple message queues based on the timestamps in the multiple message queues includes: obtaining the timestamp at the end of each message queue; obtaining multiple simulation data synchronized with time from multiple message queues based on the first timestamp among the multiple timestamps at the end; wherein the first timestamp is the earliest timestamp among the multiple timestamps at the end.

[0014] Optionally, the method of obtaining multiple simulation data synchronized with time from multiple message queues based on the first timestamp among the multiple timestamps at the end includes: obtaining first simulation data corresponding to the first timestamp from the first message queue where the first timestamp is located; and obtaining second simulation data corresponding to the second timestamp closest to the first timestamp from the remaining message queues other than the first message queue; wherein the simulation data synchronized with time include the first simulation data and the second simulation data.

[0015] Optionally, a difference between the first timestamp and the second timestamp is smaller than a time threshold.

[0016] Optionally, the method further includes: if the length of the message queue is greater than a length threshold, removing the simulation data and the timestamp at the end of the message queue.

[0017] Optionally, generating multi-target simulation data based on the multiple simulation data synchronized in time includes: merging the multiple simulation data synchronized in time to obtain the multi-target simulation data.

[0018] Optionally, merging the multiple simulation data synchronized in time to obtain the multi-target simulation data includes: converting the multiple simulation data synchronized in time into a matrix format; and merging the multiple simulation data in the matrix format to obtain the multi-target simulation data.

[0019] Optionally, merging the multiple simulation data in the matrix format to obtain the multi-eye simulation data includes: horizontally merging the multiple simulation data in the matrix format to obtain the multi-eye simulation data; or vertically merging the multiple simulation data in the matrix format to obtain the multi-eye simulation data.

[0020] Optionally, the method further includes: copying the multi-target simulation data into a memory area.

[0021] Optionally, the method further includes: determining simulation parameters of the multi-target simulation data based on simulation parameters of a plurality of simulation data synchronized in time.

[0022] Optionally, determining the simulation parameters of the multi-target simulation data based on the simulation parameters of the multiple simulation data synchronized in time includes: using the earliest timestamp among the timestamps of the multiple simulation data synchronized in time as the timestamp of the multi-target simulation data.

[0023] Optionally, determining the simulation parameters of the multi-eye simulation data based on the simulation parameters of the multiple simulation data synchronized in time includes: determining the size data of the multi-eye simulation data based on the size data of the multiple simulation data synchronized in time.

[0024] Optionally, determining the size data of the multi-eye simulation data based on the size data of the multiple simulation data synchronized in time includes: if the multi-eye simulation data is obtained based on the horizontal merging of the multiple simulation data synchronized in time, then using the height data of the multiple simulation data synchronized in time as the height data of the multi-eye simulation data, and using the sum of the width data of the multiple simulation data synchronized in time as the width data of the multi-eye simulation data; if the multi-eye simulation data is obtained based on the vertical merging of the multiple simulation data synchronized in time, then using the width data of the multiple simulation data synchronized in time as the width data of the multi-eye simulation data, and using the sum of the height data of the multiple simulation data synchronized in time as the height data of the multi-eye simulation data.

[0025] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the sensor simulation method described above is implemented.

[0026] According to a fourth aspect of the present application, a computer program product is provided, comprising a computer program, wherein the computer program implements the above-mentioned sensor simulation method when executed by a processor.

[0027] According to a fifth aspect of the present application, an electronic device is provided, comprising: a memory storing a computer program; and a processor configured to execute the computer program in the memory to implement the above-mentioned sensor simulation method.

[0028] According to a sixth aspect of the present application, a vehicle is provided, comprising the above-mentioned sensor simulation system or the above-mentioned electronic device.

[0029] In an embodiment of the present application, the simulation engine can generate simulation data and timestamps for multiple monocular cameras. The data processing module can add the simulation data and timestamps of the monocular cameras to the message queue of the monocular cameras. Then, based on the timestamps in the message queues of the multiple monocular cameras, multiple time-synchronized simulation data are obtained from the multiple message queues, and multi-camera simulation data is generated based on the multiple time-synchronized simulation data. The embodiment of the present application generates multi-camera simulation data based on the simulation data of multiple monocular cameras, thereby realizing the simulation of a multi-camera. Furthermore, the embodiment of the present application generates multi-camera simulation data based on the simulation data of multiple monocular cameras that are time-synchronized, thereby ensuring the accuracy of the multi-camera simulation data and improving the simulation effect of the multi-camera.

[0030] Other features and advantages of the present application will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort.

[0032] In order to more completely understand the present application and its beneficial effects, the following description will be given in conjunction with the accompanying drawings, wherein the same drawing numbers represent the same parts in the following description.

[0033] Figure 1 is a schematic diagram of a sensor simulation system provided in an embodiment of the present application;

[0034] Figure 2 This is a flow chart of a sensor simulation method provided by an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of another sensor simulation system provided in an embodiment of the present application;

[0036] Figure 4 is a flow chart of another sensor simulation method provided by an embodiment of the present application;

[0037] Figure 5 is a flow chart of another sensor simulation method provided by an embodiment of the present application;

[0038] Figure 6 is a flow chart of another sensor simulation method provided by an embodiment of the present application;

[0039] Figure 7 is a flow chart of another sensor simulation method provided by an embodiment of the present application;

[0040] Figure 8 It is a schematic diagram of a vehicle provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0042] According to a first aspect of the present application, an embodiment of the present application provides a sensor simulation system.

[0043] See also Figure 1 , Figure 1 Schematic diagram of a sensor simulation system provided by an embodiment of the present application. Figure 1 As shown, the sensor simulation system includes: a simulation engine 100 and a data processing module 200.

[0044] The simulation engine 100 is used to generate simulation data of multiple monocular cameras and timestamps of the simulation data.

[0045] The data processing module 200 is used to: add the simulation data and timestamp of the monocular camera to the message queue of the monocular camera; obtain multiple simulation data synchronized with time from multiple message queues according to the timestamps in multiple message queues; and generate multi-camera simulation data based on the multiple simulation data synchronized with time.

[0046] In the embodiment of the present application, multiple monocular cameras are installed in the simulation engine 100. For example, two monocular cameras can be installed in the simulation engine 100 to serve as the left and right cameras of a multi-camera. In actual applications, the number of monocular cameras installed in the simulation engine 100 can be determined based on the number of cameras in the multi-camera to be simulated. It should be understood that the simulation engine 100 can run a simulation scene, in which a simulated vehicle is installed, and multiple monocular cameras are installed in the simulated vehicle.

[0047] When setting up a monocular camera, it is also necessary to set the camera parameters. Therefore, in some embodiments, the simulation engine 100 is also used to add multiple monocular cameras to the simulator based on the camera parameters of the multi-eye camera. The embodiments of the present application do not limit the camera parameters that need to be set. In some embodiments, the camera parameters of the multi-eye camera include at least one of the following: camera intrinsic parameters, camera extrinsic parameters, camera spacing, and installation position. Among them, the installation position refers to the installation position of the multi-eye camera in the simulated vehicle. Based on the camera parameters of the multi-eye camera, the camera parameters of each monocular camera can be determined, so that the monocular camera can be added to the simulation engine 100 according to the camera parameters of each monocular camera.

[0048] During the simulation process, the simulation engine 100 can generate simulation data and timestamps of the simulation data for each monocular camera. The simulation engine 100 sends the simulation data and timestamps of multiple monocular cameras to the data processing module 200 through the first communication middleware. In some embodiments, the first communication middleware between the simulation engine 100 and the data processing module 200 includes any one of the following: ROS (Robot Operating System), CyberRT (Cyber Real-Time Operating System). The simulation engine 100 can send the simulation data and timestamps of multiple monocular cameras through the publishing component, and the data processing module 200 can obtain the simulation data and timestamps of multiple monocular cameras through the subscription component. Among them, the publishing component and the subscription component can be specifically determined based on the communication middleware supported by the platform.

[0049] After acquiring simulation data and timestamps from multiple monocular cameras, the data processing module 200 adds each monocular camera's simulation data and timestamp to the monocular camera's message queue. The simulation data in the multiple message queues is then time-synchronized to obtain multiple synchronized simulation data. Based on the synchronized simulation data, multi-camera simulation data is generated. The specific operation and method of the data processing module 200 will be described in the following embodiments and will not be elaborated upon here.

[0050] In some embodiments, as Figure 1As shown, the above-mentioned sensor simulation system also includes: an intelligent driving module 300. Among them, the data processing module 200 is also used to: send multi-eye simulation data and simulation parameters of the multi-eye simulation data to the intelligent driving module 300. The embodiment of the present application does not limit the simulation parameters of the multi-eye simulation data. In some embodiments, the simulation parameters of the multi-eye simulation data include at least one of the following: timestamp, size data, and data type. The data processing module 200 can send multi-eye simulation data and simulation parameters of the multi-eye simulation data to the intelligent driving module 300 through the second communication middleware. The second communication middleware can be the same as or different from the above-mentioned first communication middleware. In some embodiments, the second communication middleware between the data processing module 200 and the intelligent driving module 300 includes any one of the following: ROS, DDS (Data Distribution Service), UDP (User Datagram Protocol), TCP (Transmission Control Protocol).

[0051] The intelligent driving module 300 can run intelligent driving algorithms, such as perception fusion and positioning algorithms, to simulate intelligent driving functions. In some embodiments, the intelligent driving module 300 is configured to perform intelligent driving simulation based on multi-view simulation data and simulation parameters of the multi-view simulation data. The intelligent driving simulation includes at least one of the following: obstacle detection, distance detection, and environmental perception.

[0052] It should be understood that the simulation engine 100, data processing module 200, and intelligent driving module 300 can run on the same computer device or on different computer devices, and this is not limited in the present embodiment. For example, the simulation engine 100 and data processing module 200 can run on the same computer device, and the intelligent driving module 300 can run on another computer device; or the simulation engine 100, data processing module 200, and intelligent driving module 300 can each run on a different computer device.

[0053] In summary, the sensor simulation system provided by the embodiment of the present application includes a simulation engine and a data processing module, wherein the simulation engine can generate simulation data and timestamps of multiple monocular cameras, and the data processing module can determine multiple simulation data synchronized in time based on the simulation data and timestamps of multiple monocular cameras, and generate multi-eye simulation data based on the multiple simulation data synchronized in time. The embodiment of the present application realizes the simulation of a multi-eye camera through the simulation of multiple monocular cameras, and the embodiment of the present application generates multi-eye simulation data based on the simulation data of multiple monocular cameras synchronized in time, thereby ensuring the accuracy of the multi-eye simulation data and improving the simulation effect of the multi-eye camera. In addition, the various modules in the sensor simulation system provided by the embodiment of the present application can exchange messages through a variety of communication middleware, thereby improving the versatility of sensor simulation.

[0054] For further description of the steps and beneficial effects performed by each module in the sensor simulation system, please refer to the following method embodiments, which will not be elaborated here.

[0055] According to a second aspect of the present application, an embodiment of the present application provides a sensor simulation method.

[0056] See also Figure 2 , Figure 2 Schematic diagram of a sensor simulation method provided by an embodiment of the present application. The sensor simulation method can be applied to the above-mentioned sensor simulation system, for example, it can be implemented by the data processing module 200 in the above-mentioned sensor simulation system. Figure 2 As shown, the sensor simulation method may include the following steps:

[0057] Step S100: adding the simulation data of the monocular camera and the timestamp of the simulation data to the message queue of the monocular camera;

[0058] Step S200: acquiring multiple time-synchronized simulation data from the multiple message queues according to the timestamps in the multiple message queues;

[0059] Step S300: Generate multi-source simulation data based on multiple time-synchronized simulation data.

[0060] In an embodiment of the present application, a message queue can be set up for each monocular camera, and the simulation data and timestamps of the corresponding monocular camera are arranged in chronological order in the message queue. Thus, in the message queue, the simulation data and timestamp at the top are the earliest simulation data and timestamps, and the simulation data and timestamp at the end are the latest simulation data and timestamps. After obtaining the simulation data and timestamps of multiple monocular cameras, the simulation data and timestamps of each monocular camera are added to the message queue of the monocular camera.

[0061] According to the timestamps in the multiple message queues, multiple simulation data with time synchronization can be obtained from the multiple message queues. Among them, one simulation data is obtained from each message queue, so that multiple simulation data can be obtained from the multiple message queues, and the multiple simulation data meet time synchronization. Time synchronization can refer to the same time, or close time, for example, the difference between the timestamps of the multiple simulation data is within a time threshold. Thus, in step S200, multiple simulation data with the same or close time can be obtained from the multiple message queues, and these simulation data meet time synchronization. For other introductions and instructions on obtaining multiple simulation data with time synchronization, please refer to the following embodiments, which will not be elaborated here.

[0062] Based on multiple time-synchronized simulation data, multi-camera simulation data can be generated to achieve multi-camera simulation. The multi-camera simulation data can be generated by merging multiple time-synchronized simulation data, for example, by horizontally or vertically merging multiple simulation data. For detailed methods for generating multi-camera simulation data, please refer to the following embodiments and will not be elaborated on here.

[0063] In summary, the sensor simulation method provided in the embodiment of the present application adds the simulation data and timestamp of a monocular camera to the message queue of the monocular camera, then obtains multiple time-synchronized simulation data from the multiple message queues based on the timestamps in the message queues of multiple monocular cameras, and then generates multi-camera simulation data based on the multiple time-synchronized simulation data. The embodiment of the present application generates multi-camera simulation data based on the simulation data of multiple monocular cameras, thereby realizing the simulation of multiple cameras. Furthermore, the embodiment of the present application generates multi-camera simulation data based on the simulation data of multiple monocular cameras that are time-synchronized, thereby ensuring the accuracy of the multi-camera simulation data and improving the simulation effect of the multi-camera.

[0064] In some embodiments, the above step S200 may include the following steps:

[0065] Step S210: Obtain the timestamp at the end of each message queue;

[0066] Step S220: Acquire multiple simulation data synchronized in time from multiple message queues according to a first timestamp among the multiple tail timestamps.

[0067] The end timestamp refers to the timestamp of the latest or most recent message queue entry. Embodiments of the present application obtain a end timestamp from each message queue, thereby enabling the acquisition of multiple end timestamps from multiple message queues. Based on these multiple end timestamps, multiple simulation data sets with the latest time synchronization can be acquired from multiple message queues.

[0068] In an embodiment of the present application, multiple time-synchronized simulation data are obtained based on a first timestamp among the multiple trailing timestamps. The first timestamp is the earliest timestamp among the multiple trailing timestamps. The multiple time-synchronized simulation data include simulation data corresponding to timestamps that are the same as or close to the first timestamp.

[0069] In some embodiments, the above step S220 may include the following steps:

[0070] Step S221: Obtain first simulation data corresponding to the first timestamp from the first message queue where the first timestamp is located;

[0071] Step S222: obtaining second simulation data corresponding to a second timestamp closest to the first timestamp from the remaining message queues other than the first message queue.

[0072] In the embodiment of the present application, multiple may refer to two or more. Thus, the first message queue may be a message queue, and the remaining message queues other than the first message queue in the multiple message queues may be one or more message queues. In the embodiment of the present application, the second simulation data corresponding to the second timestamp closest to the first timestamp are respectively obtained from the remaining message queues other than the first message queue. Wherein, if the remaining message queues other than the first message queue are multiple message queues, a second timestamp and the second simulation data corresponding to the second timestamp are obtained from each of the remaining message queues. The time-synchronized simulation data includes the first simulation data and the second simulation data. If the multiple message queues are two message queues, the time-synchronized simulation data includes one first simulation data and one second simulation data; if the multiple message queues are more than two message queues, the time-synchronized simulation data includes one first simulation data and multiple second simulation data.

[0073] In some embodiments, obtaining the second timestamp from the remaining message queues can be implemented based on a binary search method. The current message queue can be divided into two sub-message queues based on the intermediate timestamp of the current message queue. The search range is updated by comparing the first timestamp with the intermediate timestamp of the current message queue. The intermediate timestamps are then compared in the new sub-message queues until a second timestamp closest to the first timestamp is obtained.

[0074] To improve the accuracy of time synchronization, embodiments of the present application may set data filtering conditions for time synchronization. In some embodiments, the difference between the first timestamp and the second timestamp is less than a time threshold. This time threshold is the maximum acceptable time difference between the data and can be flexibly set based on actual conditions.

[0075] In order to reduce the amount of calculation and memory overhead, in some embodiments, the above method further includes: if the length of the message queue is greater than a length threshold, removing the simulation data and timestamp at the end of the message queue.

[0076] In summary, the sensor simulation method provided by the embodiment of the present application obtains multiple simulation data synchronized with time based on the earliest timestamp among the timestamps at the end of multiple message queues, and can obtain the latest multiple simulation data synchronized with time to ensure the real-time nature of the generated multi-eye simulation data. In addition, the embodiment of the present application sets a time threshold as a data filtering condition for time synchronization to ensure that the generated multi-eye simulation data meets the maximum time difference of the data, thereby improving the accuracy of the multi-eye simulation data. In addition, the embodiment of the present application sets a length threshold for the message queue to reduce the amount of calculation and reduce memory overhead.

[0077] In some embodiments, the above step S300 may include: merging multiple time-synchronized simulation data to obtain multi-source simulation data.

[0078] The processing of simulation data in the embodiments of the present application can be implemented based on OpenCV (open source computer vision library). OpenCV is a cross-platform computer vision library distributed under the BSD (Berkeley Software Distribution) license (open source) and can run on Linux, Windows, Android, and Mac OS operating systems. It is lightweight and efficient, consisting of a series of C functions and a small number of C++ class functions. It also provides interfaces for languages such as Python, Ruby, and MATLAB, and implements many common algorithms for image processing and computer vision.

[0079] To facilitate data processing, the multiple time-synchronized simulation data may be format-converted first, and then multi-channel simulation data may be generated based on the multiple simulation data after format conversion. Thus, in some embodiments, the above-mentioned merging of the multiple time-synchronized simulation data to obtain the multi-channel simulation data may include the following steps:

[0080] Step S310: converting multiple time-synchronized simulation data into a matrix format;

[0081] Step S320: merging multiple simulation data in matrix format to obtain multi-object simulation data.

[0082] The present embodiment can obtain the data formats of multiple simulation data, and different OpenCV methods are used to convert different data formats. Based on the data formats of the multiple simulation data, an appropriate OpenCV method is selected to convert the multiple time-synchronized simulation data into a matrix format. The matrix format can be a Mat matrix type, which is not limited by the present embodiment.

[0083] The multiple simulation data in matrix format are then merged to obtain multi-eye simulation data. In some embodiments, step S320 may include: horizontally merging the multiple simulation data in matrix format to obtain multi-eye simulation data; or vertically merging the multiple simulation data in matrix format to obtain multi-eye simulation data. Horizontal merging may call the hconcat function, which is used for matrix merging and image concatenation; vertical merging may call the vconcat function.

[0084] In some embodiments, the above method further includes: copying the multi-target simulation data into a memory area. After generating the multi-target simulation data, the embodiment of the present application can copy the multi-target simulation data into a memory area by using a memcpy copy method, etc., and subsequently read the multi-target simulation data from the memory area. Since the multi-target simulation data is large and a large amount of data may need to be read and written, if a single variable is used for writing, the performance will be slowed down. Therefore, the embodiment of the present application copies the multi-target simulation data into a memory area, and reading and writing from the memory area is completed by means of pointers, which can achieve fast access to a large amount of data, a faster conversion speed, and improved data reading efficiency.

[0085] In summary, the sensor simulation method provided by the embodiments of the present application horizontally or vertically merges multiple time-synchronized simulation data into multi-view simulation data. This data merging method is simple and highly versatile. Furthermore, the embodiments of the present application copy the multi-view simulation data into a memory area, facilitating subsequent large-scale access to the multi-view simulation data and improving data processing and reading efficiency.

[0086] In some embodiments, the above method further comprises the following steps:

[0087] Step S400: Determine simulation parameters of multi-target simulation data according to simulation parameters of multiple time-synchronized simulation data.

[0088] To improve the accuracy of subsequent simulation tests, embodiments of the present application also determine simulation parameters for each multi-target simulation data set. In some embodiments, the simulation parameters for the multi-target simulation data set include, but are not limited to, timestamps, dimension data, and data types. The data type of the multi-target simulation data set can be the aforementioned matrix type, such as a Mat matrix type.

[0089] Taking the case where the simulation parameters of the multi-target simulation data include a timestamp as an example, in some embodiments, the above step S400 may include the following steps:

[0090] Step S410: The earliest timestamp among the timestamps of the multiple time-synchronized simulation data is used as the timestamp of the multi-target simulation data.

[0091] Of course, the embodiments of the present application do not exclude other methods for determining the timestamp of the multi-channel simulation data. For example, the latest timestamp among the timestamps of multiple time-synchronized simulation data is used as the timestamp of the multi-channel simulation data; or the timestamps of multiple time-synchronized simulation data are averaged or weighted averaged to obtain the timestamp of the multi-channel simulation data.

[0092] Taking the case where the simulation parameters of the multi-objective simulation data include size data as an example, in some embodiments, the above step S400 may include the following steps:

[0093] Step S420: Determine the size data of the multi-objective simulation data according to the size data of the multiple simulation data synchronized in time.

[0094] The size data includes but is not limited to: length data and width data. Depending on the different ways of merging multiple simulation data, the size data of the multi-eye simulation data has different ways of determining. In some embodiments, the above step S420 may include: if the multi-eye simulation data is obtained based on the horizontal merging of multiple simulation data synchronized in time, then the height data of the multiple simulation data synchronized in time is used as the height data of the multi-eye simulation data, and the sum of the width data of the multiple simulation data synchronized in time is used as the width data of the multi-eye simulation data; if the multi-eye simulation data is obtained based on the vertical merging of multiple simulation data synchronized in time, then the width data of the multiple simulation data synchronized in time is used as the width data of the multi-eye simulation data, and the sum of the height data of the multiple simulation data synchronized in time is used as the height data of the multi-eye simulation data.

[0095] To sum up, the sensor simulation method provided in the embodiment of the present application, after generating multi-eye simulation data, also determines the simulation parameters of the multi-eye simulation data, so as to facilitate the subsequent execution of simulation tests such as obstacle detection and distance detection based on the multi-eye simulation data and its simulation parameters, thereby improving the accuracy of the simulation test.

[0096] The sensor simulation system and sensor simulation method provided in the embodiments of the present application are introduced and explained below with several examples.

[0097] See also Figure 3 , Figure 3 Schematic diagram of another sensor simulation system provided in an embodiment of the present application. Figure 3Taking binocular camera simulation using two monocular cameras as an example, two monocular cameras are set up in the simulation engine 100 as the left and right binocular cameras. The simulation data and timestamps of the two monocular cameras are then sent to the data processing module 200 via the first communication middleware. The data processing module 200 generates binocular simulation data based on the simulation data and timestamps of the multiple monocular cameras. The binocular simulation data and its simulation parameters are then sent to the intelligent driving module 300 via the second communication middleware. Based on the binocular simulation data and its simulation parameters, the intelligent driving module 300 can perform intelligent driving simulations such as obstacle detection, distance detection, and environmental perception.

[0098] See also Figure 4 , Figure 4 This is a flow chart of another sensor simulation method provided by an embodiment of the present application. This sensor simulation method can be applied to the above Figure 1 or Figure 3 The sensor simulation system shown in Figure 2 is as follows. Figure 4 As shown, the sensor simulation method may include the following steps:

[0099] Step S010: The simulation engine adds two monocular cameras to the simulator according to the camera parameters of the binocular camera;

[0100] Step S020: The simulation engine generates simulation data of the two monocular cameras and timestamps of the simulation data respectively;

[0101] Step S030: The simulation engine sends the simulation data and timestamps of the two monocular cameras to the data processing module through the first communication middleware;

[0102] Step S040: The data processing module adds the simulation data and timestamps of the two monocular cameras to the message queues of the two monocular cameras respectively;

[0103] Step S050: The data processing module obtains two time-synchronized simulation data from the two message queues according to the timestamps in the two message queues;

[0104] Step S060: The data processing module merges the two time-synchronized simulation data to obtain binocular simulation data;

[0105] Step S070: The data processing module sends the binocular simulation data and simulation parameters of the binocular simulation data to the intelligent driving module through the second communication middleware.

[0106] See also Figure 5 , Figure 5 This is a flow chart of another sensor simulation method provided by an embodiment of the present application. This sensor simulation method can be applied to the above Figure 1 or Figure 3The sensor simulation system shown is applied to the above data processing module. Figure 5 As shown, the sensor simulation method may include the following steps:

[0107] Step S501: Setting a time threshold and a length threshold;

[0108] Step S502: Acquire simulation data and timestamps of two monocular cameras; that is, acquire simulation data and timestamps of the left camera, and simulation data and timestamps of the right camera;

[0109] Step S503: Add the simulation data and timestamps of the two monocular cameras to the message queues of the two monocular cameras respectively; that is, add the simulation data and timestamp of the left camera to the message queue of the left camera, and add the simulation data and timestamp of the right camera to the message queue of the right camera;

[0110] Step S504: Obtain the end timestamps from the two message queues respectively; that is, obtain the end timestamp of the left camera from the message queue of the left camera, and obtain the end timestamp of the right camera from the message queue of the right camera;

[0111] Step S505: Determine whether the timestamp at the end of the left camera is less than the timestamp at the end of the right camera; if so, start from the following step S506; otherwise, start from the following step S508;

[0112] Step S506: taking the timestamp at the end of the left-eye camera as the first timestamp, and taking the simulation data corresponding to the first timestamp in the message queue of the left-eye camera as the first simulation data;

[0113] Step S507: performing a binary search based on the timestamp in the message queue of the right camera to obtain a second timestamp that is closest to the first timestamp and whose difference is less than the time threshold, and second simulation data corresponding to the second timestamp;

[0114] Step S508: taking the timestamp at the end of the right camera as the first timestamp, and taking the simulation data corresponding to the first timestamp in the message queue of the right camera as the first simulation data;

[0115] Step S509: performing a binary search based on the timestamp in the message queue of the left camera to obtain a second timestamp that is closest to the first timestamp and whose difference is less than the time threshold, and second simulation data corresponding to the second timestamp;

[0116] Step S510: Determine whether the length of the message queue of the left camera is greater than a length threshold; if so, execute the following step S511; otherwise, execute the following step S512;

[0117] Step S511: Delete the simulation data and timestamp at the end of the message queue of the left camera;

[0118] Step S512: Determine whether the length of the message queue of the right camera is greater than a length threshold; if so, execute the following step S513; otherwise, execute the following step S514;

[0119] Step S513: Delete the simulation data and timestamp at the end of the message queue of the right camera;

[0120] Step S514: Output the time-synchronized first simulation data and second simulation data.

[0121] See also Figure 6 , Figure 6 This is a flow chart of another sensor simulation method provided by an embodiment of the present application. This sensor simulation method can be applied to the above Figure 1 or Figure 3 The sensor simulation system shown is applied to the above data processing module. Figure 6 As shown, the sensor simulation method may include the following steps:

[0122] Step S601: Acquire data types of first simulation data and second simulation data that are time-synchronized;

[0123] Step S602: converting the first simulation data and the second simulation data into a matrix format according to the data types of the first simulation data and the second simulation data;

[0124] Step S603: horizontally merging the first simulation data and the second simulation data in the matrix format into binocular simulation data;

[0125] Step S604: copy the binocular simulation data to the memory area.

[0126] See also Figure 7 , Figure 7 This is a flow chart of another sensor simulation method provided by an embodiment of the present application. This sensor simulation method can be applied to the above Figure 1 or Figure 3 The sensor simulation system shown is applied to the above data processing module. Figure 7 As shown, the sensor simulation method may include the following steps:

[0127] Step S701: The earliest timestamp among the timestamps of the time-synchronized first simulation data and the second simulation data is used as the timestamp of the binocular and multi-camera simulation data;

[0128] Step S702: using the height data of the first simulation data and the second simulation data as the height data of the binocular simulation data, and using the sum of the width data of the first simulation data and the second simulation data as the width data of the binocular simulation data;

[0129] Step S703: setting the data type of the binocular simulation data to a matrix format;

[0130] Step S704: sending the binocular simulation data, as well as the timestamp, height data, width data and data type of the binocular simulation data through the second communication middleware.

[0131] The sensor simulation system and sensor simulation method provided in the embodiments of the present application can solve the drawbacks of binocular camera simulation, such as data time asynchrony, non-universal data splicing methods with low conversion efficiency, and few compatible communication middleware platforms. They have the beneficial effects of good versatility, optimized time synchronization, and high computational efficiency.

[0132] According to a third aspect of the present application, embodiments of the present application further provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned sensor simulation method. This non-transitory computer-readable storage medium has all the beneficial effects of the aforementioned sensor simulation method, which are not further detailed herein.

[0133] According to the fourth aspect of the present application, an embodiment of the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the above-mentioned sensor simulation method and has all the beneficial effects of the above-mentioned sensor simulation method. This application will not go into details here.

[0134] According to a fifth aspect of the present application, an embodiment of the present application further provides an electronic device comprising: a memory and a processor, wherein the memory stores a computer program; the processor is configured to execute the computer program in the memory to implement the steps of the above-described sensor simulation method. This electronic device has all the beneficial effects of the above-described sensor simulation method, and this application will not further elaborate on them.

[0135] The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof, and this application does not specifically limit this. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0136] In some embodiments of the present application, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0137] The computer-readable storage medium may be included in the electronic device or may exist independently without being incorporated into the electronic device. The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0138] Adding the simulation data of the monocular camera and the timestamp of the simulation data to the message queue of the monocular camera;

[0139] Acquire a plurality of the simulation data synchronized in time from the plurality of the message queues according to the timestamps in the plurality of the message queues;

[0140] Generate multi-target simulation data based on the multiple simulation data synchronized in time.

[0141] Computer program code for performing the operations of some embodiments of the present application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network (including a local area network (LAN) or a wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider to connect via the Internet).

[0142] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function.

[0143] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures.

[0144] For example, two blocks shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or may be implemented using a combination of dedicated hardware and computer instructions.

[0145] The units described in some embodiments of the present application may be implemented in software or hardware, and may also be provided in a processor.

[0146] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and the like.

[0147] According to the sixth aspect of this application, Figure 8 As shown, the embodiment of the present application further provides a vehicle 10, which includes the above-mentioned sensor simulation system or the above-mentioned electronic device. The vehicle has all the beneficial effects of the above-mentioned electronic devices, etc., which will not be described in detail in this application.

[0148] The vehicle may be a fuel vehicle, a plug-in hybrid vehicle or a new energy vehicle, etc., and this application does not make any specific restrictions on this.

[0149] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0150] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0151] The embodiments, implementation methods and related technical features of the present application can be combined and replaced with each other without conflict.

[0152] The above are merely preferred embodiments of the present application and do not constitute any form of limitation to the present application. Although the descriptions of each embodiment in the embodiments of the present application have different focuses, for parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A sensor simulation system, characterized in that: The sensor simulation system comprises: a simulation engine (100) and a data processing module (200); wherein, The simulation engine (100) is used to: generate simulation data of a plurality of monocular cameras and timestamps of the simulation data; The data processing module (200) is used to: add the simulation data and the timestamp of the monocular camera to the message queue of the monocular camera; obtain multiple simulation data synchronized in time from the multiple message queues according to the timestamps in the multiple message queues; and generate multi-camera simulation data according to the multiple simulation data synchronized in time.

2. The sensor simulation system according to claim 1, characterized in that: The first communication middleware between the simulation engine (100) and the data processing module (200) includes any one of the following: ROS, CyberRT.

3. The sensor simulation system according to claim 1, wherein: The simulation engine (100) is further used for: According to the camera parameters of the multi-camera, multiple monocular cameras are added to the simulator.

4. The sensor simulation system according to claim 3, characterized in that: The camera parameters of the multi-camera include at least one of the following: camera intrinsic parameters, camera extrinsic parameters, camera spacing, and installation position.

5. The sensor simulation system according to claim 1, characterized in that: The sensor simulation system further comprises: an intelligent driving module (300); wherein, The data processing module (200) is further configured to send the multi-objective simulation data and simulation parameters of the multi-objective simulation data to the intelligent driving module (300).

6. The sensor simulation system according to claim 5, characterized in that: The second communication middleware between the data processing module (200) and the intelligent driving module (300) includes any one of the following: ROS, DDS, UDP, TCP.

7. The sensor simulation system according to claim 5, characterized in that: The simulation parameters of the multi-target simulation data include at least one of the following: the timestamp, size data, and data type.

8. The sensor simulation system according to claim 5, characterized in that: The intelligent driving module (300) is used for: Intelligent driving simulation is performed based on the multi-eye simulation data and the simulation parameters of the multi-eye simulation data; wherein the intelligent driving simulation includes at least one of the following: obstacle detection, distance detection, and environment perception.

9. A sensor simulation method, characterized in that: The method comprises: Adding the simulation data of the monocular camera and the timestamp of the simulation data to the message queue of the monocular camera; Acquire a plurality of the simulation data synchronized in time from the plurality of the message queues according to the timestamps in the plurality of the message queues; Generate multi-target simulation data based on the multiple simulation data synchronized in time.

10. The method according to claim 9, characterized in that The acquiring the multiple simulation data synchronized in time from the multiple message queues according to the timestamps in the multiple message queues comprises: Obtain the timestamp at the end of each message queue; According to a first timestamp among the multiple tail timestamps, a plurality of the simulation data with time synchronization is acquired from the multiple message queues; wherein the first timestamp is the earliest timestamp among the multiple tail timestamps.

11. The method according to claim 10, characterized in that The acquiring, according to a first timestamp among the multiple tail timestamps, the multiple simulation data with time synchronization from the multiple message queues comprises: Obtaining first simulation data corresponding to the first timestamp from the first message queue where the first timestamp is located; Obtaining, from the remaining message queues other than the first message queue, second simulation data corresponding to a second timestamp closest to the first timestamp; The time-synchronized simulation data includes the first simulation data and the second simulation data.

12. The method according to claim 11, characterized in that A difference between the first timestamp and the second timestamp is smaller than a time threshold.

13. The method according to claim 9, characterized in that The method further comprises: If the length of the message queue is greater than a length threshold, the simulation data and the timestamp at the end of the message queue are removed.

14. The method according to claim 9, characterized in that Generating multi-target simulation data according to the plurality of simulation data synchronized at the time includes: The multiple simulation data synchronized in time are merged to obtain the multi-target simulation data.

15. The method according to claim 14, characterized in that The merging of the multiple time-synchronized simulation data to obtain the multi-target simulation data includes: Converting the time-synchronized plurality of simulation data into a matrix format; Merge the plurality of simulation data in the matrix format to obtain the multi-object simulation data.

16. The method according to claim 15, characterized in that The merging of the plurality of simulation data in the matrix format to obtain the multi-objective simulation data comprises: horizontally merging the plurality of simulation data in the matrix format to obtain the multi-object simulation data; or, The plurality of simulation data in the matrix format are vertically merged to obtain the multi-object simulation data.

17. The method according to claim 14, characterized in that The method further comprises: The multi-target simulation data is copied into the memory area.

18. The method according to claim 14, characterized in that The method further comprises: The simulation parameters of the multi-target simulation data are determined according to the simulation parameters of the multiple simulation data that are synchronized in time.

19. The method according to claim 18, characterized in that Determining the simulation parameters of the multi-target simulation data according to the simulation parameters of the plurality of simulation data synchronized in time includes: The earliest timestamp among the timestamps of the multiple simulation data synchronized in time is used as the timestamp of the multi-target simulation data.

20. The method according to claim 18, wherein Determining the simulation parameters of the multi-target simulation data according to the simulation parameters of the plurality of simulation data synchronized in time includes: The size data of the multi-object simulation data is determined according to the size data of the plurality of simulation data synchronized in time.

21. The method according to claim 20, characterized in that The determining the size data of the multi-objective simulation data according to the size data of the plurality of simulation data synchronized in time includes: If the multi-eye simulation data is obtained by horizontally merging the multiple time-synchronized simulation data, the height data of the multiple time-synchronized simulation data are used as the height data of the multi-eye simulation data, and the sum of the width data of the multiple time-synchronized simulation data are used as the width data of the multi-eye simulation data; If the multi-eye simulation data is obtained by vertically merging the multiple simulation data synchronized in time, the width data of the multiple simulation data synchronized in time is used as the width data of the multi-eye simulation data, and the sum of the height data of the multiple simulation data synchronized in time is used as the height data of the multi-eye simulation data.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sensor simulation method according to any one of claims 9 to 21 is implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the sensor simulation method according to any one of claims 9 to 21 is implemented.

24. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the sensor simulation method according to any one of claims 9 to 21.

25. A vehicle, characterized in that: The sensor simulation system comprises the sensor simulation system according to any one of claims 1 to 8, or the electronic device according to claim 24.

Images