Automatic driving simulation scene rendering system, method and computer program product

Abstract

The invention discloses a rendering system and method for an automatic driving simulation scene and a computer program product. The system comprises communication middleware, a vehicle cloud routing module and a simulator, the communication middleware and the vehicle cloud routing module are deployed at a vehicle end, and the simulator is deployed at a cloud end; the communication middleware is used for acquiring road network data acquired by a sensor of the vehicle, processing the road network data acquired by the sensor according to a preset data processing rule, and sending the processed road network data to the vehicle cloud routing module; the vehicle cloud routing module is used for sending the processed road network data to the simulator; and the simulator is used for extracting the road network data of the view angle of the driver from the processed road network data, and performing scene rendering according to the road network data of the view angle of the driver.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and more specifically, to a rendering system, method, and computer program product for an autonomous driving simulation scene. Background Art

[0002] Usually, the autonomous driving system is first developed and verified in the simulator, and then deployed in the vehicle after the accuracy and safety of the autonomous driving system are confirmed. Existing autonomous driving simulators can import real road network data, generate simulation scenarios based on the imported road network data, and perform simulation tests. The current simulation scenarios are usually from a third-person perspective. R&D engineers see the scenes from a third-person perspective, which cannot better bring in the driver's perspective and cannot develop autonomous driving functions specifically for the pain points of the driver during actual driving. Summary of the invention

[0003] An object of the present invention is to provide a new technical solution for rendering autonomous driving simulation scenes.

[0004] According to a first aspect of the present invention, there is provided a rendering system for an autonomous driving simulation scene, comprising a communication middleware, a vehicle-cloud routing module and a simulation simulator, wherein the communication middleware and the vehicle-cloud routing module are deployed on a vehicle side, and the simulation simulator is deployed on a cloud side;

[0005] The communication middleware is used to obtain the road network data collected by the sensor of the vehicle, process the road network data collected by the sensor according to the preset data processing rules, and send the processed road network data to the vehicle-cloud routing module;

[0006] The vehicle-to-cloud routing module is used to send the processed road network data to the simulation simulator;

[0007] The simulation simulator is used to extract the road network data from the driver's perspective from the processed road network data, and perform scene rendering according to the road network data from the driver's perspective.

[0008] Optionally, the sensor includes a camera sensor and a laser radar, and the road network data collected by the sensor includes image data collected by the camera sensor and point cloud data collected by the laser radar;

[0009] The communication middleware is used to extract environmental data from the image data collected by the camera sensor according to a preset video SLAM algorithm based on optical flow and feature matching;

[0010] The communication middleware is used to process the point cloud data according to a preset point cloud segmentation and tracking algorithm, and extract key point cloud data from the point cloud data, wherein the key point cloud data includes at least lane point cloud data and obstacle point cloud data.

[0011] Optionally, the communication middleware is used to align the environmental data and the key point cloud data to the same coordinate system according to a preset local feature alignment algorithm, and integrate the environmental data and the key point cloud data according to time information of the environmental data and time information of the key point cloud data.

[0012] Optionally, the simulation simulator is used to obtain a viewing angle range of a driver, filter the environmental data and the key point cloud data according to the viewing angle range of the driver, and retain the environmental data and the key point cloud data within the viewing angle range of the driver;

[0013] The simulation simulator is used for performing scene rendering according to the environmental data and key point cloud data within the driver's viewing angle.

[0014] Optionally, the sensor further comprises a sensor for acquiring head posture data of the driver;

[0015] The communication middleware is used to send the driver's head posture data to the vehicle-to-cloud routing module;

[0016] The vehicle-to-cloud routing module is used to send the driver's head posture data to the simulation simulator;

[0017] The simulation simulator is used to determine the driver's viewing angle range according to the driver's head posture.

[0018] Optionally, after the communication middleware processes the road network data collected by the sensor according to preset data processing rules, the communication middleware is also used to encode and compress the processed road network data, and send the encoded and compressed road network data to the vehicle-cloud routing module.

[0019] Optionally, the communication middleware is used to send the encoded and compressed road network data to the vehicle-cloud routing module through a real-time communication protocol that supports point-to-point communication and low-latency data transmission.

[0020] Optionally, the communication middleware is used to convert the processed road network data into string data, and send the string data to the vehicle-cloud routing module; the vehicle-cloud routing module is used to send the string data to the simulation simulator; the simulation simulator is used to extract the road network data from the driver's perspective from the string data, and perform scene rendering based on the road network data from the driver's perspective.

[0021] According to a second aspect of the present invention, a method for rendering an autonomous driving simulation scene is provided, the method being applied to the rendering system for the autonomous driving simulation scene according to the first aspect of the present invention, the method comprising:

[0022] The communication middleware obtains the road network data collected by the sensor, processes the road network data collected by the sensor according to a preset data processing rule, and sends the processed road network data to the vehicle-to-cloud routing module;

[0023] The vehicle-to-cloud routing module sends the processed road network data to the simulation simulator;

[0024] The simulation simulator extracts the road network data from the driver's perspective from the processed road network data, and performs scene rendering according to the road network data from the driver's perspective.

[0025] According to a third aspect of the present invention, there is provided a computer program product, comprising a computer program / instruction, which implements the method according to the second aspect of the present invention when the computer program / instruction processor is executed.

[0026] According to one embodiment of the present invention, the present invention collects road network data through sensors installed on the vehicle, filters the road network data, retains only the road network data from the driver's perspective, and finally renders the scene based on the road network data from the driver's perspective to display the simulated scene from the driver's perspective, which can better simulate the real driving experience, allowing research and development engineers to more realistically feel the movement of the vehicle and changes in the surrounding environment, and more clearly understand the pain points of the driver in actual driving, so that research and development engineers can develop autonomous driving functions based on the pain points of the driver in actual driving.

[0027] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0029] Figure 1 It is a schematic diagram of a rendering system for an autonomous driving simulation scene in an embodiment of the present application.

[0030] Figure 2 It is a flowchart of a method for rendering an autonomous driving simulation scene in an embodiment of the present application. DETAILED DESCRIPTION

[0031] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0032] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0033] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0034] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0035] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0036] In the existing autonomous driving simulation test technology, a simulation environment is usually created through an existing data set, and the vehicle's autonomous driving function is tested in the simulation environment. In the existing technology, the autonomous driving level is divided according to the degree of automation of the autonomous driving. Level L0 autonomous driving: no automation, the throttle, brake, and steering wheel are all controlled by the driver. Level L1 autonomous driving: driving control is the main, and the system assists in time. Level L2 autonomous driving: partial automation, the driver still needs to concentrate on the road conditions. Level L3 autonomous driving: conditional automatic control, the system can automatically control the vehicle's driving under most road conditions, and the driver's attention does not need to be focused on the road conditions. Level L4 autonomous driving: highly automated, and also has interfaces such as steering wheels to provide timely driving control. Level L5 autonomous driving: fully automated, humans become passengers completely.

[0037] On the one hand, the autonomous driving system can control the vehicle to drive automatically when the user does not manually control the vehicle. On the other hand, the autonomous driving system can also assist the user when the user manually controls the vehicle, such as displaying the driving route and reminding the user to go straight or turn through voice.

[0038] This embodiment introduces a rendering system for an autonomous driving simulation scene, including a communication middleware, a vehicle-cloud routing module, and a simulation simulator. The communication middleware and the vehicle-cloud routing module are deployed on the vehicle side, and the simulation simulator is deployed on the cloud side.

[0039] The communication middleware is used to obtain the road network data collected by the vehicle's sensor, process the road network data collected by the sensor according to preset data processing rules, and send the processed road network data to the vehicle-cloud routing module.

[0040] The vehicle-to-cloud routing module is used to send the processed road network data to the simulation simulator.

[0041] The simulation simulator is used to extract the road network data from the driver's perspective from the processed road network data, and perform scene rendering according to the road network data from the driver's perspective.

[0042] Sensors are installed on the vehicle, and the road network data around the vehicle can be collected through the sensors. There is a driver on the vehicle, and the driver drives the vehicle on a real road. During the driving process of the vehicle, the sensors on the vehicle collect real road network data. Road network data can include lane data, data of other surrounding vehicles, data of surrounding obstacles, etc. After obtaining the road network data, the road network data is processed according to the preset data processing rules. Data processing is an important step in the data preprocessing process, which helps to ensure that the data is accurate, complete and consistent. For example, the preset data processing rules include processing missing values, processing outliers, processing duplicate data, and processing inconsistent data.

[0043] Missing values ​​can occur due to reasons such as data entry errors or incomplete data. Techniques for handling missing values ​​include imputation (replacing missing values ​​with estimated values), deletion (removing rows or columns with missing values), or using algorithms that can handle missing values ​​directly.

[0044] An outlier is a data point that is significantly different from the other data points in the data set. Outliers can be caused by measurement errors or other anomalies in the data. Techniques for dealing with outliers include removing them from the data set, transforming the data to reduce the impact of outliers, or using algorithms that are robust to outliers.

[0045] Duplicate data can occur due to data entry errors or other reasons. Techniques for dealing with duplicate data include removing duplicates from the data set, merging them into a single record, or using algorithms that can handle duplicate data directly.

[0046] Inconsistent data can arise due to data entry errors or other reasons. Techniques for dealing with inconsistent data include standardizing the data (e.g. converting all units to a common scale), manually correcting errors, or using algorithms that can handle inconsistent data directly.

[0047] After processing the road network data collected by the sensor, the processed road network data is sent to the simulation simulator through the vehicle-cloud routing module. The simulation simulator extracts the road network data from the driver's perspective from the processed road network data.

[0048] Specifically, the sensor includes a camera sensor and a laser radar, and the road network data collected by the sensor includes image data collected by the camera sensor and point cloud data collected by the laser radar.

[0049] The camera sensor can be set in front of the top of the vehicle. The wide angle of the camera sensor exceeds the viewing angle of the human eye, so that the camera sensor can collect all the content that the human eye sees. For example, the camera sensor uses a 4K ultra-clear camera with a frame rate of 60FPS and a shooting angle of 140 degrees. The laser radar can be embedded in the front bumper of the car.

[0050] Usually, for autonomous driving vehicles, multiple camera sensors and multiple laser radars are set up to obtain road network data around the vehicle. The data collected by the camera sensor includes lanes, obstacles, other vehicles and other data. Similarly, the data collected by the laser radar also includes lanes, obstacles, other vehicles and other data. The data collected by the camera sensor is image data, while the data collected by the laser radar is point cloud data. Since the data collected by a single sensor may not be accurate, by using data collected by different sensors, the accuracy of the data can be improved and the authenticity of the simulation scene can be improved.

[0051] Usually, the driver can only see part of the area around the vehicle. There are blind spots in the driver's field of vision. The blind spots mainly include the front blind spot, the rearview mirror blind spot, and the blind spot formed by the AB pillars of the vehicle. The driver cannot see the content in these blind spots. The processed road network data will contain data in these blind spots, which needs to be filtered to delete the data in the blind spots and only keep the data from the driver's perspective. The simulation simulator renders the scene according to the data from the driver's perspective, and the final displayed scene is the scene actually seen by the driver.

[0052] This embodiment collects road network data through sensors installed on the vehicle, filters the road network data, and only retains the road network data from the driver's perspective. Finally, the scene is rendered based on the road network data from the driver's perspective to display the simulated scene from the driver's perspective. This can better simulate the real driving experience, allowing research and development engineers to more realistically feel the movement of the vehicle and changes in the surrounding environment, and more clearly understand the pain points of the driver in actual driving, so that research and development engineers can develop autonomous driving functions based on the pain points of the driver in actual driving.

[0053] In this embodiment, the communication middleware is used to extract environmental data from the image data collected by the camera sensor according to a preset video SLAM algorithm based on optical flow and feature matching. The communication middleware is used to process the point cloud data according to a preset point cloud segmentation and tracking algorithm, and extract key point cloud data from the point cloud data, and the key point cloud data at least includes lane point cloud data and obstacle point cloud data.

[0054] The image data collected by the camera sensor and the point cloud data collected by the LiDAR will contain a lot of useless data. Useless data refers to data that will not be used in the scene rendering process. For example, the image data collected by the camera sensor will include distant environmental data, but when rendering the scene, only the data closer to the current vehicle will be used, and the distant environmental data will not be used.

[0055] After acquiring the data collected by the sensor, the communication middleware processes the data collected by the sensor, extracts environmental data from the image data collected by the camera sensor, and extracts key point cloud data from the point cloud data collected by the lidar. Both environmental data and key point cloud data are data needed in the scene rendering process. For example, environmental data includes the position, speed, size and other data of other vehicles near the current vehicle. Key point cloud data includes lane point cloud data and obstacle point cloud data. Obstacles can be obstacles around the current vehicle, such as fences between left and right lanes, flower beds on the side of the road, etc. The communication middleware processes the image data collected by the camera sensor and the point cloud data collected by the lidar, filters out useless data, reduces the amount of data that needs to be transmitted, and improves efficiency.

[0056] In this embodiment, the communication middleware is used to align the environmental data and the key point cloud data to the same coordinate system according to a preset local feature alignment algorithm, and integrate the environmental data and the key point cloud data according to the time information of the environmental data and the time information of the key point cloud data.

[0057] Environmental data is collected by camera sensors, while key point cloud data is collected by lidar. The coordinates of environmental data and key point cloud data are no longer in the same coordinate system. For example, when the camera sensor collects environmental data, the coordinate system is constructed with the position of the camera sensor as the coordinate origin, while when the lidar collects point cloud data, the coordinate system is constructed with the position of the lidar as the coordinate origin. It cannot be used directly when rendering a scene. In order to ensure the accuracy of scene rendering, the coordinates of the environmental data and the coordinates of the key point cloud data need to be aligned to the same coordinate system, for example, by aligning the coordinates by translation, rotation, etc. After alignment to the same coordinate system, the relative position between the environmental data and the key point cloud data is accurate.

[0058] Since environmental data and key point cloud data also have time attributes, they also need to be integrated from a time perspective. For example, the environmental data collected by the camera sensor at the first second needs to correspond to the key point cloud data collected by the lidar at the first second. When rendering the simulation scene at the first second, the environmental data and key point cloud data collected at the first second are used instead of the environmental data and key point cloud data collected at other times to ensure real-time synchronization between the simulation scene and the real scene.

[0059] This embodiment improves the realism of the simulation scene by aligning the environmental data and the key point cloud data to the same coordinate system, and integrates the data according to the time information to ensure the real-time synchronization of the simulation scene and the real scene.

[0060] In this embodiment, the simulation simulator is used to obtain the driver's viewing angle range, filter the environmental data and the key point cloud data according to the driver's viewing angle range, and retain the environmental data and key point cloud data within the driver's viewing angle range. The simulation simulator is used to perform scene rendering according to the environmental data and key point cloud data within the driver's viewing angle range.

[0061] After receiving the environment data and key point cloud data, the simulation simulator filters the data according to the driver's viewing angle to render the simulation scene from the driver's perspective. The rendered simulation scene from the driver's perspective only contains information that the driver can see. What the driver cannot see is represented in black in the simulation scene, and the content that the driver cannot see will not be displayed in the simulation scene.

[0062] In this embodiment, the sensor further includes a sensor for acquiring head posture data of the driver. The communication middleware is used to send the head posture data of the driver to the vehicle cloud routing module. The vehicle cloud routing module is used to send the head posture data of the driver to the simulation simulator. The simulation simulator is used to determine the viewing angle range of the driver according to the head posture of the driver.

[0063] During the driving process, the driver's head posture may change, such as the driver's head tilting to the left or right. As the driver's head posture changes, the driver's viewing angle will also change.

[0064] A sensor for obtaining the driver's head posture data is also provided inside the vehicle. The sensor may be a camera sensor. The camera sensor obtains an image containing the driver's head, and the driver's head posture is analyzed by a built-in algorithm. The driver's head posture data includes the deflection angle of the driver's head. The simulation simulator determines the driver's viewing angle according to the driver's head posture data. For example, if the driver's head deflects to the left, the driver's viewing angle will also deflect to the left, and the driver's view of the environment on the left side of the vehicle will increase, while the driver's view of the environment on the right side of the vehicle will decrease.

[0065] In this embodiment, after the communication middleware processes the road network data collected by the sensor according to preset data processing rules, the communication middleware is also used to encode and compress the processed road network data, and send the encoded and compressed road network data to the vehicle-cloud routing module.

[0066] For example, for image data collected by camera sensors, H.265 encoding compression can be used. For point cloud data collected by lidar, LAS ZIP compression can be used. By compressing and encoding image data and point cloud data, the data size can be reduced, thereby reducing the bandwidth requirements for data transmission.

[0067] In this embodiment, the communication middleware is used to send the encoded and compressed road network data to the vehicle-cloud routing module through a real-time communication protocol that supports point-to-point communication and low-latency data transmission.

[0068] The communication middleware transmits data streams by supporting real-time communication protocols for point-to-point communication and low-latency data transmission, which can achieve low-bitrate streaming data transmission and shorten data processing time and response time.

[0069] In this embodiment, the communication middleware is used to convert the processed road network data into string data, and send the string data to the vehicle-to-cloud routing module. The vehicle-to-cloud routing module is used to send the string data to the simulation simulator. The simulation simulator is used to extract the road network data from the driver's perspective from the string data, and perform scene rendering according to the road network data from the driver's perspective.

[0070] Road network data includes image data detected by camera sensors and point cloud data detected by lidar. Image data and point cloud data are large and take a long time to transmit. Communication middleware can convert road network data into string data and use string data to represent corresponding information. For example, road network data includes obstacle information and lane information. The string data converted from road network data also includes the obstacle information and lane information. The string data is smaller, which can speed up data transmission and improve efficiency.

[0071] The vehicle is also equipped with a SOA (Service Oriented Architecture) system, which includes multiple services. A service refers to a specific functional unit that can implement a specific function. When other modules of the vehicle need to use a certain function, they can implement it by calling the corresponding service. For example, the vehicle's laser radar service can obtain road information around the vehicle and information about other vehicles.

[0072] There are multiple domains in the vehicle, such as the smart driving domain, smart cabin domain, body domain, and vehicle-connected gateway domain. Each domain of the vehicle has its own independent service. For example, the smart cabin domain has the FOD (Function on Demand) service, and users can activate the desired vehicle-side services through the FOD service according to their own needs. The vehicle-connected gateway domain has a cellular network management service, which is responsible for monitoring the status of the dial-up cellular network.

[0073] The communication middleware can also be used to obtain data on services in various domains of the vehicle, and send the data on services in various domains of the vehicle to the vehicle-cloud routing module, wherein the service data in various domains of the vehicle include at least one of the following: service data in the smart cabin domain, service data in the smart driving domain, service data in the vehicle networking gateway domain, and service data in the vehicle body domain.

[0074] The vehicle-to-cloud routing module is used to process the received data according to preset data processing rules and send the processed data to the simulation simulator.

[0075] The simulation simulator is used to perform autonomous driving simulation training based on data received from the vehicle-to-cloud routing module.

[0076] The communication middleware can be connected to all services in the vehicle and obtain data from each service. For services in each domain of the vehicle, the communication middleware can obtain data from the service. For example, for the lidar perception data service in the intelligent driving domain, during the operation of the vehicle, the service can obtain the road network data around the vehicle and the data of other vehicles through the vehicle's lidar. The communication middleware can be connected to the lidar perception data service and obtain the road network data around the vehicle and the data of other vehicles. For example, for the cellular network management service in the vehicle-connected gateway domain, the service can monitor the status of the vehicle's cellular network, such as whether the vehicle is connected to the Internet. The communication middleware can also be connected to the cellular network management service to obtain the status information of the vehicle's cellular network.

[0077] After obtaining the data of services in various domains of the vehicle, the communication middleware sends the obtained data to the vehicle-to-cloud routing module. A variety of data processing rules are pre-configured in the vehicle-to-cloud routing module, and the vehicle-to-cloud routing module processes the received data according to the pre-configured data processing rules. Data processing is an important step in the data preprocessing process, which helps to ensure that the data is accurate, complete, and consistent. For example, the preset data processing rules include processing missing values, processing outliers, processing duplicate data, and processing inconsistent data.

[0078] After receiving the processed data sent by the vehicle-to-cloud routing module, the simulation simulator performs autonomous driving simulation training. For example, a vehicle is equipped with multiple laser radars, which can be used to sense the road network data around the vehicle. The communication middleware senses the data service through the laser radar, obtains the road network data, and sends the road network data to the vehicle-to-cloud routing module. The vehicle-to-cloud routing module processes the road network data and sends the processed road network data to the simulation simulator. The simulation simulator generates a simulation scenario based on the received processed road network data, and performs autonomous driving simulation tests in the simulation scenario.

[0079] The present invention arranges communication middleware on the vehicle side and connects the communication middleware with services in various domains of the vehicle. The communication middleware can obtain data of various services in real-time operation of the vehicle, and send the data of services in various domains of the vehicle to the simulation simulator through the vehicle-cloud routing module. The simulation simulator performs autonomous driving simulation training based on the data of services in various domains of the vehicle, thereby improving the diversity of vehicle-side simulation information transmission, allowing the simulation simulator to use more vehicle service data for autonomous driving simulation training, and improving the accuracy of autonomous driving.

[0080] In this embodiment, the communication middleware is used to obtain image data of the intelligent driving domain and the intelligent cabin domain, convert the image data into corresponding string data, and send the string data to the vehicle-cloud routing module.

[0081] The vehicle-to-cloud routing module is used to process the received string data according to the preset data processing rules, and send the processed string data to the simulation simulator.

[0082] The simulation simulator performs autonomous driving simulation training according to the processed character string data.

[0083] The communication middleware can obtain image data from the intelligent driving domain and the intelligent cabin domain. For example, the intelligent driving domain includes the lidar perception data service, and the intelligent cabin domain includes the camera monitoring service. The communication middleware obtains the environmental image data around the vehicle through the lidar data perception service of the intelligent driving domain, and the communication middleware obtains the image data inside the vehicle through the camera monitoring service of the intelligent cabin domain.

[0084] These image data are acquired in real time by laser radar, cameras and other equipment during the driving process of the vehicle. Usually, the image data is large. If the image data is sent directly to the simulation simulator deployed in the cloud, on the one hand, the data transmission takes a long time, and on the other hand, the amount of data that the simulation simulator needs to process is large, which affects the efficiency of the autonomous driving simulation test.

[0085] In this embodiment, the communication middleware can convert image data into string data. For example, the image data of the vehicle's surrounding environment is converted into string data of the vehicle's surrounding environment through the communication middleware, and the string data of the vehicle's surrounding environment also contains the environmental information around the vehicle. Specifically, the communication middleware can complete the conversion of image data through a serialization library, such as JSON, XML or Protobuf. The simulation simulator can generate a simulation environment based on the string data of the vehicle's surrounding environment. Compared with image data, the amount of string data is smaller and can be transmitted to the simulation simulator in the cloud more quickly. It can also reduce the time it takes for the simulation simulator to process data and improve the efficiency of autonomous driving simulation testing.

[0086] In this embodiment, the communication middleware is used to obtain the data of the function on-demand activation service of the smart cabin domain, and send the data of the function on-demand activation service to the vehicle-cloud routing module. The data of the function on-demand activation service includes the activation status data of the target function of the vehicle.

[0087] The vehicle-to-cloud routing module is used to process the data of the function-on-demand service according to the preset data processing rules, and send the processed data of the function-on-demand service to the simulation simulator.

[0088] The simulation simulator is used to configure the target function of the vehicle in simulation training according to the activation state data of the target function of the vehicle.

[0089] In order to adapt to the personalized needs of users, the vehicle has a function on-demand activation service. Users can use the function on-demand activation service to activate the functions they need, such as point to point autopilot (PPA) and home automated valet parking assist (HAVP).

[0090] The simulation simulator can obtain the activated functions of the vehicle based on the data of the function on-demand activation service. If the target function is activated, the simulation simulator will also configure the target function for the vehicle during simulation training, so that the functions of the vehicle during simulation training are consistent with the functions of the real vehicle. The simulation simulator can perform autonomous driving training for vehicles with different functions, making the autonomous driving training process more targeted.

[0091] In this embodiment, the communication middleware is used to obtain the data of the digital key service in the body domain and the data of the vehicle status mode service in the vehicle connection gateway domain, and send the data of the digital key service in the body domain and the data of the vehicle status mode service in the vehicle connection gateway domain to the vehicle cloud routing module.

[0092] The vehicle-cloud routing module is used to process the data of the digital key service in the vehicle body domain and the data of the vehicle status mode service in the vehicle connection gateway domain according to the preset data processing rules, and send the processed data of the digital key service in the vehicle body domain and the data of the vehicle status mode service in the vehicle connection gateway domain to the simulation simulator.

[0093] The simulation simulator is used to determine whether the vehicle is in a state of about to start or a locked state based on the data of the digital key service and the data of the vehicle status mode service.

[0094] When the vehicle is about to start, the simulation simulator is used to determine that the vehicle needs to be tracked.

[0095] When the vehicle is in a locked state, the simulation simulator is used to determine that there is no need to track the vehicle.

[0096] After the digital key service is enabled, a digital key for the vehicle can be generated in the user's electronic device, and the user can use the digital key in the electronic device to open or lock the car door. For example, a digital key for the vehicle is generated in the user's mobile phone, and the car door is automatically opened when the user puts the mobile phone close to the sensing area of ​​the car door.

[0097] The vehicle's status mode service can be used to obtain the vehicle's status, such as whether the vehicle is in the started state or the unstarted state. Combining the data of the vehicle's status mode service and the data of the digital key service, it can be determined whether the vehicle is in the state of about to start or locked.

[0098] If the vehicle is about to start, then after the vehicle starts, the services of each domain of the vehicle will run and generate data. At this time, it is necessary to track the vehicle to obtain the data of the services of each domain of the vehicle, and perform autonomous driving simulation training based on the obtained data.

[0099] If the vehicle is locked, it will not start and no service data will be generated. In this case, there is no need to track the vehicle.

[0100] This embodiment allows the communication middleware to obtain the data of the digital key service and the data of the vehicle status mode service, and determines whether the vehicle needs to be tracked based on the obtained data, thereby avoiding tracking of locked vehicles and improving the efficiency of autonomous driving simulation training.

[0101] In this embodiment, the communication middleware is used to obtain data of the cellular network management service of the vehicle-connected gateway domain, and send the data of the cellular network management service to the vehicle-cloud routing module.

[0102] The vehicle-to-cloud routing module is used to process the data of the cellular network management service of the vehicle-connected gateway domain according to the preset data processing rules, and send the processed data of the cellular network management service of the vehicle-connected gateway domain to the simulation simulator.

[0103] The simulation simulator is used to distinguish the regional scope of each mobile communication provider based on the data of the cellular network management service of the vehicle connection gateway domain, and confirm the scope and status of the vehicle signal in the simulation scenario.

[0104] The vehicle has a cellular network management service. The cellular network management service is used for dialing and network status monitoring. After dialing successfully, the vehicle is connected to the cellular network. The cellular network management service can also be used to monitor the vehicle's network status and confirm whether the vehicle is connected to the network. Different vehicles can use network services provided by different mobile communication providers.

[0105] The simulation simulator can combine the data of the cellular network management service and the simulation scenario to confirm the range and status of each vehicle signal in the simulation scenario. When the vehicle signal can be monitored, the vehicle can be tracked, and the service data of each domain of the vehicle can be obtained in real time to conduct autonomous driving simulation training.

[0106] In this embodiment, the communication middleware is used to obtain data of the lidar perception data service of the intelligent driving domain, and send the data of the lidar perception data service of the intelligent driving domain to the vehicle-cloud routing module. The data of the lidar perception data service of the intelligent driving domain includes at least road network data.

[0107] The vehicle-cloud routing module is used to process the data of the laser radar perception data service of the intelligent driving domain according to the preset data processing rules, and send the processed data of the laser radar perception data service of the intelligent driving domain to the simulation simulator.

[0108] The simulation simulator is used to generate a simulation scenario based on the processed data of the lidar perception data service of the intelligent driving domain.

[0109] The vehicle has a laser radar perception data service. For example, the vehicle is equipped with multiple laser radars, which can be used to perceive the road network data around the vehicle. The communication middleware can obtain the road network data around the vehicle through the laser radar perception data service. The simulation simulator generates a simulation scenario based on the received road network data.

[0110] In this embodiment, after the communication middleware obtains the data of services in various domains of the vehicle, the communication middleware is also used to encrypt the data of services in various domains of the vehicle and send the encrypted data to the vehicle-cloud routing module.

[0111] After obtaining the service data of each domain of the vehicle, the communication middleware encrypts the obtained data through a preset encryption algorithm. The data transmitted from the communication middleware to the vehicle-cloud routing module is encrypted data to improve security.

[0112] Furthermore, the data of each service can be classified in advance, and the data can be divided into important data and non-important data according to the function of each service. If the data of the service obtained by the communication middleware is important data, the communication middleware encrypts the important data and sends it to the car-cloud routing module. If the data obtained by the communication middleware is non-important data, the communication middleware can directly send the non-important data to the car-cloud routing module.

[0113] By dividing the data of each service into important data and unimportant data, the communication middleware only encrypts the important data and does not encrypt the unimportant data, thereby further improving efficiency while ensuring security.

[0114] In this embodiment, the communication middleware is used to send the data of services in various domains of the vehicle to the vehicle-cloud routing module through a real-time communication protocol that supports point-to-point communication and low-latency data transmission.

[0115] The communication middleware transmits data streams by supporting real-time communication protocols for point-to-point communication and low-latency data transmission, which can achieve low-bitrate streaming data transmission and shorten data processing time and response time.

[0116] The communication middleware can also use the method of defining a state machine to manage the state of the transmission data event. The state machine is a computer mathematical model that describes the behavior of the system. It contains 11 states: closed, listening, SYN received, established, waiting for FIN1, waiting for FIN2, waiting for closing, closing, last confirmation, waiting time, and aborted. The closed state is the initial state. The state machine is used to indicate whether the communication middleware has sent the data to the car-cloud routing module.

[0117] For vehicle services, they usually include a server and a client. The server is used to provide services, and the client is used to use services. Each service is provided by a server, and a service can be used by multiple clients. Communication middleware can be used for service registration and discovery. The service discovery function can be implemented using a service registry, which includes available services and the address information of the service. The client discovers available services and the address information of the service by querying the service registry, and calls the service through the address information of the available service.

[0118] like Figure 2 As shown, this embodiment introduces a rendering method for an autonomous driving simulation scene, and the method is applied to the rendering system for an autonomous driving simulation scene described in any embodiment of the present invention, and the method includes steps 2100-2300.

[0119] Step 2100: The communication middleware obtains the road network data collected by the sensor, processes the road network data collected by the sensor according to preset data processing rules, and sends the processed road network data to the vehicle-cloud routing module.

[0120] Step 2200: The vehicle-to-cloud routing module sends the processed road network data to the simulation simulator.

[0121] Step 2300: The simulation simulator extracts the road network data from the driver's perspective from the processed road network data, and performs scene rendering based on the road network data from the driver's perspective.

[0122] The present invention collects road network data through sensors installed on the vehicle, filters the road network data, retains only the road network data from the driver's perspective, and finally renders the scene according to the road network data from the driver's perspective to display the simulated scene from the driver's perspective, which can better simulate the real driving experience, allowing research and development engineers to more realistically feel the movement of the vehicle and changes in the surrounding environment, and more clearly understand the pain points of the driver in actual driving, so that research and development engineers can develop automatic driving functions based on the pain points of the driver in actual driving.

[0123] Optionally, the sensor includes a camera sensor and a laser radar, and the road network data collected by the sensor includes image data collected by the camera sensor and point cloud data collected by the laser radar.

[0124] Optionally, the processing of the road network data collected by the sensor according to a preset data processing rule includes:

[0125] The communication middleware extracts environmental data from the image data collected by the camera sensor according to a preset video SLAM algorithm based on optical flow and feature matching;

[0126] The communication middleware processes the point cloud data according to a preset point cloud segmentation and tracking algorithm, and extracts key point cloud data from the point cloud data, wherein the key point cloud data includes at least lane point cloud data and obstacle point cloud data.

[0127] Optionally, the processing of the road network data collected by the sensor according to a preset data processing rule further includes:

[0128] The communication middleware aligns the environmental data and the key point cloud data to the same coordinate system according to a preset local feature alignment algorithm, and integrates the environmental data and the key point cloud data according to the time information of the environmental data and the time information of the key point cloud data.

[0129] Optionally, the step 2300 includes:

[0130] The simulation simulator obtains the driver's viewing angle range, filters the environmental data and the key point cloud data according to the driver's viewing angle range, and retains the environmental data and the key point cloud data within the driver's viewing angle range;

[0131] The simulation simulator performs scene rendering according to the environmental data and key point cloud data within the driver's viewing angle.

[0132] Optionally, the sensor further comprises a sensor for acquiring head posture data of the driver;

[0133] Before the simulation simulator acquires the driver's viewing angle range, the method further includes:

[0134] The communication middleware sends the driver's head posture data to the vehicle-cloud routing module;

[0135] The vehicle-to-cloud routing module sends the driver's head posture data to the simulation simulator;

[0136] The simulation simulator obtains the driver's viewing angle range, including:

[0137] The simulation simulator determines the driver's viewing angle range according to the driver's head posture.

[0138] Optionally, the method further comprises:

[0139] After the communication middleware processes the road network data collected by the sensor according to preset data processing rules, the communication middleware encodes and compresses the processed road network data, and sends the encoded and compressed road network data to the vehicle-to-cloud routing module.

[0140] This embodiment introduces a computer program product, including a computer program / instruction, which, when executed by a processor, implements a method for rendering an autonomous driving simulation scene as described in any embodiment of the present invention.

[0141] This embodiment introduces a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the rendering method of the autonomous driving simulation scene as described in any embodiment of the present invention are implemented.

[0142] The method in the present application can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the present application is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0143] The computer programs / instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing / processing device.

[0144] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0145] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0146] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0147] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0148] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0149] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A rendering system for an autonomous driving simulation scene, characterized in that: It includes communication middleware, vehicle-to-cloud routing module and simulation simulator, wherein the communication middleware and vehicle-to-cloud routing module are deployed on the vehicle side, and the simulation simulator is deployed on the cloud side; The communication middleware is used to obtain the road network data collected by the sensor of the vehicle, process the road network data collected by the sensor according to the preset data processing rules, and send the processed road network data to the vehicle-cloud routing module; The vehicle-to-cloud routing module is used to send the processed road network data to the simulation simulator; The simulation simulator is used to extract the road network data from the driver's perspective from the processed road network data, and perform scene rendering according to the road network data from the driver's perspective.

2. The system according to claim 1, characterized in that The sensor includes a camera sensor and a laser radar, and the road network data collected by the sensor includes image data collected by the camera sensor and point cloud data collected by the laser radar; The communication middleware is used to extract environmental data from the image data collected by the camera sensor according to a preset video SLAM algorithm based on optical flow and feature matching; The communication middleware is used to process the point cloud data according to a preset point cloud segmentation and tracking algorithm, and extract key point cloud data from the point cloud data, wherein the key point cloud data includes at least lane point cloud data and obstacle point cloud data.

3. The system according to claim 2, characterized in that The communication middleware is used to align the environmental data and the key point cloud data to the same coordinate system according to a preset local feature alignment algorithm, and integrate the environmental data and the key point cloud data according to the time information of the environmental data and the time information of the key point cloud data.

4. The system according to claim 3, characterized in that The simulation simulator is used to obtain the viewing angle range of the driver, filter the environmental data and the key point cloud data according to the viewing angle range of the driver, and retain the environmental data and the key point cloud data within the viewing angle range of the driver; The simulation simulator is used for performing scene rendering according to the environmental data and key point cloud data within the driver's viewing angle.

5. The system according to claim 4, characterized in that The sensor also includes a sensor for acquiring head posture data of the driver; The communication middleware is used to send the driver's head posture data to the vehicle-to-cloud routing module; The vehicle-to-cloud routing module is used to send the driver's head posture data to the simulation simulator; The simulation simulator is used to determine the driver's viewing angle range according to the driver's head posture.

6. The system according to claim 1, characterized in that After the communication middleware processes the road network data collected by the sensor according to preset data processing rules, the communication middleware is also used to encode and compress the processed road network data, and send the encoded and compressed road network data to the vehicle-cloud routing module.

7. The system according to claim 6, characterized in that The communication middleware is used to send the encoded and compressed road network data to the vehicle-cloud routing module through a real-time communication protocol that supports point-to-point communication and low-latency data transmission.

8. The system according to claim 1, characterized in that The communication middleware is used to convert the processed road network data into character string data, and send the character string data to the vehicle cloud routing module; The vehicle-to-cloud routing module is used to send the string data to the simulation simulator; The simulation simulator is used to extract the road network data from the driver's perspective from the character string data, and perform scene rendering according to the road network data from the driver's perspective.

9. A method for rendering an autonomous driving simulation scene, characterized in that: The method is applied to a rendering system of an autonomous driving simulation scene according to any one of claims 1 to 8, and the method comprises: The communication middleware obtains the road network data collected by the sensor, processes the road network data collected by the sensor according to a preset data processing rule, and sends the processed road network data to the vehicle-to-cloud routing module; The vehicle-to-cloud routing module sends the processed road network data to the simulation simulator; The simulation simulator extracts the road network data from the driver's perspective from the processed road network data, and performs scene rendering according to the road network data from the driver's perspective.

10. A computer program product, characterized in that The method comprises a computer program / instruction which, when executed by a processor, implements the method as claimed in claim 9.

Images