Virtual and real vehicle simulation method and system based on digital twinning and electronic equipment

Abstract

The invention discloses a virtual and real vehicle simulation method and system based on digital twinning and electronic equipment. The method comprises the steps that first real state information used for representing the vehicle state of a real vehicle and first virtual state information used for representing the vehicle state of a virtual vehicle are acquired; based on the first real state information and the first virtual state information, predicting the next vehicle state of the real vehicle to generate a first prediction result, and predicting the next vehicle state of the virtual vehicle to generate a second prediction result; adjusting the first real state information based on the first prediction result to generate corresponding second real state information, and adjusting the first virtual state information based on the second prediction result to generate corresponding second virtual state information; and sending the second real state information to the virtual vehicle, and sending the second virtual state information to the real vehicle. According to the method, the digital twinning technology can be utilized to realize accurate simulation of the virtual vehicle to the real vehicle.

Description

Technical Field

[0001] The present application relates to the field of simulation of autonomous driving vehicles, and in particular to a virtual-reality vehicle simulation method, system, and electronic equipment based on digital twins. Background Art

[0002] In the field of autonomous driving, with the accelerated development of connectivity and intelligent driving, digital and connected vehicle features are being developed. Testing these features in real-world environments requires constructing diverse driving environments, requiring inestimable resources, manpower, and costs. Furthermore, specific test scenarios and operating conditions can pose safety risks. This creates increasingly high demands on autonomous vehicle simulation platforms. However, current simulation platforms lack the ability to fully interact between virtual and real vehicles, making it difficult to accurately simulate real vehicles using virtual vehicles. Summary of the Invention

[0003] The purpose of the embodiments of the present application is to provide a virtual-reality vehicle simulation method, system and electronic equipment based on digital twins, which can use the technology of digital twins to achieve accurate simulation of virtual vehicles and real vehicles.

[0004] To achieve the above objectives, the present application provides a virtual-real vehicle simulation method based on digital twins, comprising:

[0005] Acquiring, through the digital twin system, first real state information representing a vehicle state of a real vehicle and first virtual state information representing a vehicle state of a virtual vehicle, wherein a driving action between the real vehicle and the virtual vehicle is associated;

[0006] Based on the first real state information and the first virtual state information, in the digital twin system, predicting a next vehicle state of the real vehicle to generate a first prediction result, and predicting a next vehicle state of the virtual vehicle to generate a second prediction result;

[0007] Adjusting the first real state information based on the first prediction result to generate corresponding second real state information, and adjusting the first virtual state information based on the second prediction result to generate corresponding second virtual state information;

[0008] The second real state information is sent to the virtual vehicle, and the second virtual state information is sent to the real vehicle, so that the real vehicle adjusts its driving behavior based on the second virtual state information, and the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle.

[0009] Optionally, the method further includes:

[0010] When the real vehicle adjusts its driving behavior based on the second virtual state information and performs an obstacle avoidance action, reacquiring the first real state information;

[0011] When the virtual vehicle adjusts its driving behavior based on the second real state information and performs an obstacle avoidance action, the first virtual state information is reacquired.

[0012] Optionally, predicting the next vehicle state of the real vehicle to generate a first prediction result, and predicting the next vehicle state of the virtual vehicle to generate a second prediction result, include:

[0013] Determining, based on a Kalman filter model, a first optimal predicted estimate of a next vehicle state of the real vehicle and a second optimal predicted estimate of a next vehicle state of the virtual vehicle;

[0014] determining a first prediction error for predicting a next vehicle state of the real vehicle and a second prediction error for predicting a next vehicle state of the virtual vehicle;

[0015] determining the first prediction result based on the first optimal prediction estimate and the first prediction error;

[0016] The second prediction result is determined based on the second optimal prediction estimate and the second prediction error.

[0017] Optionally, the method further includes:

[0018] determining a corresponding first posterior probability based on the first prediction error;

[0019] Accordingly, determining the first prediction result based on the first optimal prediction estimate and the first prediction error includes:

[0020] The first prediction result is determined based on the first optimal prediction estimate and the first posterior probability.

[0021] Optionally, the method further includes:

[0022] determining a corresponding second posterior probability based on the second prediction error;

[0023] Accordingly, determining the second prediction result based on the second optimal prediction estimate and the second prediction error includes:

[0024] The second prediction result is determined based on the second optimal prediction estimate and the second posterior probability.

[0025] Optionally, the digital twin system includes a physical space and an information space that are interactively mapped to each other, the physical space includes entities in reality, and the entities have corresponding physical relationships, and the information space includes physical models corresponding to the entities, and the physical models are interactively mapped to the entities;

[0026] Accordingly, the method further includes associating driving actions between the real vehicle and the virtual vehicle based on an interactive mapping between the physical model and the entity.

[0027] Optionally, the method further includes:

[0028] Based on the simulated operation of the virtual vehicle on the real vehicle, the standardization and robustness of the autonomous driving perception, decision information and / or planning algorithm of the real vehicle are verified.

[0029] Optionally, the first real state information includes at least one of the following: real-time changes in the speed, position, and orientation of the real vehicle, wherein the position of the real vehicle is the coordinate position of the real vehicle in the road model, and the orientation of the real vehicle is the angle between the vehicle head direction and the geometric coordinate axis of the real vehicle itself;

[0030] The first virtual state information includes at least one of the following: real-time changes in the speed, position and orientation of the virtual vehicle, wherein the position of the virtual vehicle is the coordinate position of the virtual vehicle in the road model, and the orientation of the virtual vehicle is the angle between the front direction of the vehicle and the geometric coordinate axis of the virtual vehicle itself.

[0031] The embodiment of the present application also provides a virtual-reality vehicle simulation system based on digital twins, including:

[0032] an acquisition module configured to acquire, through the digital twin system, first real state information representing a vehicle state of a real vehicle and first virtual state information representing a vehicle state of a virtual vehicle, wherein a driving action between the real vehicle and the virtual vehicle is associated;

[0033] a prediction module configured to, based on the first real state information and the first virtual state information, predict a next vehicle state of the real vehicle in the digital twin system to generate a first prediction result, and predict a next vehicle state of the virtual vehicle to generate a second prediction result;

[0034] an adjustment module configured to adjust the first real state information based on the first prediction result to generate corresponding second real state information, and to adjust the first virtual state information based on the second prediction result to generate corresponding second virtual state information;

[0035] A processing module is configured to send the second real state information to the virtual vehicle, and send the second virtual state information to the real vehicle, so that the real vehicle adjusts its driving behavior based on the second virtual state information, and so that the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle.

[0036] An embodiment of the present application further provides an electronic device, including a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method described above.

[0037] The simulation method of the embodiment of the present application utilizes the technology of digital twins to send the accurate driving status of a real vehicle to a virtual vehicle, and vice versa, thereby realizing virtual-reality interaction between virtual scenes and actual scenes, and solving the vehicle status differences between virtual vehicles and real vehicles caused by information transmission errors, thereby realizing accurate simulation of real vehicles using virtual vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flowchart of a virtual-real vehicle simulation method based on digital twins according to an embodiment of the present application;

[0039] Figure 2 For the embodiment of this application Figure 1 Flowchart of an embodiment of step S200;

[0040] Figure 3 Schematic diagram of the connection relationship between the digital twin system of an embodiment of the present application and the real vehicle and virtual vehicle respectively;

[0041] Figure 4 A flowchart of a specific embodiment of the virtual-real vehicle simulation method according to an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a process for maintaining the temporal and spatial consistency of a real vehicle and a virtual vehicle through a digital twin system in an embodiment of the present application;

[0043] Figure 6 Schematic diagram of the structure of the digital twin system in the embodiment of this application;

[0044] Figure 7 This is a structural block diagram of a virtual-reality vehicle simulation system based on digital twins according to an embodiment of the present application;

[0045] Figure 8 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] Various aspects and features of the present application are described herein with reference to the accompanying drawings.

[0047] It should be understood that various modifications may be made to the embodiments of the present application. Therefore, the above description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present application will occur to those skilled in the art.

[0048] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present application given above and the detailed description of the embodiments given below, serve to explain the principles of the present application.

[0049] These and other characteristics of the present application will become apparent from the following description of a preferred form of embodiment given as a non-limiting example with reference to the accompanying drawings.

[0050] It should also be understood that although the present application has been described with reference to certain specific examples, those skilled in the art will readily be able to implement many other equivalent forms of the present application.

[0051] The above and other aspects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0052] Specific embodiments of the present application will be described hereinafter with reference to the accompanying drawings; however, it should be understood that the embodiments described are merely examples of the present application and may be implemented in a variety of ways. Familiar and / or repetitive functions and structures are not described in detail to avoid obscuring the present application with unnecessary or redundant details. Therefore, the specific structural and functional details described herein are not intended to be limiting, but rather serve merely as a basis and representative basis for the claims to teach those skilled in the art to variously utilize the present application with substantially any suitable detailed structure.

[0053] This specification may use the phrases "in one embodiment," "in another embodiment," "in yet another embodiment," or "in other embodiments," which may all refer to one or more of the same or different embodiments according to the present application.

[0054] The present application provides a method for simulating a virtual and real vehicle based on digital twins. This method, based on digital twin-related technologies, can transmit the vehicle status of both the virtual vehicle and the real vehicle to the digital twin system. The real vehicle can be driven autonomously by an autonomous driving system, while the virtual vehicle can be controlled by a driving simulator. The digital twin system can be a simulation platform built based on the digital twin.

[0055] Since the process of sending the vehicle status to the digital twin system will generate communication delays, packet loss, data duplication and other problems. The digital twin system will predict the next vehicle status of the virtual vehicle and the real vehicle respectively, thereby adjusting the vehicle status input into the digital twin system. Then, the adjusted second real state information of the real vehicle is sent to the virtual vehicle, and the second virtual state information of the virtual vehicle is sent to the real vehicle, so that the real vehicle adjusts its autonomous driving behavior based on the second virtual state information, and the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle. This adjustment of driving behavior can be realized dynamically and continuously, thereby avoiding differences when the virtual vehicle in the digital twin system verifies the real vehicle, and then the correctness and robustness of the autonomous driving algorithm in the autonomous driving system of the real vehicle can be verified based on the simulation platform.

[0056] The following is a detailed description of the digital twin-based virtual-reality vehicle simulation method of the present application with reference to the accompanying drawings. Figure 1 This is a flow chart of a virtual-real vehicle simulation method based on digital twins according to an embodiment of the present application, as shown in FIG. Figure 1 Shown and combined Figure 3 and Figure 4 , the method comprises the following steps:

[0057] S100, obtaining, through a digital twin system, first real state information for characterizing a vehicle state of a real vehicle, and first virtual state information for characterizing a vehicle state of a virtual vehicle, wherein the driving actions between the real vehicle and the virtual vehicle are associated.

[0058] For example, a digital twin is a virtual entity that creates a physical entity in a digital way. It is a technical means to simulate, verify, predict, and control the entire life cycle of a physical entity with the help of historical data, real-time data, and algorithmic models. Digital twins use information technologies such as perception, computing, and modeling to describe, diagnose, predict, and make decisions about physical space through software definition, thereby realizing interactive mapping between physical space and cyberspace (i.e., digital virtual space). The digital twin system in this embodiment is built based on digital twins. The digital twin system can be used as a simulation platform, in which a virtual vehicle associated with a real vehicle is constructed, and it can process information related to the real vehicle and the virtual vehicle based on its own characteristics, thereby realizing interactive mapping between the two.

[0059] Specifically, the real vehicle can be a physical vehicle in a physical space. In one embodiment, the real vehicle is an intelligent connected car with vehicle-road communication in a physical space, and the car can be driven by an automatic driving system. The first real state information corresponding to the real vehicle represents relevant information about the vehicle state of the real vehicle. The vehicle state can be the vehicle state when the relevant information is collected, and it is the previous vehicle state relative to the next vehicle state. For example, it can be information such as the driving speed and position of the real vehicle. The first real state information can be collected by the real vehicle's own equipment and / or road collection equipment, for example, by a road test unit, wherein the road test unit includes sensors, communication equipment, etc. The collected first real state information is then sent to the digital twin system through network communication (such as ZMQ communication).

[0060] A virtual vehicle can be a virtual vehicle that is constructed by linking the driving actions of a real vehicle to simulate the real vehicle. For example, by linking the driving actions of a virtual vehicle to a real vehicle, the virtual vehicle can simulate the driving scenarios of the real vehicle. If the real vehicle is involved in an accident or conflicts with other vehicles in physical space, the associated virtual vehicle can also simulate the corresponding actions in the virtual space.

[0061] The first virtual state information of the virtual vehicle can represent relevant information about the vehicle state of the virtual vehicle. The vehicle state can be the vehicle state when the relevant information is collected, which is the previous vehicle state relative to the next vehicle state. For example, it can be the driving speed, position information, etc. of the virtual vehicle. The virtual vehicle can be controlled based on a driving simulator (such as the ScanerStudio driving simulator), and the first virtual state information can be obtained by the driving simulator and sent to the digital twin system.

[0062] S200, based on the first real state information and the first virtual state information, in the digital twin system, predicting the next vehicle state of the real vehicle to generate a first prediction result, and predicting the next vehicle state of the virtual vehicle to generate a second prediction result.

[0063] For example, combined Figure 5In the process of sending the first real state information and the first virtual state information to the digital twin system, transmission errors may occur due to various reasons. For example, when the ScanerStudio driving simulator and the road test unit send the first virtual state information and the first real state information of the virtual vehicle and the real vehicle respectively to the digital twin system via ZMQ communication, communication delays, packet loss, data duplication and other problems may occur. In this embodiment, the digital twin system is used to predict the next vehicle state of the real vehicle and the virtual vehicle based on the first real state information, the first virtual state information and the current driving scenario, thereby determining the action behavior of the real vehicle and the virtual vehicle at the next moment.

[0064] In one embodiment, in a digital twin system, the next vehicle state of a real vehicle and a virtual vehicle can be predicted based on a Kalman filter model. This includes predicting the next vehicle state of the real vehicle to generate a first prediction result, and predicting the next vehicle state of the virtual vehicle to generate a second prediction result. For example, the first prediction result represents the vehicle state at the next moment when the digital twin system receives the first real state of the real vehicle, and is a prediction of the state of the real vehicle. This can overcome problems such as communication delay, packet loss, and data duplication that occur when the first real state information is sent to the digital twin system. The second prediction result represents the vehicle state at the next moment when the digital twin system receives the first virtual state of the virtual vehicle, and is a prediction of the state of the virtual vehicle. This can overcome problems such as communication delay, packet loss, and data duplication that occur when the first virtual state information is sent to the digital twin system.

[0065] S300, adjusting the first real state information based on the first prediction result to generate corresponding second real state information, and adjusting the first virtual state information based on the second prediction result to generate corresponding second virtual state information.

[0066] For example, combined Figure 4 The first prediction result represents the vehicle state at the next moment when the digital twin system receives the first real-world state information of the real vehicle. Adjusting the first real-world state information of the real vehicle to the second real-world state information based on the first prediction result can compensate for errors that occur when the first real-world state information is sent to the digital twin system. The adjusted second real-world state information truly reflects the current state of the real vehicle, such as its current speed and position. The vehicle state of the real vehicle, represented by the second real-world state information, thus ensures that the real vehicle maintains consistency in its performance in the real world and in the digital twin system.

[0067] The second prediction result represents the vehicle state at the next moment when the digital twin system receives the first virtual state information of the virtual vehicle. Adjusting the first virtual state information of the virtual vehicle to the second virtual state information based on the second prediction result can compensate for errors that occur when the first virtual state information is sent to the digital twin system. The adjusted second virtual state information truly reflects the current state of the virtual vehicle, such as the current driving speed and position of the virtual vehicle. The vehicle state of the virtual vehicle, represented by the second virtual state information, thus ensures that the virtual vehicle maintains consistency in its performance in the virtual scene and in the digital twin system.

[0068] S400, sending the second real state information to the virtual vehicle, and sending the second virtual state information to the real vehicle, so that the real vehicle adjusts its driving behavior based on the second virtual state information, and the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle.

[0069] For example, the virtual vehicle can adjust its driving behavior based on the current state of the real vehicle, thereby being able to interactively map with the real vehicle and accurately simulate the driving state of the real vehicle. Through the digital twin system, the second real state information can be sent to the virtual vehicle, and the second real state information truly reflects the current state of the real vehicle. Therefore, after receiving the second real state information, the virtual vehicle can adjust its driving behavior based on the second real state information to accurately simulate the real vehicle. For example, the second real state information is mapped in the ScanerStudio driving simulator of the virtual vehicle and the ScanerStudio driving simulator takes corresponding behavioral actions in a timely manner.

[0070] In addition, combined Figure 4 The digital twin system can also send the second virtual state information to the real vehicle, and the second virtual state information truly reflects the current state of the virtual vehicle. After receiving the second virtual state information, the real vehicle can adjust the driving behavior of the real vehicle based on the second virtual state information, so that the decision module of the real vehicle can obtain relevant information of the virtual vehicle in time and make corresponding decision behaviors, thereby keeping the vehicle state of the real vehicle consistent with that of the virtual vehicle.

[0071] The simulation method of the embodiment of the present application utilizes digital twin technology to transmit the accurate driving status of the real vehicle to the virtual vehicle, and vice versa, thereby realizing virtual-reality interaction between the virtual scene (simulation platform) and the actual scene, and resolving the vehicle status difference between the virtual vehicle and the real vehicle due to information transmission errors, thereby realizing accurate simulation of the real vehicle using the virtual vehicle. Furthermore, it is also possible to verify the correctness and robustness of the autonomous driving algorithm in the autonomous driving system of the real vehicle in real time, and enable the real vehicle to make self-driving adjustments based on the status information received from the virtual vehicle, thereby increasing the safety and intelligence of the real vehicle.

[0072] In one embodiment of the present application, the method further comprises the following steps:

[0073] When the real vehicle adjusts its driving behavior based on the second virtual state information and performs an obstacle avoidance action, reacquiring the first real state information;

[0074] When the virtual vehicle adjusts its driving behavior based on the second real state information and performs an obstacle avoidance action, the first virtual state information is reacquired.

[0075] Exemplarily, the real and virtual vehicles have cycles in the process of adjusting their driving behaviors, with each cycle transitioning to the next. This prevents logical confusion during the continuous adjustment of driving behaviors between the real and virtual vehicles. Specifically, when the real vehicle adjusts its driving behavior based on the second virtual state information and implements an obstacle avoidance maneuver, it can be determined that the real vehicle has completed the current cycle of driving behavior adjustment and will enter the next cycle of driving behavior adjustment. The real vehicle then reacquires the first real state information and retransmits it to the digital twin system.

[0076] Similarly, when the virtual vehicle adjusts its driving behavior based on the second real state information and implements obstacle avoidance action, it can be determined that the virtual vehicle has completed the driving behavior adjustment of the current cycle and will enter the driving behavior adjustment of the next cycle, so that the virtual vehicle re-acquires the first virtual state information and re-sends the first virtual state information to the digital twin system.

[0077] In one embodiment of the present application, the next vehicle state of the real vehicle is predicted to generate a first prediction result, and the next vehicle state of the virtual vehicle is predicted to generate a second prediction result, such as Figure 2 Shown and combined Figure 5 ,include:

[0078] S210, determining a first optimal predicted estimated value for the next vehicle state of the real vehicle and a second optimal predicted estimated value for the next vehicle state of the virtual vehicle based on a Kalman filter model;

[0079] For example, Kalman filtering is an algorithm that uses linear system state equations and observation data from the system input and output to optimally estimate the system state. In this embodiment, a Kalman filter model can be used in the digital twin system to predict the next vehicle state of each of the real vehicle and the virtual vehicle.

[0080] In a specific embodiment, the process of using the Kalman filter model includes:

[0081] For each vehicle, from time k-1 to time k, the vehicle state at time k is predicted based on the vehicle state at time k-1, as shown in formula (1):

[0082] x k =F k x k-1 +B k u k +ω k (1)

[0083] Formula (1) is the system state prediction equation, where F k is the state transfer matrix. k It is the input-control model acting on the controller vector. This term does not exist in general kinematics because it is impossible to measure the internal control quantity of the target to be detected, so it is simplified to 0. k is the process noise. Its mean is 0. This corresponds to the mean of the Gaussian distribution, so this term can be simplified to 0.

[0084] Define z based on k-1 moments k is the observation value at the current moment, as shown in formula (2):

[0085] z k =H k x k-1 +v k (2)

[0086] Formula (2) is the state estimation transfer equation, where H k is the observation matrix, which maps the real state space into the observation space, v k is the observation noise, with a mean of 0. This corresponds to the mean of the Gaussian distribution, so this term can be simplified to 0.

[0087] Predict the state at time k based on the state at time (k-1), where the specific form is: define the state at time k-1 as x k-1, and its corresponding covariance is As shown in formula (3):

[0088]

[0089] Formula (3) is the state estimation equation, where: is the state prediction at time k-1, is the state estimate at time k-1, F k is the state transition matrix.

[0090] Estimate the system prediction error P at the kth moment based on the coefficient prediction error at the k-1th moment k|k-1 , as shown in formula (4):

[0091]

[0092] Formula (4) is the prediction estimation covariance equation, which calculates the prior probability. k-1|k-1 is the posterior estimation error covariance matrix at k-1 time, which measures the accuracy of the estimated value, P k|k-1 is the estimated error covariance matrix from time k-1 to time k, Q k is the process noise covariance matrix, the larger it is, the less confident the prediction is.

[0093] The Kalman gain is calculated based on the estimated error covariance matrix, as shown in formula (5):

[0094]

[0095] Formula (3) is the Kalman gain equation, where K k is the optimal Kalman gain. R k is the measurement noise covariance matrix, the larger it is, the less trustworthy the observation is.

[0096] According to formulas (1), (2), (3), (4), and (5), the optimal estimated value of the system is calculated as follows:

[0097]

[0098] Equation (6) is the updated state estimation equation, which is the final filtering state result.

[0099] In the process of predicting the next vehicle state for the real vehicle and the virtual vehicle respectively, the above-mentioned processing flow is implemented respectively using the Kalman filter model, so that the first optimal prediction valuation of the real vehicle as represented by formula (6) and the second optimal prediction valuation of the virtual vehicle as represented by formula (6) are calculated by the above-mentioned formulas (1) to (5).

[0100] S220, determining a first prediction error for predicting the next vehicle state of the real vehicle and a second prediction error for predicting the next vehicle state of the virtual vehicle;

[0101] For example, when using the Kalman filter model to predict the next vehicle state of each of the real vehicle and the virtual vehicle, errors may occur in the prediction process. In this embodiment, a first prediction error corresponding to the real vehicle and a second prediction error corresponding to the virtual vehicle are determined.

[0102] Continuing with the above specific embodiment, the system prediction error at the current moment can be calculated according to the above formulas (1), (2), (3), (4), and (5), as shown in formula (7):

[0103] P k|k =(IK k H k )P k|k-1 (7)

[0104] Formula (7) is the updated estimated covariance equation. The first prediction error corresponding to the real vehicle and the second prediction error corresponding to the virtual vehicle can be expressed based on the above formula (7).

[0105] S230, determining the first prediction result based on the first optimal prediction estimate and the first prediction error;

[0106] S240: Determine the second prediction result based on the second optimal prediction estimate and the second prediction error.

[0107] For example, the first prediction error can be used to adjust the first optimal prediction estimate, thereby eliminating the impact of the first prediction error on the prediction result. Based on the first optimal prediction estimate and the first prediction error, the impact of the first prediction error on the prediction result of the next vehicle state of the real vehicle is eliminated, resulting in a first prediction result. Based on the second optimal prediction estimate and the second prediction error, the impact of the second prediction error on the prediction result of the next vehicle state of the virtual vehicle is eliminated, resulting in a second prediction result.

[0108] In one embodiment, the Kalman filter model is used to predict the next state information based on the current position of the vehicle (including the real vehicle and the virtual vehicle), and then the subsequent state information is predicted based on the next state information. In this way, the digital twin system is used to keep the virtual vehicle and the real vehicle synchronized in time and space.

[0109] In one embodiment of the present application, the method further comprises the following steps:

[0110] determining a corresponding first posterior probability based on the first prediction error;

[0111] Accordingly, determining the first prediction result based on the first optimal prediction estimate and the first prediction error includes:

[0112] The first prediction result is determined based on the first optimal prediction estimate and the first posterior probability.

[0113] Exemplarily, in the process of determining the first prediction result, the first optimal prediction estimate can also be re-corrected. The specific correction process can be to realize the correction of the first optimal prediction estimate based on the first posterior probability corresponding to the first prediction error. The posterior probability refers to the probability that is re-corrected after obtaining the result information. In this embodiment, the corresponding first posterior probability is determined based on the first prediction error. Combined with the above specific embodiment again. For the prediction of the next vehicle state of the actual vehicle, the Kalman filter model is used to process the above formula (7) as the updated estimated covariance equation, and the first posterior probability is calculated based on the updated estimated covariance equation. The first optimal prediction estimate is adjusted based on the first posterior probability, so as to accurately calculate the first prediction result.

[0114] In one embodiment of the present application, the method further includes:

[0115] determining a corresponding second posterior probability based on the second prediction error;

[0116] Accordingly, determining the second prediction result based on the second optimal prediction estimate and the second prediction error includes:

[0117] The second prediction result is determined based on the second optimal prediction estimate and the second posterior probability.

[0118] Exemplarily, similar to the above steps, the second optimal prediction estimate can also be re-corrected in the process of determining the second prediction result. The specific correction process can be to correct the second optimal prediction estimate based on the second posterior probability corresponding to the second prediction error. The posterior probability refers to the probability that is re-corrected after obtaining the result information. In this embodiment, the corresponding second posterior probability is determined based on the second prediction error. Combined with the above specific embodiment again. For the prediction of the next vehicle state of the virtual vehicle, the Kalman filter model is used to process the above formula (7) as the updated estimated covariance equation, and the second posterior probability is calculated based on the updated estimated covariance equation. The second optimal prediction estimate is adjusted based on the second posterior probability, so as to accurately calculate the second prediction result.

[0119] In one embodiment of the present application, Figure 6As shown, the digital twin system includes a physical space and an information space that are interactively mapped with each other, the physical space includes entities in reality, and the entities have corresponding physical relationships, the information space includes physical models corresponding to the entities, and the physical models are interactively mapped with the entities;

[0120] Accordingly, the method further includes associating driving actions between the real vehicle and the virtual vehicle based on an interactive mapping between the physical model and the entity.

[0121] Exemplarily, a digital twin system includes a physical space and an information space that are interactively mapped to each other. The physical space can receive information about the real-world scene of a real vehicle, including information about real-world entities, such as information about each entity and the physical relationships between them. For example, the entities can be a pedestrian module, a vehicle module, and a road module. The pedestrian module includes pedestrian shape and motion information; the vehicle module includes vehicle geometry and physical information; and the road module includes lane information and environmental information. The information space corresponds to the virtual vehicle and includes multiple physical models, each corresponding to its own entity. The virtual vehicle can send first virtual state information representing the state of the virtual vehicle to the information space. For example, the information space includes a pedestrian motion model, a vehicle physical model, and an environmental model, with the pedestrian model corresponding to the pedestrian module, the vehicle model corresponding to the vehicle module, and the road model corresponding to the road module. Through the interactive mapping between the information space and the physical space, simulation of the real space using the digital virtual space is achieved, including simulation of the real vehicle using the virtual vehicle.

[0122] In one embodiment of the present application, the method further comprises the following steps:

[0123] Based on the simulated operation of the virtual vehicle on the real vehicle, the standardization and robustness of the autonomous driving perception, decision information and / or planning algorithm of the real vehicle are verified.

[0124] For example, the virtual vehicle's simulated operation of a real vehicle can simulate the real vehicle's actual state in real physical space, accurately displaying the real vehicle's autonomous driving system through simulation, including problems within the autonomous driving system. This allows for verification and adjustment of the autonomous driving system based on the simulation results, including validation of the compliance and robustness of the autonomous driving perception, decision-making information, and / or planning algorithms within the autonomous driving system. This reduces simulation costs while improving the autonomous driving of the real vehicle.

[0125] In one embodiment of the present application, the first real state information includes at least one of the following: real-time changes in the speed, position, and orientation of the real vehicle, wherein the position of the real vehicle is the coordinate position of the real vehicle in the road model, and the orientation of the real vehicle is the angle between the vehicle head direction and the geometric coordinate axis of the real vehicle itself;

[0126] Exemplarily, the first real-world state information represents relevant information about the vehicle state of the real vehicle, including real-time changes in the speed, position, and orientation of the real vehicle. Based on the real-time changes in speed, position, and orientation, the vehicle state of the real vehicle at the time of collecting the above-mentioned relevant information can be determined. The position of the real vehicle is the coordinate position of the real vehicle in the road model, and the orientation of the real vehicle is the angle between the vehicle head direction and the real vehicle's own geometric coordinate axis. The road model can be a model representing the road scene in the digital twin system, and the orientation of the real vehicle is the vehicle head direction and the geometric coordinate axis can be constructed based on pre-set orientation information.

[0127] The first virtual state information includes at least one of the following: real-time changes in the speed, position and orientation of the virtual vehicle, wherein the position of the virtual vehicle is the coordinate position of the virtual vehicle in the road model, and the orientation of the virtual vehicle is the angle between the front direction of the vehicle and the geometric coordinate axis of the virtual vehicle itself.

[0128] Similarly, the first virtual state information represents relevant information about the vehicle state of the virtual vehicle, including real-time changes in the speed, position, and orientation of the virtual vehicle. Based on the real-time changes in speed, position, and orientation, the vehicle state of the virtual vehicle at the time of collecting the above-mentioned relevant information can be determined. The position of the virtual vehicle is the coordinate position of the virtual vehicle in the road model, and the orientation of the virtual vehicle is the angle between the head direction and the geometric coordinate axis of the virtual vehicle itself. The road model can be a model that represents the road scene in the digital twin system, and the orientation of the virtual vehicle is the head direction and the geometric coordinate axis can be constructed based on pre-set orientation information.

[0129] Based on the same inventive concept, the embodiment of the present application also provides a virtual-real vehicle simulation system based on digital twins, such as Figure 7 Shown, including:

[0130] An acquisition module is configured to acquire, through a digital twin system, first real state information for characterizing a vehicle state of a real vehicle and first virtual state information for characterizing a vehicle state of a virtual vehicle, wherein a driving action between the real vehicle and the virtual vehicle is associated.

[0131] For example, a digital twin is a virtual entity that creates a physical entity in a digital way. It is a technical means to simulate, verify, predict, and control the entire life cycle of a physical entity with the help of historical data, real-time data, and algorithmic models. Digital twins use information technologies such as perception, computing, and modeling to describe, diagnose, predict, and make decisions about physical space through software definition, thereby realizing interactive mapping between physical space and cyberspace (i.e., digital virtual space). The digital twin system in this embodiment is built based on digital twins. The digital twin system can be used as a simulation platform, in which a virtual vehicle associated with a real vehicle is constructed, and it can process information related to the real vehicle and the virtual vehicle based on its own characteristics, thereby realizing interactive mapping between the two.

[0132] Specifically, the real vehicle can be a physical vehicle in a physical space. In one embodiment, the real vehicle is an intelligent connected car with vehicle-road communication in a physical space, which can be driven by an automatic driving system. The first real state information corresponding to the real vehicle represents relevant information about the vehicle state of the real vehicle. The vehicle state can be the vehicle state when the relevant information is collected, which is the previous vehicle state relative to the next vehicle state. For example, it can be information such as the driving speed and position of the real vehicle. For the collection of the first real state information, the acquisition module can collect it through the real vehicle's own equipment and / or road collection equipment, for example, through the road test unit, where the road test unit includes sensors, communication equipment, etc. The collected first real state information is then sent to the digital twin system through network communication (such as ZMQ communication).

[0133] A virtual vehicle can be a virtual vehicle that is constructed by linking the driving actions of a real vehicle to simulate the real vehicle. For example, by linking the driving actions of a virtual vehicle to a real vehicle, the virtual vehicle can simulate the driving scenarios of the real vehicle. If the real vehicle is involved in an accident or conflicts with other vehicles in physical space, the associated virtual vehicle can also simulate the corresponding actions in the virtual space.

[0134] The first virtual state information of the virtual vehicle can represent relevant information about the vehicle state of the virtual vehicle. The vehicle state can be the vehicle state at the time the relevant information is collected, which is the previous vehicle state relative to the next vehicle state. For example, it can be the driving speed, position information, etc. of the virtual vehicle. The virtual vehicle can be controlled based on a driving simulator (such as the ScanerStudio driving simulator), and the acquisition module can obtain the first virtual state information through the driving simulator.

[0135] A prediction module is configured to predict the next vehicle state of the real vehicle in the digital twin system to generate a first prediction result based on the first real state information and the first virtual state information, and to predict the next vehicle state of the virtual vehicle to generate a second prediction result.

[0136] For example, in the process of sending the first real state information and the first virtual state information to the digital twin system, transmission errors may occur due to various reasons. For example, in the process of the ScanerStudio driving simulator and the road test unit sending the first virtual state information and the first real state information of the virtual vehicle and the real vehicle to the digital twin system through ZMQ communication, communication delay, packet loss, data duplication and other problems may occur. In this embodiment, the prediction module uses the digital twin system and predicts the next vehicle state of the real vehicle and the virtual vehicle based on the first real state information, the first virtual state information and the current driving scene, thereby determining the action behavior of the real vehicle and the virtual vehicle at the next moment.

[0137] In one embodiment, in a digital twin system, a prediction module can predict the next vehicle state of a real vehicle and a virtual vehicle based on a Kalman filter model. This includes predicting the next vehicle state of the real vehicle to generate a first prediction result, and predicting the next vehicle state of the virtual vehicle to generate a second prediction result. For example, the first prediction result represents the vehicle state at the next moment when the digital twin system receives the first real state of the real vehicle, and is a prediction of the state of the real vehicle. This can overcome problems such as communication delay, packet loss, and data duplication that occur when the first real state information is sent to the digital twin system. The second prediction result represents the vehicle state at the next moment when the digital twin system receives the first virtual state of the virtual vehicle, and is a prediction of the state of the virtual vehicle. This can overcome problems such as communication delay, packet loss, and data duplication that occur when the first virtual state information is sent to the digital twin system.

[0138] an adjustment module configured to adjust the first real state information based on the first prediction result to generate corresponding second real state information, and to adjust the first virtual state information based on the second prediction result to generate corresponding second virtual state information;

[0139] For example, the first prediction result represents the vehicle state at the next moment when the digital twin system receives the first real-world state information of the real vehicle. Based on the first prediction result, the adjustment module adjusts the first real-world state information of the real vehicle to the second real-world state information, which can compensate for errors in the process of sending the first real-world state information to the digital twin system. The second real-world state information generated after adjustment truly reflects the current state of the real vehicle, such as the current driving speed and position of the real vehicle. Therefore, the vehicle state of the real vehicle represented by the second real-world state information ensures that the real vehicle maintains consistency in the real scene and in the digital twin system.

[0140] The second prediction result represents the vehicle state at the next moment when the digital twin system receives the virtual vehicle's first virtual state information. Based on the second prediction result, the adjustment module adjusts the virtual vehicle's first virtual state information to the second virtual state information, compensating for errors that occur when the first virtual state information is sent to the digital twin system. The adjusted second virtual state information truly reflects the virtual vehicle's current state, such as its current speed and position. The virtual vehicle's state, represented by the second virtual state information, thus ensures that the virtual vehicle maintains consistency in both the virtual scene and the digital twin system.

[0141] A processing module is configured to send the second real state information to the virtual vehicle, and send the second virtual state information to the real vehicle, so that the real vehicle adjusts its driving behavior based on the second virtual state information, and so that the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle.

[0142] For example, the virtual vehicle can adjust its driving behavior based on the current state of the real vehicle, so that it can interact and map with the real vehicle, and then accurately simulate the driving state of the real vehicle. The processing module can send the second real state information to the virtual vehicle through the digital twin system, and the second real state information truly reflects the current state of the real vehicle. After receiving the second real state information, the virtual vehicle can adjust its driving behavior based on the second real state information to accurately simulate the real vehicle. For example, the second real state information is mapped in the ScanerStudio driving simulator of the virtual vehicle and enables the ScanerStudio driving simulator to perform corresponding behavioral actions in a timely manner.

[0143] In addition, the processing module can also send the second virtual state information to the real vehicle through the digital twin system, and the second virtual state information truly reflects the current state of the virtual vehicle. After receiving the second virtual state information, the real vehicle can adjust the driving behavior of the real vehicle based on the second virtual state information, so that the decision module of the real vehicle can obtain relevant information of the virtual vehicle in a timely manner and make corresponding decision-making behaviors, thereby keeping the real vehicle consistent with the vehicle state of the virtual vehicle.

[0144] In one embodiment of the present application, the acquisition module is further configured to:

[0145] When the real vehicle adjusts its driving behavior based on the second virtual state information and performs an obstacle avoidance action, reacquiring the first real state information;

[0146] When the virtual vehicle adjusts its driving behavior based on the second real state information and performs an obstacle avoidance action, the first virtual state information is reacquired.

[0147] In one embodiment of the present application, the prediction module is further configured to:

[0148] Determining, based on a Kalman filter model, a first optimal predicted estimate of a next vehicle state of the real vehicle and a second optimal predicted estimate of a next vehicle state of the virtual vehicle;

[0149] determining a first prediction error for predicting a next vehicle state of the real vehicle and a second prediction error for predicting a next vehicle state of the virtual vehicle;

[0150] determining the first prediction result based on the first optimal prediction estimate and the first prediction error;

[0151] The second prediction result is determined based on the second optimal prediction estimate and the second prediction error.

[0152] In one embodiment of the present application, the prediction module is further configured to:

[0153] determining a corresponding first posterior probability based on the first prediction error;

[0154] Accordingly, determining the first prediction result based on the first optimal prediction estimate and the first prediction error includes:

[0155] The first prediction result is determined based on the first optimal prediction estimate and the first posterior probability.

[0156] In one embodiment of the present application, the prediction module is further configured to:

[0157] determining a corresponding second posterior probability based on the second prediction error;

[0158] Accordingly, determining the second prediction result based on the second optimal prediction estimate and the second prediction error includes:

[0159] The second prediction result is determined based on the second optimal prediction estimate and the second posterior probability.

[0160] In one embodiment of the present application, the digital twin system includes a physical space and an information space that are interactively mapped to each other, the physical space includes entities in reality, and the entities have corresponding physical relationships, the information space includes physical models corresponding to the entities, and the physical models are interactively mapped to the entities;

[0161] Correspondingly, the virtual-reality vehicle simulation system further includes an association module, which is configured to associate driving actions between the real vehicle and the virtual vehicle based on the interactive mapping between the physical model and the entity.

[0162] In one embodiment of the present application, the processing module is further configured to:

[0163] Based on the simulated operation of the virtual vehicle on the real vehicle, the standardization and robustness of the autonomous driving perception, decision information and / or planning algorithm of the real vehicle are verified.

[0164] In one embodiment of the present application, the first real state information includes at least one of the following: real-time changes in the speed, position, and orientation of the real vehicle, wherein the position of the real vehicle is the coordinate position of the real vehicle in the road model, and the orientation of the real vehicle is the angle between the vehicle head direction and the geometric coordinate axis of the real vehicle itself;

[0165] The first virtual state information includes at least one of the following: real-time changes in the speed, position and orientation of the virtual vehicle, wherein the position of the virtual vehicle is the coordinate position of the virtual vehicle in the road model, and the orientation of the virtual vehicle is the angle between the front direction of the vehicle and the geometric coordinate axis of the virtual vehicle itself.

[0166] The present application also provides an electronic device, such as Figure 8 As shown, it includes a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method as described above.

[0167] An embodiment of the present application further provides a computer-readable storage medium, which carries one or more computer programs. When the one or more computer programs are executed by a processor, the steps of the above method are implemented.

[0168] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the application can adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the application can adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0169] The processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or any combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0170] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0171] The above-mentioned readable storage medium can be a magnetic disk, an optical disk, a DVD, a USB, a read-only memory (ROM) or a random access memory (RAM), etc. This application does not limit the specific form of the storage medium.

[0172] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A virtual-real vehicle simulation method based on digital twins, characterized in that: include: Acquiring, through the digital twin system, first real state information representing a vehicle state of a real vehicle and first virtual state information representing a vehicle state of a virtual vehicle, wherein a driving action between the real vehicle and the virtual vehicle is associated; Based on the first real state information and the first virtual state information, in the digital twin system, predicting a next vehicle state of the real vehicle to generate a first prediction result, and predicting a next vehicle state of the virtual vehicle to generate a second prediction result; Adjusting the first real state information based on the first prediction result to generate corresponding second real state information, and adjusting the first virtual state information based on the second prediction result to generate corresponding second virtual state information; The second real state information is sent to the virtual vehicle, and the second virtual state information is sent to the real vehicle, so that the real vehicle adjusts its driving behavior based on the second virtual state information, and the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle.

2. The method according to claim 1, characterized in that The method further comprises: When the real vehicle adjusts its driving behavior based on the second virtual state information and performs an obstacle avoidance action, reacquiring the first real state information; When the virtual vehicle adjusts its driving behavior based on the second real state information and performs an obstacle avoidance action, the first virtual state information is reacquired.

3. The method according to claim 1, characterized in that The predicting the next vehicle state of the real vehicle to generate a first prediction result, and predicting the next vehicle state of the virtual vehicle to generate a second prediction result, includes: Determining, based on a Kalman filter model, a first optimal predicted estimate of a next vehicle state of the real vehicle and a second optimal predicted estimate of a next vehicle state of the virtual vehicle; determining a first prediction error for predicting a next vehicle state of the real vehicle and a second prediction error for predicting a next vehicle state of the virtual vehicle; determining the first prediction result based on the first optimal prediction estimate and the first prediction error; The second prediction result is determined based on the second optimal prediction estimate and the second prediction error.

4. The method according to claim 3, characterized in that The method further comprises: determining a corresponding first posterior probability based on the first prediction error; Accordingly, determining the first prediction result based on the first optimal prediction estimate and the first prediction error includes: The first prediction result is determined based on the first optimal prediction estimate and the first posterior probability.

5. The method according to claim 3, characterized in that The method further comprises: determining a corresponding second posterior probability based on the second prediction error; Accordingly, determining the second prediction result based on the second optimal prediction estimate and the second prediction error includes: The second prediction result is determined based on the second optimal prediction estimate and the second posterior probability.

6. The method according to claim 1, wherein in, The digital twin system includes a physical space and an information space that are interactively mapped with each other. The physical space includes entities in reality, and the entities have corresponding physical relationships. The information space includes physical models corresponding to the entities, and the physical models are interactively mapped with the entities. Accordingly, the method further includes associating driving actions between the real vehicle and the virtual vehicle based on an interactive mapping between the physical model and the entity.

7. The method according to claim 1, characterized in that The method further comprises: Based on the simulated operation of the virtual vehicle on the real vehicle, the standardization and robustness of the autonomous driving perception, decision information and / or planning algorithm of the real vehicle are verified.

8. The method according to claim 1, characterized in that in, The first real-world state information includes at least one of the following: real-time changes in the speed, position, and orientation of the real vehicle, wherein the position of the real vehicle is the coordinate position of the real vehicle in the road model, and the orientation of the real vehicle is the angle between the vehicle head direction and the geometric coordinate axis of the real vehicle itself; The first virtual state information includes at least one of the following: real-time changes in the speed, position and orientation of the virtual vehicle, wherein the position of the virtual vehicle is the coordinate position of the virtual vehicle in the road model, and the orientation of the virtual vehicle is the angle between the front direction of the vehicle and the geometric coordinate axis of the virtual vehicle itself.

9. A virtual-real vehicle simulation system based on digital twins, characterized in that: include: an acquisition module configured to acquire, through the digital twin system, first real state information representing a vehicle state of a real vehicle and first virtual state information representing a vehicle state of a virtual vehicle, wherein a driving action between the real vehicle and the virtual vehicle is associated; a prediction module configured to, based on the first real state information and the first virtual state information, predict a next vehicle state of the real vehicle in the digital twin system to generate a first prediction result, and predict a next vehicle state of the virtual vehicle to generate a second prediction result; an adjustment module configured to adjust the first real state information based on the first prediction result to generate corresponding second real state information, and to adjust the first virtual state information based on the second prediction result to generate corresponding second virtual state information; A processing module is configured to send the second real state information to the virtual vehicle, and send the second virtual state information to the real vehicle, so that the real vehicle adjusts its driving behavior based on the second virtual state information, and so that the virtual vehicle adjusts its driving behavior based on the second real state information to simulate the real vehicle.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein an executable program is stored in the memory, and the processor executes the executable program to perform the steps of the method according to any one of claims 1 to 8.

Images