A digital twin system and construction method for autonomous driving scenarios

Abstract

The present invention belongs to the field of autonomous driving technology, specifically a digital twin system and construction method for autonomous driving scenarios. The present invention takes digital twin technology as the core, constructs data interaction between the real world and the virtual world, and is used for autonomous driving system status monitoring and historical state fault location; it includes a digital twin definition module and a digital twin scene building module; the former defines the twin of the physical entity in the autonomous driving scene according to the needs of autonomous driving; the latter models different physical entities through three-dimensional reconstruction, builds a twin model library, and builds a virtual twin world; maps the physical entity data collected in the real world to the twin model, uses real data to drive the operation of the autonomous driving system in the digital twin scene, and realizes real-time mapping and synchronization of real space and virtual space; and realizes autonomous driving system status monitoring and historical state fault location.

Description

Technical Field

[0001] The present invention belongs to the field of autonomous driving technology, and specifically relates to a digital twin system and construction method for autonomous driving scenarios. Background Art

[0002] Autonomous driving technology is a cutting-edge technology that relies on computers and artificial intelligence technology to complete complete, safe and efficient driving without human operation.

[0003] When developing and designing existing autonomous driving systems, it is usually necessary to verify the relevant algorithms on an autonomous driving simulation platform before further deploying them to real-world scenarios. However, verification on an autonomous driving simulation platform has the following disadvantages: first, the scene construction on the simulation platform is usually inconsistent with the real world, and cannot fully reflect the state of the real scene, and the simulation platform cannot reflect the actual operating state of the actual vehicle and sensor; second, the verification on the simulation platform is earlier than the real-world verification in the time domain, and cannot be synchronized in real time; finally, the simulation platform has no connection with the real world, and it is impossible to build data interaction between the two. Summary of the invention

[0004] The purpose of the present invention is to provide a digital twin system and method for autonomous driving scenarios. When the autonomous driving system is running, the real operating status of the actual vehicle and sensors can be reflected in the virtual space, so as to realize the status monitoring of the autonomous driving system and the fault location of the historical status.

[0005] The digital twin system for autonomous driving scenarios proposed in the present invention includes a digital twin definition module and a digital twin scenario building module, wherein:

[0006] The digital twin definition module defines twins of physical entities in the autonomous driving scenario according to the requirements of autonomous driving, including the definition of twin categories, twin data, and twin data encapsulation;

[0007] The digital twin scene building module models different physical entities through three-dimensional reconstruction technology and constructs a twin model library, and then builds a virtual twin world based on game engine software; finally, the physical entity data collected in the real world is mapped to the corresponding twin model, and the real data is used to drive the operation of the autonomous driving system in the digital twin scene, realizing real-time mapping and synchronization of real space and virtual space; specifically, it includes twin modeling, weather system modeling, road modeling, and scene modeling.

[0008] Furthermore, in the digital twin definition module:

[0009] The digital twin definition module defines twins of physical entities in the autonomous driving scenario according to the requirements of autonomous driving, including the definition of twin categories, twin data, and twin data encapsulation; wherein:

[0010] The twin categories are used to construct twin categories in autonomous driving scenarios, mainly including vehicle twins, sensor twins, road traffic twins, autonomous driving system twins, etc., and are subdivided by inheritance;

[0011] The twin data defines different data for different types of twins, and each type of twin has specific data, where:

[0012] The vehicle twin includes data such as the vehicle's current speed, steering wheel direction, headlight status, vehicle engine status, and tire pressure;

[0013] The sensor twin includes data such as RGB images, depth images, laser point clouds, temperature and humidity;

[0014] The road traffic twin includes data such as traffic signs, traffic signals and road conditions;

[0015] The autonomous driving system twin includes data such as perception, positioning, path planning and decision-making;

[0016] The twin data encapsulation abstracts and encapsulates the above-mentioned twin data based on object-oriented thinking, including building a unified twin data format, building a twin database for storage and reading, and building a data transmission link for fast transmission.

[0017] Furthermore, in the digital twin scene building module:

[0018] The twin modeling is to use three-dimensional reconstruction technology to digitally model the physical entities in the autonomous driving scene and build a twin model library;

[0019] The weather system modeling is to use the game engine to simulate different weather conditions, including sunny days, cloudy days, rainy days, snowy days, etc., to build a weather system, so as to verify the operation effect of the autonomous driving system under different weather conditions;

[0020] The road modeling is to model the autonomous driving road, including buildings around the road, road structure, road signs, road traffic lights, etc.

[0021] The scenario modeling is to build an autonomous driving scenario by comprehensively using the twin model library and the weather system, and to build a digital twin scene that is restored in proportion based on the real autonomous driving scenario.

[0022] The present invention also proposes a twin construction method for autonomous driving scenarios, which provides a unified data object for real-world object entities involved in autonomous driving scenarios, including data encapsulation, data reading and writing, and data transmission; first, the physical entities in the real world are classified, and after data encapsulation, they become twin forms independent of the original data of the physical entities. At the same time, the twin data can interact with the database in a storage or reading manner, and finally be transmitted to the corresponding twin model in the virtual space, thereby driving the operation of the autonomous driving system in the virtual space.

[0023] The data encapsulation provides a unified data object construction method for real-world entities involved in the autonomous driving scenario. The specific steps are as follows:

[0024] Step 1: Construct data objects of abstract categories, such as vehicle twin objects, sensor twin objects, road traffic twin objects, and autonomous driving system twin objects;

[0025] It should be noted that the abstract class only has the declaration interface of the member function, and needs to use the characteristics of polymorphism to implement specific functional requirements in the derived class, and the object has no specific data members;

[0026] Step 2: Extend the abstract object by inheritance. For example, by inheriting from the sensor object and deriving it as a depth camera sensor object, you can obtain the declaration interface of the sensor object member function and further add the unique properties of the derived object.

[0027] Step 3: Add unique data members to the derived class obtained in step 2, such as adding depth image data to the depth camera sensor object and adding laser point cloud information to the lidar sensor object;

[0028] It should be noted that the format of the data member of the derived class depends on the format of the original data of the corresponding different physical entities. Therefore, the derived class method can be used to conveniently manage and expand the data of different types of physical entities.

[0029] Step 4: The member functions of the derived class inherit from the member function declaration interface defined by its abstract base class, and each implements its unique functional requirements according to different data categories.

[0030] The data transmission described above builds a unified data subscription and publishing interface, and the specific steps are as follows:

[0031] Step 1: Using the polymorphic characteristics of the derived class, use the derived class object to instantiate the base class pointer;

[0032] Step 2: Use the instantiated base class pointer to establish a subscription mechanism, obtain the data of the corresponding physical entity in real time, and construct the data object;

[0033] Step 3: After the data object is constructed, it is published through the instantiated base class pointer, publishing the encapsulated data object, including the data member information of the data object and the timestamp information of the data observation;

[0034] Step 4: The twin model corresponding to the digital twin scene receives the data object, maps the data and state of the object to the virtual digital twin, and visualizes its data members in real time, completing the mapping function from the real world to the virtual world.

[0035] The data reading and writing, establishing the association between the encapsulated data object and the database, storing the data object in the database or reading the data object from the database, specifically includes:

[0036] 1. Database Storage

[0037] Step 1: After the data object is constructed, a multi-level indexing method for the database is constructed based on the category of the data object and the timestamp information of the current moment. First, the abstract category to which the object belongs is used as the key value for the first indexing, and then

[0038] The specific derived category of the object is indexed a second time, and finally the timestamp information is used as the key value for the third index;

[0039] Step 2: Store the data members of the derived class into the database according to the index to complete the data storage;

[0040] (II) Reading the database

[0041] Step 1: According to the twin category and timestamp information of interest, find the data member at the corresponding position through the multi-level indexing method. If the corresponding data can be found through the index information, read the data; if the corresponding data cannot be found through the index information, it is considered that the reading has failed and the database reading process ends;

[0042] Step 2: Repackage the successfully acquired related data into a twin object. At this time, the data transmission method can be called again to complete the data mapping function.

[0043] It should be noted that through the construction of the twin database, the storage function of the database essentially realizes the status monitoring of the real-world autonomous driving system, and the reading function of the database is essentially to read historical data, and then locate the fault of the historical status of the autonomous driving system in the virtual world.

[0044] The digital twin system and construction method for autonomous driving scenarios described in the present invention are different from previous autonomous driving simulation platforms. They are centered on digital twin technology and can achieve real-time mapping and synchronization between the real world and the virtual world. When the autonomous driving system is running, the actual operating status of the actual vehicle and sensors can be reflected in the virtual space. Data interaction between the real world and the virtual world can be constructed for autonomous driving system status monitoring and historical fault location. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a framework diagram of the system of the present invention.

[0046] Figure 2 It is a specific embodiment of the system of the present invention regarding an RGB camera.

[0047] Figure 3 It is a flowchart of the data encapsulation method of the present invention.

[0048] Figure 4 It is a flowchart of the data transmission method of the present invention.

[0049] Figure 5 It is a flowchart of the data reading and writing method of the present invention. DETAILED DESCRIPTION

[0050] The present invention will be further described below in conjunction with the accompanying drawings.

[0051] like Figure 1 As shown in the figure, the digital twin system for autonomous driving scenarios starts from the physical space entity in the autonomous driving scenario, defines the digital twin at the data level and performs three-dimensional modeling of the twin at the model level, and finally completes the real-time mapping and synchronization of the real space and the virtual space.

[0052] Specifically, starting from the physical entity, the entity is defined as a twin at the data level, which is divided into two processes: category definition and data definition. In terms of category definition, it is divided into several categories according to the different physical entities, such as vehicle twins, sensor twins, road traffic twins, and autonomous driving system twins, and the categories are subdivided through class inheritance. After distinguishing the categories, the corresponding data definitions are made for different categories of physical entities. The vehicle twin includes data such as the current speed of the vehicle, steering wheel steering, headlight status, car engine status, and tire pressure. The sensor twin includes data such as RGB images, depth images, laser point clouds, temperature and humidity. The road traffic twin includes data such as traffic signs, traffic signals, and road conditions. The autonomous driving system twin includes data such as perception, positioning, path planning, and decision-making. After completing the definition of the digital twin, the twin data is encapsulated, and the above-mentioned twin data is abstracted and encapsulated based on object-oriented thinking, and the read and write interface and data transmission interface of the twin database are defined. The database read and write interface can be used to store the twin data at the current moment into the database or read the twin data at a historical moment from the database. The data transmission interface completes the data mapping of the twin, so that the data generated by physical entities in the real world can be mapped to the digital twin scene in real time.

[0053] Then, at the model level, starting from the physical object, the 3D model is obtained offline using 3D reconstruction technology, and then the model resource library of the twin is constructed by 3D modeling of the different physical entities in the autonomous driving scene. At the same time, the game engine is used to simulate different weather conditions, including sunny, cloudy, rainy, snowy and other weather conditions, so as to build a weather system, which can be used to verify the operation effect of the autonomous driving system in different weather conditions; and the autonomous driving road is modeled, including the buildings around the road, road structure, road signs, road traffic lights, etc., to complete the standard traffic scene construction. By combining the above-mentioned model resource library, weather system and road modeling, the digital twin scene construction in the entire virtual space is finally completed in the game engine software.

[0054] Finally, through the above two process branches, the twin data of each physical entity at the data level and the corresponding model at the model level are obtained. The corresponding data mapping is completed using the data transmission interface defined by the twin package, and the relevant data in the real space is mapped to the corresponding twin model in the virtual space. In this way, the real data is used to drive the operation of the autonomous driving system in the digital twin scenario, realizing real-time mapping and synchronization of the real space and the virtual space.

[0055] like Figure 2As shown in the figure, it is a specific embodiment of the digital twin system for the autonomous driving scene, and an RGB camera is used as an example for illustration. First, the RGB camera three-dimensional model is loaded from the constructed twin model library and placed in the digital twin scene. Then the twin is constructed at the data level, and its category is defined as the sensor category, and the data is defined as RGB image data. Next, the data encapsulation of the twin is performed. From the category definition, it can be seen that the RGB camera twin needs to inherit from the camera base class Class Camera, and the derived class name is Class RGB_Camera; from the data definition, it can be seen that the main data member of the twin is the image. In this example, the image data is stored in a two-dimensional byte array, specifically char Image[][]; at the same time, the timestamp information is obtained according to the clock trigger signal of the data actually collected by the sensor, which is represented as timeStamp in the data member. According to the storage flag isSaved, it is judged whether the data needs to be stored. If it needs to be stored, the save() function is called through the base class pointer instantiated by the derived class to write the database. If it is not needed, it does not need to be called. Finally, the publish() function is called to transmit data, and the subscribe() script function is executed in the digital twin scene to receive data, and the image data is displayed in the digital twin scene to complete the data mapping.

[0056] like Figure 3-5 As shown, it is a flow chart of the data encapsulation method, data transmission method and data reading and writing method proposed by the present invention.

[0057] The data encapsulation method provides a unified data object construction method for real-world object entities involved in the autonomous driving scenario, and includes the following steps:

[0058] Step 1: Construct data objects of abstract categories, such as vehicle twin objects, sensor twin objects, road traffic twin objects, and autonomous driving system twin objects;

[0059] It should be noted that the abstract class only has the declaration interface of the member function, and needs to use the characteristics of polymorphism to implement specific functional requirements in the derived class, and the object has no specific data members;

[0060] Step 2: Extend the abstract object by inheritance. For example, by inheriting from the sensor object and deriving it as a depth camera sensor object, you can obtain the declaration interface of the sensor object member function and further add the unique properties of the derived object.

[0061] Step 3: Add unique data members to the derived class obtained in step 2, such as adding depth image data to the depth camera sensor object and adding laser point cloud information to the lidar sensor object;

[0062] It should be noted that the format of the data member of the derived class depends on the format of the original data of the corresponding different physical entities. Therefore, the derived class method can be used to conveniently manage and expand the data of different types of physical entities.

[0063] Step 4: The member functions of the derived class inherit from the member function declaration interface defined by its abstract base class, and each implements its unique functional requirements according to different data categories.

[0064] The data transmission method described above constructs a unified data subscription and publishing interface, and includes the following steps:

[0065] Step 1: Using the polymorphic characteristics of the derived class, use the derived class object to instantiate the base class pointer;

[0066] Step 2: Use the instantiated base class pointer to establish a subscription mechanism, obtain the data of the corresponding physical entity in real time, and construct the data object;

[0067] Step 3: After the data object is constructed, it is published through the instantiated base class pointer, publishing the encapsulated data object, including the data member information of the data object and the timestamp information of the data observation;

[0068] Step 4: The twin model corresponding to the digital twin scene receives the data object, maps the data and state of the object to the virtual digital twin, and visualizes its data members in real time, completing the mapping function from the real world to the virtual world.

[0069] The data reading and writing method establishes an association between the encapsulated data object and the database, stores the data object in the database or reads the data object from the database, and includes the following steps:

[0070] Database storage:

[0071] Step 1: After the data object is constructed, a multi-level indexing method for the database is constructed based on the category of the data object and the timestamp information of the current moment. First, the abstract category to which the object belongs is used as the key value for the first indexing, and then

[0072] The specific derived category of the object is indexed a second time, and finally the timestamp information is used as the key value for the third index;

[0073] Step 2: Store the data members of the derived class into the database according to the index to complete the data storage;

[0074] Reading the database:

[0075] Step 1: According to the twin category and timestamp information of interest, find the data member at the corresponding position through the multi-level indexing method. If the corresponding data can be found through the index information, read the data; if the corresponding data cannot be found through the index information, it is considered that the reading has failed and the database reading process ends;

[0076] Step 2: Repackage the successfully acquired related data into a twin object. At this time, the data transmission method can be called again to complete the data mapping function.

[0077] It should be noted that through the construction of the twin database, the storage function of the database essentially realizes the status monitoring of the real-world autonomous driving system, and the reading function of the database is essentially to read historical data, and then locate the fault of the historical status of the autonomous driving system in the virtual world.

[0078] To sum up, the digital twin system and method for autonomous driving scenarios described in the present invention are different from previous autonomous driving simulation platforms. They are based on digital twin technology, can fully reflect the status of real scenes, and solve the problem that the simulation platform cannot reflect the actual operating status of actual vehicles and sensors. Finally, real-time mapping and synchronization of the real world and the virtual world are realized. In addition, it can be used for autonomous driving system status monitoring and historical fault location.

Claims

1. A digital twin system for autonomous driving scenarios, characterized in that: With digital twin technology as the core, it realizes real-time mapping and synchronization between the real world and the virtual world; when the autonomous driving system is running, the real operating status of the actual vehicle and sensors is reflected in the virtual space; data interaction between the real world and the virtual world is constructed for autonomous driving system status monitoring and historical fault location; it includes digital twin definition module and digital twin scenario construction module, among which: The digital twin definition module defines twins of physical entities in the autonomous driving scenario according to the requirements of autonomous driving, including the definition of twin categories, twin data, and twin data encapsulation; The digital twin scene building module models different physical entities through 3D reconstruction technology, builds a twin model library, and then builds a virtual twin world based on game engine software; finally, the physical entity data collected in the real world is mapped to the corresponding twin model, and the operation of the autonomous driving system in the digital twin scene is driven by real data, realizing real-time mapping and synchronization of real space and virtual space; specifically, it includes twin modeling, weather system modeling, road modeling, and scene modeling; In the digital twin definition module: The twin categories are used to construct twin categories in autonomous driving scenarios, including vehicle twins, sensor twins, road traffic twins, and autonomous driving system twins, and are subdivided by inheritance; The twin data defines different data for different types of twins, and each type of twin has specific data, where: The vehicle twin includes the vehicle's current speed, steering wheel direction, headlight status, vehicle engine status, and tire pressure data; The sensor twins include RGB images, depth images, laser point clouds, temperature and humidity data; The road traffic twin includes traffic signs, traffic signals and road condition data; The autonomous driving system twin includes perception, positioning, path planning and decision-making data; The twin data encapsulation abstracts and encapsulates the twin data involved above based on object-oriented thinking, including constructing a unified twin data format, constructing a twin database for storage and reading, and constructing a data transmission link for fast transmission; Provide unified data objects for real-world entities involved in autonomous driving scenarios, including data encapsulation, data reading and writing, and data transmission; first classify the physical entities in the real world, and after data encapsulation, they become twins independent of the original data of the physical entities. At the same time, the twin data interacts with the database in a storage or reading manner, and is finally transmitted to the corresponding twin model in the virtual space, thereby driving the operation of the autonomous driving system in the virtual space; The data encapsulation provides a unified data object construction method for real-world entities involved in the autonomous driving scenario. The specific steps are as follows: Step 1: Construct data objects of abstract categories, including vehicle twin objects, sensor twin objects, road traffic twin objects, and autonomous driving system twin objects; Step 2: Extend the abstract object by inheritance, including inheriting from the sensor object and deriving it as a depth camera sensor object, so as to obtain the declaration interface of the sensor object member function and further add the unique properties of the derived object; Step 3: Add unique data members to the derived class obtained in step 2, including adding depth image data to the depth camera sensor object and adding laser point cloud information to the lidar sensor object; Step 4: The member functions of the derived class inherit from the member function declaration interface defined by its abstract base class, and each implements its unique functional requirements according to different data categories.

2. The digital twin system for autonomous driving scenarios according to claim 1, characterized in that: In the digital twin scene building module: The twin modeling is to use three-dimensional reconstruction technology to digitally model the physical entities in the autonomous driving scene and build a twin model library; The weather system modeling is to use the game engine to simulate different weather conditions, including sunny, cloudy, rainy and snowy weather, to build a weather system, so as to verify the operation effect of the autonomous driving system under different weather conditions; The road modeling is to model the autonomous driving road, including buildings around the road, road structure, road signs, and road traffic lights; The scenario modeling is to build an autonomous driving scenario by comprehensively using the twin model library and the weather system, and to build a proportionally restored digital twin scene based on the real autonomous driving scenario.

3. The digital twin system for autonomous driving scenarios according to claim 1, characterized in that: The data transmission is to build a unified data subscription and publishing interface, the specific steps are: Step 1: Using the polymorphic characteristics of the derived class, use the derived class object to instantiate the base class pointer; Step 2: Use the instantiated base class pointer to establish a subscription mechanism, obtain the data of the corresponding physical entity in real time, and construct the data object; Step 3: After the data object is constructed, it is published through the instantiated base class pointer, publishing the encapsulated data object, including the data member information of the data object and the timestamp information of the data observation; Step 4: The twin model corresponding to the digital twin scene receives the data object, maps the data and state of the object to the virtual digital twin, and visualizes its data members in real time, completing the mapping function from the real world to the virtual world.

4. The digital twin system for autonomous driving scenarios according to claim 1, characterized in that: The data reading and writing is to establish the association between the encapsulated data object and the database, store the data object in the database or read the data object from the database, which specifically includes:

1. Database storage: Step 1: After the data object is constructed, a multi-level index of the database is constructed according to the category of the data object and the timestamp information of the current moment, wherein the first index is performed using the abstract category to which the object belongs as the key value, the second index is performed according to the specific derived category of the object, and finally the third index is performed using the timestamp information as the key value; Step 2: Store the data members of the derived class into the database according to the index to complete the data storage; (II) Reading the database: Step 1: According to the twin category and timestamp information of interest, find the data member at the corresponding position through multi-level indexing. If the corresponding data can be found through the index information, read the data; if the corresponding data cannot be found through the index information, it is considered that the reading has failed and the database reading process ends; Step 2: Repackage the successfully acquired related data into a twin object, and call the data transmission method again to complete the data mapping function.

Images