Virtual-reality testing methods, equipment and media for intelligent connected vehicles based on digital twins

Abstract

The present application discloses a virtual-reality test method, equipment and medium for intelligent networked vehicles based on digital twins, which is used to solve the technical problems that the existing simulation process is too idealized and lacks fidelity. The method includes: obtaining corresponding image data and point cloud data; splicing the image data at the junction of open road type and tunnel road type according to satellite positioning information to obtain a digital surface model, and preprocessing the point cloud data to obtain a digital elevation model; fusing the visible light overall image and the three-dimensional point cloud terrain map to obtain the corresponding three-dimensional terrain model; obtaining the dynamic perception information of the traffic participating elements of the highway and integrating the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements to obtain the highway digital twin system; placing the constructed vehicle simulation system into the highway digital twin system for operation, and placing the automatic driving system into the vehicle simulation system for operation, so as to realize the virtual-reality test of the intelligent networked vehicles on the highway.

Description

Technical Field

[0001] The present application relates to the field of vehicle testing technology, and in particular to a virtual-reality testing method, equipment and medium for intelligent connected vehicles based on digital twins. Background Art

[0002] At present, simulation testing provides a method to reproduce the characteristics of real vehicle systems in a virtual environment, constructs internal and external factors and conditions that interact with the system, and is simple, low-cost and easy to replicate. However, the test scenarios are mainly based on simulated or preset states, and do not take into account the behavioral differences of traffic participants due to different time and space, and do not fully consider the irrational behavior of scene participants. The simulation process is idealized and rationalized, and the fidelity is insufficient, making it difficult to simulate 100% of the real complex traffic scenarios. Summary of the invention

[0003] The embodiments of the present application provide a virtual-reality testing method, device and medium for intelligent connected vehicles based on digital twins, so as to solve the technical problems that the simulated traffic scenarios in the prior art do not take into account the behavioral differences of traffic participants due to different time and space, do not fully consider the irrational behavior of scene participants, and the simulation process is idealized and rationalized, and lacks fidelity.

[0004] On the one hand, the embodiment of the present application provides a virtual-real testing method for an intelligent connected vehicle based on digital twins, including:

[0005] Determine the terrain type corresponding to the expressway, and obtain image data and point cloud data of the terrain type corresponding to the expressway through a collection device corresponding to the terrain type; the terrain type includes an open road type and a tunnel road type;

[0006] splicing the image data of the junction of the open road type and the tunnel road type according to the satellite positioning information to obtain a corresponding digital surface model, and preprocessing the point cloud data to obtain a corresponding digital elevation model;

[0007] Fusing the visible light overall image in the digital surface model with the three-dimensional point cloud topographic map in the digital elevation model to obtain a three-dimensional terrain model corresponding to the highway;

[0008] The real-time monitoring device for the expressway is used to obtain dynamic perception information of traffic participating elements of the expressway, and the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are integrated to obtain a corresponding digital twin system of the expressway;

[0009] Construct a vehicle simulation system to place the vehicle simulation system into the highway digital twin system for operation, and place the autonomous driving system into the vehicle simulation system in the highway digital twin system for operation, so as to realize virtual-reality testing of intelligent connected vehicles on the highway.

[0010] In one implementation of the present application, before the image data of the junction of the open road type and the tunnel road type are spliced ​​according to the satellite positioning information to obtain the corresponding digital surface model, the method further includes:

[0011] Sharpening the image data by using a preset function, and reducing noise on the sharpened image data by using a median filter, thereby preprocessing the image data;

[0012] Acquire the time corresponding to the image data from the image data, and sort the preprocessed multiple image data in order from early to late;

[0013] According to the image registration technology and the fusion reconstruction technology, and in accordance with the corresponding order of the plurality of image data, the plurality of image data are sequentially spliced;

[0014] The spliced ​​image data are fused through a convolutional neural network model to obtain an overall image corresponding to the highway.

[0015] In one implementation of the present application, the image data of the junction of the open road type and the tunnel road type are spliced ​​according to the satellite positioning information to obtain the corresponding digital surface model, which specifically includes:

[0016] Acquire satellite positioning information corresponding to the image data from the image data, and respectively determine a Laplacian pyramid of the image to be stitched corresponding to the image data and a masked Gaussian pyramid of the same size;

[0017] According to the masked Gaussian pyramid of the current size, images corresponding to the Laplacian pyramids of two image data with resolutions from low to high are spliced, and a corresponding digital surface model is obtained; the digital surface model includes a visible light overall image with the satellite positioning information.

[0018] In one implementation of the present application, the preprocessing of the point cloud data to obtain a corresponding digital elevation model specifically includes:

[0019] Sorting the acquired plurality of point cloud data in ascending order according to the time sequence from earliest to latest, and filtering the point cloud data by a median filter to remove noise in the point cloud data;

[0020] Classifying the denoised point cloud data by a pre-trained convolutional neural network, and performing affine transformation on the images corresponding to two adjacent point cloud data at the feature points of the highway according to the classification;

[0021] The feature points in the images corresponding to the two adjacent point cloud data are aligned to realize the splicing of the point cloud data and obtain the corresponding digital elevation model; the digital elevation model includes a three-dimensional point cloud topographic map corresponding to the highway.

[0022] In one implementation of the present application, the fusion of the visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model to obtain the three-dimensional terrain model corresponding to the highway specifically includes:

[0023] Performing feature detection on the visible light overall image in the digital surface model, and generating a region of interest corresponding to the visible light overall image;

[0024] The satellite positioning information corresponding to the region of interest is calculated by a key point calibration algorithm and a geometric algorithm, and the region of interest is projected into a three-dimensional point cloud space according to the satellite positioning information to generate a corresponding three-dimensional region of interest;

[0025] According to the three-dimensional region of interest, the visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model are fused to obtain a three-dimensional terrain model corresponding to the expressway.

[0026] In one implementation of the present application, obtaining the dynamic perception information of traffic participating elements of the expressway through the expressway real-time monitoring device specifically includes:

[0027] A road monitoring device in the real-time monitoring device for the expressway is constructed by means of monitoring equipment, transmission equipment and power supply equipment;

[0028] The tunnel monitoring device in the real-time monitoring device of the highway is constructed by ventilation, lighting, traffic control, monitoring, emergency telephone broadcasting, fire alarm and radar event detection devices as well as sensor equipment, transmission equipment and power supply equipment;

[0029] And, constructing the traffic operation monitoring device in the real-time monitoring device of the expressway through traffic flow monitoring, traffic event monitoring and traffic weather monitoring equipment;

[0030] Obtaining dynamic perception information of traffic participating elements corresponding to the expressway through the front-end sensors in the road monitoring device, the tunnel monitoring device and the traffic operation monitoring device;

[0031] The encapsulated dynamic perception information of the traffic participating elements is transmitted to the three-dimensional terrain model through a unified data transmission protocol.

[0032] In one implementation of the present application, the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are integrated to obtain the corresponding highway digital twin system, which specifically includes:

[0033] According to the acquired dynamic perception information of the traffic participating elements of the expressway, a key information data set of the expressway is constructed, wherein the key information represents information data corresponding to each key point in the expressway, and the key points include at least: a toll station, a service area, an overpass, and a construction point;

[0034] By software programming, a geometric model, a functional model, a performance model and an evolution model related to key information of the highway are constructed in a digital space;

[0035] Inputting the data corresponding to each of the key points in the key information data set into the geometric model, the functional model, the performance model and the evolution model to realize page browsing and human-computer interaction interface;

[0036] The static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are input into the geometric model, the functional model, the performance model and the evolution model to realize the integration of the static data and the dynamic perception information of the traffic participating elements and obtain the corresponding highway digital twin system.

[0037] In one implementation of the present application, the geometric model includes at least: size information and shape information, the functional model includes at least: function information and list information, the performance model includes normal data information and limit data information, and the evolution model includes at least: rule information and association relationship information.

[0038] On the other hand, the embodiment of the present application also provides a virtual-reality test device for intelligent connected vehicles based on digital twins, the device comprising:

[0039] at least one processor;

[0040] and, a memory communicatively coupled to the at least one processor;

[0041] Among them, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the virtual-reality testing method for intelligent connected vehicles based on digital twins as mentioned above.

[0042] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0043] Such as the virtual-reality testing method of intelligent connected vehicles based on digital twins mentioned above.

[0044] The embodiments of the present application provide a virtual-reality test method, device, and medium for intelligent connected vehicles based on digital twins, which at least have the following beneficial effects:

[0045] The highway is divided into two terrain types: open road type and tunnel road type. In this way, the corresponding acquisition equipment can be used for the two different terrain types to obtain the image data and point cloud data of the corresponding terrain type of the highway in real time, so as to collect the real highway operation status in real time, which is convenient for the subsequent processing of the image data and point cloud data.

[0046] According to the satellite positioning information corresponding to the expressway, the image data of the junction of the open road type and the tunnel road type are spliced ​​to obtain the digital surface model corresponding to the expressway, that is, to construct the visible light overall image corresponding to the expressway, and by preprocessing the point cloud data, the digital elevation model corresponding to the expressway can be obtained, that is, to construct the three-dimensional point cloud topographic map corresponding to the expressway; and then by fusing the visible light image and the three-dimensional point cloud topographic map, the three-dimensional terrain model corresponding to the expressway can be obtained;

[0047] Then, through the real-time monitoring devices installed on various sections of the highway, the dynamic perception information of the traffic participating elements of the highway can be obtained. By integrating the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements, the corresponding highway digital twin system can be obtained; by placing the constructed vehicle simulation system into the highway digital twin system for operation, the vehicle can be simulated in the digital twin system, and then the autonomous driving system is placed in the vehicle simulation system, that is, the autonomous driving system is run in the simulated vehicle, so that the virtual and real test of the intelligent connected vehicle on the highway can be realized. In this way, a real digital space highway scene is constructed through the digital twin technology, which can meet the needs of the real test environment of the autonomous driving vehicle, and participate in the real environment test of different roads according to the needs of different customers, which can improve the fidelity of the test and save special time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1A schematic diagram of a flow chart of a virtual-reality testing method for an intelligent connected vehicle based on digital twins provided in an embodiment of the present application;

[0050] Figure 2 A schematic flow chart of another method for virtual-reality testing of an intelligent connected vehicle based on digital twins provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the internal structure of a virtual-reality testing device for an intelligent connected vehicle based on digital twins provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0053] Through searching, it was found that an existing patent for a dynamic closed test system for intelligent connected vehicles includes a straight-line acceleration area, a left-turn test area, and a right-turn special-shaped dynamic test area, etc. The test vehicle takes the middle straight-line acceleration lane as the core, and continuously performs multiple left and right steering and braking tests without idle test time in the middle, which improves the test efficiency; the return lane is eliminated, and the test vehicle does not need to return to the acceleration starting point after completing a single straight-line acceleration. The end of each single test is the beginning of the next test, which saves test time and saves the energy consumed on the original return road, improving the test economy; the intensive use of land resources is maximized, and there is no functional interference with the surrounding test areas / roads, thereby improving the overall land utilization rate; the number of tests completed per unit time is increased, and the test vehicle is saved from returning to the field, reducing exhaust emissions, which is beneficial to the improvement of the test field environment.

[0054] The existing patented autonomous vehicle testing system and method includes: a test site, a simulation analysis platform, and an autonomous vehicle to be tested. The system uses the simulation analysis platform to build a visualized virtual model of the environment and a virtual model of the vehicle, and compares the real information with the virtual information. When the real information and the virtual information are consistent, the simulation analysis platform is also used to output control instructions to its hardware simulation bench according to the virtual information to perform power simulation. The simulation analysis platform compares the simulation execution results with the real vehicle execution decision information, and outputs the test analysis data of the autonomous vehicle to be tested. By combining virtual simulation with real site testing, a comprehensive verification of the perception, decision-making and execution capabilities of the autonomous vehicle is achieved.

[0055] The existing patented intelligent connected vehicle simulation test system and method calculates the vehicle dynamics model and outputs the vehicle motion state; performs motion changes on the environmental model according to the vehicle motion state, and outputs a simulation picture and simulated target information; obtains the simulation picture and the vehicle motion state, simulates the motion performance of the actual vehicle according to the vehicle motion state, performs driving operations, and controls the movement of the vehicle dynamics model according to the driving operations; calculates target recognition information according to the simulation picture and the simulated target information; calculates the vehicle control information according to the target recognition information, and controls the movement of the vehicle dynamics model according to the vehicle control information.

[0056] The embodiments of the present application provide a virtual-reality testing method, equipment and medium for intelligent connected vehicles based on digital twins, which divide highways into two terrain types: open road type and tunnel road type. In this way, for the two different terrain types, the corresponding acquisition equipment can be used to obtain image data and point cloud data of the terrain type corresponding to the highway in real time, so as to collect the actual operation status of the highway in real time, which is convenient for the subsequent processing of image data and point cloud data. According to the satellite positioning information corresponding to the highway, the image data of the junction of the open road type and the tunnel road type are spliced ​​to obtain the digital surface model corresponding to the highway, that is, to construct the overall visible light image corresponding to the highway, and by preprocessing the point cloud data, the digital surface model corresponding to the highway can be obtained. The digital elevation model is constructed, that is, a three-dimensional point cloud terrain map corresponding to the highway is constructed; then, by fusing the visible light image and the three-dimensional point cloud terrain map, the three-dimensional terrain model corresponding to the highway can be obtained; then, through the real-time monitoring devices set up on each section of the highway, the dynamic perception information of the traffic participating elements of the highway can be obtained, and by integrating the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements, the corresponding highway digital twin system can be obtained; by placing the constructed vehicle simulation system into the highway digital twin system for operation, the vehicle can be simulated in the digital twin system, and then the automatic driving system is placed in the vehicle simulation system, that is, the automatic driving system is run in the simulated vehicle, so that the virtual and real test of the intelligent networked vehicle on the highway can be realized. The technical problems that the simulated traffic scene in the existing technology does not take into account the behavioral differences of traffic participants due to different time and space, and does not fully consider the irrational behavior of the scene participants, the simulation process is idealized and rational, and the fidelity is lacking.

[0057] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0058] Figure 1 The flowchart of the virtual-real testing method of the intelligent networked vehicle based on digital twin provided in the embodiment of the present application is as follows. Figure 1As shown, the virtual-reality testing method for intelligent connected vehicles based on digital twins provided in the embodiment of the present application may mainly include the following steps:

[0059] Step 101: determine the terrain type corresponding to the expressway, and obtain image data and point cloud data of the terrain type corresponding to the expressway through a collection device corresponding to the terrain type.

[0060] This application determines the terrain type corresponding to the highway by dividing the terrain type of the highway, and for each terrain type, selects the acquisition equipment corresponding to each terrain type to obtain the image data and point cloud data corresponding to the highway. It should be noted that the terrain types in the embodiments of this application include two categories: open road type and tunnel road type. For highways with open road types, drones carrying cameras and laser radars are used to obtain image data and point cloud data of open roads in highways; for highways with tunnel road types, unmanned vehicles equipped with linear array cameras, fill light equipment and laser radar scanning equipment are used to obtain image data and point cloud data of closed roads such as tunnels in highways.

[0061] It should be noted that the image data in the embodiment of the present application includes the creation time corresponding to the image and the satellite positioning information corresponding to the image, which facilitates the subsequent processing of the image data according to the creation time of the image and the satellite positioning information.

[0062] Step 102: splice the image data of the junction of the open road type and the tunnel road type according to the satellite positioning information to obtain the corresponding digital surface model, and pre-process the point cloud data to obtain the corresponding digital elevation model.

[0063] The server determines the adjacent open roads corresponding to the open road type and the tunnels corresponding to the tunnel road type according to the satellite positioning information corresponding to the image in the image data, and then splices the junction of the open road and the tunnel according to the satellite positioning information to obtain the digital surface model corresponding to the expressway. In addition, the server successively uses point cloud filtering, point cloud feature points, point cloud calibration, segmentation and classification, target recognition and retrieval, 3D reconstruction and other methods to pre-process the point cloud data, so as to obtain the digital elevation model corresponding to the expressway.

[0064] Specifically, the server obtains the satellite positioning information corresponding to the image data from the image data, calculates the Laplacian pyramid of the image to be spliced ​​and the masked Gaussian pyramid of the same size corresponding to the image data, and splices the two image data from the low resolution to the high resolution Laplacian pyramid corresponding to the image at each resolution according to the masked Gaussian pyramid of the current scale, and finally obtains the digital surface model corresponding to the high-altitude highway. It should be noted that the digital surface model in the embodiment of the present application includes the visible light overall image with satellite positioning information.

[0065] The server will obtain multiple point cloud data in ascending order from early to late, and filter the point cloud data through a median filter to remove obvious noise points in the point cloud data, and then classify the denoised point cloud data through a pre-trained convolutional neural network. The convolutional neural network is trained on highway pavement, isolation belts, guardrails, vegetation such as trees, toll booths, etc. Secondly, at the feature points of the highway, such as: pavement, isolation belts, guardrails, vegetation (such as trees), toll booths and other landmarks, the server performs affine transformations such as translation, rotation, scaling, flipping and shearing on the images corresponding to the two adjacent point cloud data according to the classification corresponding to the point cloud data, so as to align the feature points in the two images, realize the splicing of the point cloud data, and obtain the digital elevation model corresponding to the highway. It should be noted that the digital elevation model in the embodiment of the present application includes a three-dimensional point cloud topographic map corresponding to the highway.

[0066] In one embodiment of the present application, the server splices the image data of the junction of the open road type and the tunnel road type according to the satellite positioning information to obtain the corresponding digital surface model, and then sharpens the image data through a preset function to increase the contrast of the image edge, and then reduces the noise of the sharpened image data through a median filter to implement the preprocessing operation of the image data. It should be noted that the preset function in the embodiment of the present application uses the filter2D() function in OpenCV to sharpen the image data.

[0067] The server obtains the creation time corresponding to each image from each image data, and then sorts the image set in ascending order according to the image creation time, with the earliest time at the front, and then sorts it downwards, with the time being specific to hours, minutes and seconds. Then, the image containing the toll gate is selected from several images as the interval point, and the guardrails or signs on both sides of the highway are selected as feature points. Starting from the first segmentation point image, the features of multiple images between each two interval point images are extracted for matching. By calculating the transformation structure between the images, image mapping is achieved and feature points are aligned. According to the image registration technology and fusion reconstruction technology, multiple images are stitched in sequence according to the corresponding order of multiple image data. The server fuses the spliced ​​image data through the convolutional neural network model to obtain the overall image corresponding to the open road and tunnel in the highway scanning area.

[0068] Step 103: fuse the visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model to obtain a three-dimensional terrain model corresponding to the expressway.

[0069] Beidou satellite navigation system ground reference stations are established in characteristic areas such as bridge and tunnel stations or toll plazas, and accurate satellite positioning data of gantries and tunnel entrances and exits along the route are obtained through high-precision Beidou positioning devices. The server uses data superposition technology to fuse the visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model, achieving data superposition of the visible light overall image and the three-dimensional point cloud topographic map, thereby obtaining the corresponding three-dimensional terrain model of the highway.

[0070] Specifically, the server first performs feature detection on the overall visible light image in the digital surface model and generates a region of interest corresponding to the overall visible light image. The region of interest is generated mainly because the visible light image includes all image information within the field of view, but in actual use, only scene information related to the highway is required, such as: road surface, isolation belt, guardrail, toll station and other data, while information on buildings, lakes, ponds on both sides of the highway that is not related to the highway scene will not be used.

[0071] The server calculates the satellite positioning information corresponding to the region of interest in the visible light overall image through key point calibration algorithms and geometric algorithms, and projects the region of interest into the three-dimensional point cloud space based on the satellite positioning information corresponding to the region of interest, thereby generating the corresponding three-dimensional region of interest. Then, based on the generated three-dimensional region of interest, the server fuses the texture information of the visible light overall image in the digital surface model into the high-reflection characteristics of the three-dimensional point cloud topographic map in the digital elevation model, and realizes the fusion of two-dimensional image information and three-dimensional point cloud data through convolutional neural networks, thereby constructing a high-precision three-dimensional terrain model of a static entity of a highway.

[0072] Step 104: obtain dynamic perception information of traffic participating elements of the highway through the real-time monitoring device of the highway, and integrate the static data corresponding to the three-dimensional terrain model and the dynamic perception information of traffic participating elements to obtain the corresponding highway digital twin system.

[0073] Specifically, the present application constructs a road monitoring device in a real-time monitoring device for a highway through monitoring equipment, transmission equipment and power supply equipment, constructs a tunnel monitoring device in a real-time monitoring device for a highway through ventilation, lighting, traffic control, monitoring, emergency telephone broadcasting, fire alarm and radar event detection devices as well as sensing equipment, transmission equipment and power supply equipment, and constructs a traffic operation monitoring device in a real-time monitoring device for a highway through monitoring equipment such as traffic flow monitoring, traffic event monitoring and traffic weather monitoring. Then, the server obtains the dynamic perception information of the traffic participating elements corresponding to the highway in real time through the front-end sensors of the road monitoring device, tunnel monitoring device and traffic operation monitoring device in the real-time monitoring device for the highway.

[0074] It should be noted that the front-end sensor for obtaining dynamic perception information of traffic participating elements in the embodiment of the present application has an Ethernet interface and adopts a unified data transmission protocol. After the dynamic perception information of traffic participating elements is encapsulated, it is transmitted to the three-dimensional terrain model via a dedicated optical fiber, thereby realizing the real-time perception function of highway road surface information.

[0075] The server constructs a key information data set of the highway based on the acquired dynamic perception information of the traffic participating elements of the highway. It should be noted that the key information in the embodiment of the present application represents the information data corresponding to each key point in the highway, such as toll stations, tunnels, service areas, overpasses, merging areas, diversion areas, construction sites, road sections, routes, road networks, etc.

[0076] Through software programming, the server constructs geometric models, functional models, performance models and evolution models related to the key information of the highway in the digital space, and then inputs the data corresponding to each key point in the key information data set into the geometric model, functional model, performance model and evolution model, thereby realizing the functions of page browsing and human-computer interaction interface, and can display data, switch pages, and perform information query, etc., so that it can operate normally.

[0077] Finally, the server inputs the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements into the geometric model, functional model, performance model and evolution model, realizes the integration of static data and dynamic perception information of the traffic participating elements, and reproduces the physical space information in the digital space to generate the corresponding highway digital twin system.

[0078] It should be noted that the geometric model in the embodiment of the present application includes at least: size information and shape information, the functional model includes at least: function information and list information, the performance model includes normal data information and limit data information, and the evolution model includes at least: rule information and association relationship information.

[0079] Step 105: construct a vehicle simulation system to place the vehicle simulation system into the highway digital twin system for operation, and place the autonomous driving system into the vehicle simulation system in the highway digital twin system for operation, so as to realize virtual-real testing of intelligent connected vehicles on the highway.

[0080] Specifically, the server builds a vehicle simulation system in an indoor environment to simulate an intelligent connected car, and then places the constructed vehicle simulation system into the generated highway digital twin system, so that a simulated car can be generated in the highway digital twin system. After that, the autonomous driving system is placed in the vehicle simulation system in the highway digital twin system, and the starting motion position of the vehicle is set. The static operation data of the road and the dynamic operation data of the traffic flow in the highway digital twin system are transmitted to the sensors in the autonomous driving system in real time through the vehicle simulation system. Through a two-way data connection, the autonomous driving vehicle system under test can operate on real highway roads and real traffic flows. The highway digital twin system can record the operating status of the autonomous driving system in real time, record abnormal status, and form a video playback.

[0081] This application achieves the authenticity reproduction of the self-driving car test environment through the above method, and provides a variety of choices for self-driving test sites. This application collects the real highway operation status in real time, and builds a real digital space highway scene through digital twin technology, which meets the needs of the real test environment of self-driving cars. It can also participate in real environment tests on different roads according to the needs of different customers, improve the fidelity of the test, and save special time.

[0082] Figure 2 A flowchart of another method for virtual-real testing of intelligent connected vehicles based on digital twins provided in an embodiment of the present application. Figure 2As shown, the present application inputs the real-time monitoring data of multiple highway-related scenes such as tunnels, bridges, and road sections corresponding to the real highway into the highway digital twin system, and also places the pre-built vehicle simulation system into the highway digital twin system, and then places the autonomous driving system into the vehicle simulation system in the highway digital twin system, thereby conducting virtual-reality tests of autonomous driving intelligent connected vehicles in the real highway scene provided by the highway digital twin system, making the test more realistic, improving the assurance of the test, and avoiding the technical problems of simulated traffic scenes in the prior art, not considering the behavioral differences of traffic participants due to different time and space, not fully considering the irrational behavior of scene participants, idealization and rationalization of the simulation process, and lack of fidelity.

[0083] It should be noted that Figure 2 The method shown is similar to Figure 1 The methods shown are essentially the same, therefore, Figure 3 For details of the parts not described in detail, please refer to Figure 1 The relevant description in the present application will not be repeated here.

[0084] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a virtual-real test device for intelligent networked vehicles based on digital twins, and its structure is as follows: Figure 3 shown.

[0085] Figure 3 This is a schematic diagram of the internal structure of the virtual-real test equipment for intelligent networked vehicles based on digital twins provided in the embodiment of the present application. Figure 3 As shown, the device includes:

[0086] at least one processor;

[0087] and, a memory communicatively coupled to the at least one processor;

[0088] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to:

[0089] Determine the terrain type corresponding to the expressway, and obtain image data and point cloud data of the terrain type corresponding to the expressway through acquisition equipment corresponding to the terrain type; the terrain type includes open road type and tunnel road type;

[0090] According to the satellite positioning information, the image data of the junction of the open road type and the tunnel road type are spliced ​​to obtain the corresponding digital surface model, and the point cloud data is pre-processed to obtain the corresponding digital elevation model;

[0091] The visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model are fused to obtain a three-dimensional terrain model corresponding to the highway;

[0092] The real-time monitoring device of the expressway is used to obtain the dynamic perception information of the traffic participating elements of the expressway, and the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are integrated to obtain the corresponding expressway digital twin system;

[0093] Build a vehicle simulation system to place the vehicle simulation system into the highway digital twin system for operation, and place the autonomous driving system into the vehicle simulation system in the highway digital twin system for operation, to achieve virtual and real testing of intelligent connected vehicles on the highway.

[0094] The embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0095] Determine the terrain type corresponding to the expressway, and obtain image data and point cloud data of the terrain type corresponding to the expressway through acquisition equipment corresponding to the terrain type; the terrain type includes open road type and tunnel road type;

[0096] According to the satellite positioning information, the image data of the junction of the open road type and the tunnel road type are spliced ​​to obtain the corresponding digital surface model, and the point cloud data is pre-processed to obtain the corresponding digital elevation model;

[0097] The visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model are fused to obtain a three-dimensional terrain model corresponding to the highway;

[0098] The real-time monitoring device of the expressway is used to obtain the dynamic perception information of the traffic participating elements of the expressway, and the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are integrated to obtain the corresponding expressway digital twin system;

[0099] Build a vehicle simulation system to place the vehicle simulation system into the highway digital twin system for operation, and place the autonomous driving system into the vehicle simulation system in the highway digital twin system for operation, to achieve virtual and real testing of intelligent connected vehicles on the highway.

[0100] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0101] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0103] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0104] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0105] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0106] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0107] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0108] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0109] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0110] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A virtual-real testing method for intelligent connected vehicles based on digital twins, characterized in that: The method comprises: Determine the terrain type corresponding to the expressway, and obtain image data and point cloud data of the terrain type corresponding to the expressway through a collection device corresponding to the terrain type; the terrain type includes an open road type and a tunnel road type; splicing the image data of the junction of the open road type and the tunnel road type according to the satellite positioning information to obtain a corresponding digital surface model, and preprocessing the point cloud data to obtain a corresponding digital elevation model; Fusing the visible light overall image in the digital surface model with the three-dimensional point cloud topographic map in the digital elevation model to obtain a three-dimensional terrain model corresponding to the highway; The real-time monitoring device for the expressway is used to obtain dynamic perception information of traffic participating elements of the expressway, and the static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are integrated to obtain a corresponding digital twin system of the expressway; Constructing a vehicle simulation system to place the vehicle simulation system into the highway digital twin system for operation, and placing the autonomous driving system into the vehicle simulation system in the highway digital twin system for operation, so as to implement virtual and real testing of intelligent connected vehicles on the highway; Before splicing the image data of the junction of the open road type and the tunnel road type according to the satellite positioning information to obtain the corresponding digital surface model, the method further includes: Sharpening the image data by using a preset function, and reducing noise on the sharpened image data by using a median filter, thereby preprocessing the image data; Acquire the time corresponding to the image data from the image data, and sort the preprocessed multiple image data in order from early to late; According to the image registration technology and the fusion reconstruction technology, and in accordance with the corresponding order of the plurality of image data, the plurality of image data are sequentially spliced; The spliced ​​image data are fused through a convolutional neural network model to obtain an overall image corresponding to the highway; The step of splicing the image data of the junction of the open road type and the tunnel road type according to the satellite positioning information to obtain the corresponding digital surface model specifically includes: Acquire satellite positioning information corresponding to the image data from the image data, and respectively determine a Laplacian pyramid of the image to be stitched corresponding to the image data and a masked Gaussian pyramid of the same size; splicing two images corresponding to the Laplacian pyramids with data resolutions from low to high according to the masked Gaussian pyramid of the current size, and obtaining a corresponding digital surface model; the digital surface model includes a visible light overall image with the satellite positioning information; The preprocessing of the point cloud data to obtain a corresponding digital elevation model specifically includes: Sorting the acquired plurality of point cloud data in ascending order according to the time sequence from earliest to latest, and filtering the point cloud data by a median filter to remove noise in the point cloud data; The denoised point cloud data is classified by a pre-trained convolutional neural network, and at the feature points of the highway, an affine transformation is performed on the images corresponding to two adjacent point cloud data according to the classification; Aligning the feature points in the images corresponding to the two adjacent point cloud data to achieve splicing of the point cloud data and obtain a corresponding digital elevation model; the digital elevation model includes a three-dimensional point cloud topographic map corresponding to the highway; The step of fusing the visible light overall image in the digital surface model with the three-dimensional point cloud topographic map in the digital elevation model to obtain a three-dimensional terrain model corresponding to the highway specifically includes: Performing feature detection on the visible light overall image in the digital surface model, and generating a region of interest corresponding to the visible light overall image; The satellite positioning information corresponding to the region of interest is calculated by a key point calibration algorithm and a geometric algorithm, and the region of interest is projected into a three-dimensional point cloud space according to the satellite positioning information to generate a corresponding three-dimensional region of interest; According to the three-dimensional region of interest, the visible light overall image in the digital surface model and the three-dimensional point cloud topographic map in the digital elevation model are fused to obtain a three-dimensional terrain model corresponding to the expressway.

2. The virtual-real testing method for intelligent connected vehicles based on digital twins according to claim 1 is characterized in that: The obtaining of dynamic perception information of traffic participating elements of the expressway through the expressway real-time monitoring device specifically includes: A road monitoring device in the real-time monitoring device for the expressway is constructed by means of monitoring equipment, transmission equipment and power supply equipment; The tunnel monitoring device in the real-time monitoring device of the highway is constructed by ventilation, lighting, traffic control, monitoring, emergency telephone broadcasting, fire alarm and radar event detection devices as well as sensor equipment, transmission equipment and power supply equipment; And, constructing the traffic operation monitoring device in the real-time monitoring device of the expressway through traffic flow monitoring, traffic event monitoring and traffic weather monitoring equipment; Obtaining dynamic perception information of traffic participating elements corresponding to the expressway through the front-end sensors in the road monitoring device, the tunnel monitoring device and the traffic operation monitoring device; The encapsulated dynamic perception information of the traffic participating elements is transmitted to the three-dimensional terrain model through a unified data transmission protocol.

3. The virtual-real testing method of intelligent connected vehicles based on digital twins according to claim 1 is characterized in that: The static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are integrated to obtain the corresponding highway digital twin system, which specifically includes: According to the acquired dynamic perception information of the traffic participating elements of the expressway, a key information data set of the expressway is constructed, wherein the key information represents information data corresponding to each key point in the expressway, and the key points include at least: a toll station, a service area, an overpass, and a construction point; By software programming, a geometric model, a functional model, a performance model and an evolution model related to key information of the highway are constructed in a digital space; Inputting the data corresponding to each of the key points in the key information data set into the geometric model, the functional model, the performance model and the evolution model to realize page browsing and human-computer interaction interface; The static data corresponding to the three-dimensional terrain model and the dynamic perception information of the traffic participating elements are input into the geometric model, the functional model, the performance model and the evolution model to realize the integration of the static data and the dynamic perception information of the traffic participating elements and obtain the corresponding highway digital twin system.

4. The virtual-real testing method for intelligent connected vehicles based on digital twins according to claim 3 is characterized in that: The geometric model includes at least: size information and shape information, the functional model includes at least: function information and list information, the performance model includes normal data information and limit data information, and the evolution model includes at least: rule information and association relationship information.

5. The virtual and real test equipment of intelligent connected vehicles based on digital twins is characterized by: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the virtual-reality testing method for intelligent connected vehicles based on digital twins as described in any one of claims 1-4.

6. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A virtual-reality testing method for an intelligent connected vehicle based on digital twins as described in any one of claims 1 to 4.

Images