Vehicle three-dimensional dynamic simulation method and system under high-speed high-definition video driving

Abstract

The invention provides a vehicle three-dimensional dynamic simulation method and system driven by a high-speed high-definition video. The simulation method comprises the following steps: performing equipment registration and access on various cameras on an expressway; a streaming media processing service is constructed, and vehicle information extraction and standardized output based on real-time videos are realized; and in combination with a basic highway digital twin scene and a vehicle three-dimensional model, carrying out three-dimensional dynamic real-time simulation on the vehicle in a simulation system under the driving of vehicle information data generated in real time. According to the invention, AI and multiple algorithms are combined, effective cloning of a real dynamic environment in a digital twinning world can be realized, and more immersive experience can be provided, so that a worker can randomly check the real-time running condition of a vehicle at any position in an expressway digital twinning scene by means of the system. Meanwhile, traffic flow management can be effectively improved, road safety is improved, and operation cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and specifically to a vehicle three-dimensional dynamic simulation method and system driven by high-speed high-definition video. Background Art

[0002] The highway monitoring system can promptly detect and handle various traffic anomalies, such as traffic accidents and road congestion, by monitoring road conditions in real time, thus ensuring driving safety. In addition, the monitoring system can also monitor road conditions in real time through high-definition cameras and sensor networks, including road conditions, weather changes, etc., to provide drivers with accurate navigation guidance and avoid safety hazards.

[0003] In the current highway monitoring system, the monitoring center mainly relies on large-screen video monitoring systems to manage traffic by displaying video streams transmitted by various cameras. Although this method can provide real-time video information, it has obvious limitations and cannot provide comprehensive and intuitive traffic status perception, and cannot meet the growing traffic management needs. The problems it has include: (1) Viewing angle limitation: The existing system at the monitoring center can only display the two-dimensional view of a fixed camera. This viewing angle limits the comprehensive understanding of the entire highway section.

[0004] (2) Information overload: A large amount of video may cause monitoring personnel to fail to process information in a timely manner when faced with emergencies, making it difficult for them to respond quickly.

[0005] (3) Lack of immersion: Two-dimensional video surveillance lacks three-dimensional spatial perception, making it difficult for monitoring personnel to intuitively understand the dynamic distribution and movement trends of vehicles. Summary of the invention

[0006] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a three-dimensional dynamic simulation method and system for vehicles driven by high-speed high-definition video to solve the problems raised in the above-mentioned background technology. The present invention can realize the effective cloning of the real dynamic environment (vehicles, etc.) in the digital twin world, and can provide a more immersive experience. It can significantly enhance the monitoring and management capabilities of the highway monitoring center, and achieve the goals of improving road safety, reducing operating costs, and realizing intelligent traffic management.

[0007] In order to achieve the above object, the present invention is implemented by the following technical solution: a vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video, comprising the following steps: S1. Register and access various cameras on the highway. General IoT management platforms have this capability to achieve integrated management of camera devices and access real-time video through HTTP, HTTPS, RTP, RTSP and other protocols. S2. Build streaming media processing services to extract vehicle information based on real-time video and output it in a standardized manner; S3. Combining the basic highway digital twin scene and the vehicle three-dimensional model, driven by the vehicle information (JSON format) data generated in real time in step S2, a three-dimensional dynamic real-time simulation of the vehicle is performed through the simulation system.

[0008] Furthermore, in S1, the range of the camera's frustum is combined to cover the entire highway area, and the equipment is numbered according to the direction of the road.

[0009] Furthermore, cameras are installed at key locations, including toll booths, intersections, bridges, tunnels, etc., and the coverage area of ​​the viewing frustum (tetrahedron) of each camera is calculated based on the camera's installation position, height, and viewing angle. Then the union of all camera areas is taken, and only the union covers the entire highway.

[0010] Furthermore, the processing process of step S2 is as follows: S2.1, read a frame of the video; S2.2. Use the YOLOv8 model to detect vehicles, identify the type of vehicle, the pixel position of the vehicle, and segment the image containing the vehicle; S2.3, based on the vehicle image segmented and output in the previous step, perform algorithm analysis on the vehicle color; S2.4, calculate the vehicle's geographic location, and combine the lane-level "high-precision navigation data" to output the vehicle's location, driving direction, and lane number; S2.5. Define specifications and standardize the output of vehicle information.

[0011] Furthermore, the image processing process in S2.2 includes loading images, target detection, drawing target bounding boxes, outputting vehicle types, pixel box positions, segmenting images containing vehicles, saving segmented vehicle images, and license plate recognition, wherein license plate recognition uses OCR (Optical Character Recognition) for character recognition and finally outputs the license plate number.

[0012] Furthermore, the specific process analyzed in S2.2 includes color space conversion, from RGB to HSV. HSV color space (hue, saturation, brightness) is more suitable for color recognition because it separates color information from brightness and saturation; color segmentation, according to the HSV range of the target color, the image is thresholded to extract the target color area. At the same time, morphological operations such as dilation and erosion can be used to remove noise and connect adjacent color areas; color classification, using the K-means clustering algorithm to classify colors and determine the main color of the vehicle; color output, the output of the main color is the vehicle color.

[0013] Furthermore, the S2.5 obtains the detection time, license plate number, vehicle position, vehicle color, vehicle type, driving direction, driving speed (combined with the position distance of the previous and next frames / the time taken for the previous and next frames), lane number, and stake number through real-time video, and outputs them in JSON format.

[0014] Furthermore, in step S3, the highway digital twin scene and vehicle three-dimensional model are constructed by combining programmatic art with manual modeling, and the vehicle three-dimensional model includes four categories: truck, car, bus, and motorcycle, and each type of vehicle model supports external setting of vehicle color.

[0015] As for the simulation system used in the above-mentioned simulation method, the simulation system includes a camera device, a high-speed high-definition road network model, a vehicle asset model and an Internet of Things management platform, wherein the camera device extracts vehicle information from the collected video based on the streaming media processing service, and the streaming media processing service is used to realize vehicle identification in real-time video, vehicle geographic location calculation, and standardized output of vehicle information.

[0016] Furthermore, the vehicle asset model is simulated in real time by a three-dimensional rendering engine, and the camera device is connected to the Internet of Things management platform and registered.

[0017] Beneficial effects of the present invention: 1. This vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video uses AI and a combination of multiple algorithms to achieve effective cloning of real dynamic environments (vehicles, etc.) in the digital twin world, providing a more immersive experience.

[0018] 2. The vehicle three-dimensional dynamic simulation system driven by high-speed high-definition video can significantly improve the monitoring and management capabilities of the highway monitoring center, allowing staff to use the system to freely view the real-time operation status of vehicles at any location in the highway digital twin scene, and obtain information more flexibly and comprehensively.

[0019] 3. The dynamic simulation scenario constructed by the present invention can effectively connect to the real-time warning system and resource allocation system, which will bring faster response time and more optimized resource allocation. Therefore, this solution can not only effectively improve traffic flow management and improve road safety, but also reduce operating costs and achieve the goal of intelligent traffic management. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video of the present invention; Figure 2 This is a camera coverage diagram in the present invention; Figure 3 A flow chart of the streaming media processing service in the present invention; Figure 4 This is a flow chart for implementing vehicle detection in YOLOv8 in the present invention; Figure 5 This is a flow chart for implementing vehicle color analysis in the present invention; Figure 6 It is an overall diagram of the vehicle simulation timing of the present invention; Figure 7 for Figure 6 A partial enlarged view on the middle left; Figure 8 for Figure 6 A partial enlargement of the right side. DETAILED DESCRIPTION

[0021] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0022] Embodiment 1 See also Figures 1 to 8 The present invention provides the following technical solution, which specifically includes the following steps: S1, register and access various cameras on the highway (such as: gun cameras, "Ray Vision All-in-One Machine", etc.). General IoT management platforms have this capability to achieve integrated management of camera devices and access real-time video through HTTP, HTTPS, RTP, RTSP and other protocols.

[0023] In order to achieve seamless simulation of vehicles on the entire highway, on the one hand, the camera's frustum range needs to be combined to cover the entire highway area, and on the other hand, the equipment needs to be numbered according to the direction of the road. The specific requirements are as follows.

[0024] S1.1 Camera Location 1. Key locations need to be installed, including toll booths, intersections, bridges, tunnels, etc.; 2. Calculate the coverage area of ​​each camera's frustum (tetrahedron) based on the camera's installation position, height, and viewing angle, and then take the union of all camera areas. Only the union can cover the entire highway. The coverage area of ​​a single camera, such as Figure 2 As shown: The bottom surface of the quadrangular pyramid of the camera is a quadrilateral S.

[0025] 1. Calculate the horizontal distance of the viewing cone

[0026] 2. Calculate the vertical distance of the viewing cone

[0027] 3. Calculate the coordinates of the four vertices of the viewing cone Lower left corner coordinates

[0028] Lower right corner coordinates

[0029] Upper left corner coordinates

[0030] Upper right corner coordinates

[0031] Here, R is the average radius of the Earth, which is approximately 6371 kilometers.

[0032] S1.2 Camera number A unified numbering format "CSN" is used, where C represents the camera, S represents the section number, and N represents the camera number (serial number).

[0033] S2, builds streaming media processing services to realize vehicle information extraction and standardized output based on real-time video.

[0034] like Figure 4 The specific processing process is as follows: S2.1 reads a frame of the video.

[0035] S2.2 uses the YOLOv8 model to detect vehicles, identify the type of vehicle, the pixel location of the vehicle, and segment the image containing the vehicle. YOLOv8 is a SOTA model that builds on the success of previous YOLO versions and introduces new features and improvements to further improve performance and flexibility.

[0036] In order to simulate as real-time as possible and avoid time differences caused by calculations, we use the lightest model in YOLOv8, the YOLOv8 Nano model. On a GPU, the inference of a 1080P image can usually be completed within a few milliseconds.

[0037] The specific implementation process is as follows Figure 4 shown.

[0038] No additional model training is done, and the vehicle types provided in YOLOv8 are used directly, including truck, car, bus, and motorcycle.

[0039] License plate recognition uses OCR (Optical Character Recognition) to perform character recognition and ultimately output the license plate number.

[0040] At the same time, the pixel position of the vehicle in the image can be output .

[0041] S2.3 performs algorithmic analysis on the vehicle color based on the vehicle image segmented and outputted in the previous step.

[0042] The specific implementation process is as follows Figure 5 shown.

[0043] S2.3.1 Color space conversion, from RGB to HSV. The HSV color space (hue, saturation, brightness) is more suitable for color recognition because it separates color information from brightness and saturation.

[0044] S2.3.2 Color segmentation, according to the HSV range of the target color, the image is thresholded to extract the target color area. At the same time, morphological operations such as dilation and erosion can be used to remove noise and connect adjacent color areas.

[0045] S2.3.3 Color classification, use K-means clustering algorithm to classify colors and determine the main color of the vehicle.

[0046] S2.3.4 Color output, the output main color is the vehicle color.

[0047] S2.4 Vehicle geographic location calculation, combined with "high-precision navigation data" (lane level), can output the vehicle's location, driving direction (for example: Beijing-Shanghai Expressway up or down), and driving lane number.

[0048] First, according to the camera parameters (horizontal field of view angle , vertical field of view , Image resolution 、 focal length ), camera location , camera orientation (azimuth , north is 0 degrees, clockwise), camera installation height ,Vehicle pixel position , calculate the geographic location of the corresponding vehicle in the video .

[0049] S2.4.1 Image center coordinates: Image Center

[0050] S2.4.2 Calculate the horizontal and vertical angles: Horizontal angle :

[0051] Vertical angle :

[0052] S2.4.3 Synthesis of actual direction: Angle of the vehicle relative to the north direction:

[0053] S2.4.4 Estimation of vehicle distance:

[0054] S2.4.5 Calculation of the geographical location of the vehicle: Relative longitude and latitude: Radius of the earth kilometer

[0055] Vehicle longitude and latitude coordinates:

[0056] Then, based on the geographical location of the vehicle and combined with the "high-precision navigation data", through the spatial point positioning method (GIS spatial relationship), the driving direction and lane number of the vehicle can be easily obtained.

[0057] S2.5 Define specifications and standardize the output of vehicle information. Combining the previous steps, it can be seen that through real-time video, the detection time, license plate number, vehicle location, vehicle color, vehicle type, driving direction, driving speed (combined with the distance between the front and rear frames / the time taken for the front and rear frames), lane number, and stake number can be obtained, and the output is in JSON format. The standard structure is defined as follows: { "pathList": { "plateNo": "E ******", "latitude": 30.91406661872873, "longitude": 114.04554724913011, "plateColor": "Black", "originalType": 1, "direction": 1, "speed": 72.66583, "laneNo": 3, "stakeId": "K7+859", }, { "plateNo": "Lu******", "latitude": 30.91479178586509, "longitude": 114.0454401327111, "plateColor": "Red", "originalType": 2, "direction": 1, "speed": 80.18831, "laneNo": 2, "stakeId": "K7+424", } , "time": 1723026152850, "timeStamp": "2024-08-22 10:05:00:000" } S3. Combining with the basic digital twin scenario of the highway and the 3D vehicle model, driven by the vehicle information (in JSON format) data generated in real time in step 2, the 3D dynamic real-time simulation of the vehicle is carried out through the simulation system.

[0058] The digital twin scenario of the highway and the 3D vehicle model can be constructed by combining procedural art and manual modeling (using tools such as 3DMax, Maya, Blender, etc.), which will not be elaborated here. It should be noted that the 3D vehicle model needs to include four types: truck, car, bus, and motorcycle, and each type of vehicle model supports external setting of the vehicle color. Through the recognition of a certain frame of high-speed high-definition video, the simulation of a certain frame can be realized, that is, obtaining the vehicle JSON information of a certain frame and simulating it in the simulation system.

[0059] Example 2

[0060] In this embodiment, the complete simulation of vehicles on the entire highway can be realized. From the physical topology architecture, each camera on the highway has a corresponding distributed node for real-time calculation and detection, without a sequential order, that is, each camera has been running in real time for detection: In order to ensure the continuity of vehicle driving from a visual perspective, that is, to ensure that the vehicle's driving path on the highway is complete from the vehicle's perspective. From the system implementation level, it is necessary to effectively manage the high-speed high-definition camera detection: (1) At the entrance of the highway, it is necessary to detect and record the management of vehicles (add the corresponding vehicle model in the simulation system); (2) On the highway, orderly detection is carried out according to the camera number "CSN". When a vehicle disappears from the current camera, it automatically switches to the next camera detection to ensure continuity; (3) At the exit of the highway, it is necessary to detect and record the management of vehicles (remove the corresponding vehicle simulation model).

[0061] If the highway entrance camera is , highway cameras are ,in , the highway exit camera is The vehicle simulation timing diagram based on camera video recognition on the highway from the perspective of the vehicle is as follows: Figures 6 to 8 shown.

[0062] In this embodiment, it is necessary to combine the front and rear frame positions of the vehicle to construct the vehicle running vector (vehicle driving direction, angle). When the next frame information is not obtained, it can be realized by interpolation along the vector direction to meet the continuity of vehicle simulation. Finally, the real-time dynamic simulation of all vehicles running on the entire highway is realized.

[0063] The above method uses AI and a combination of multiple algorithms to achieve effective cloning of real dynamic environments (vehicles, etc.) in the digital twin world, providing a more immersive experience. However, in order to be as consistent as possible, high requirements are placed on data transmission and computing timeliness, which is very dependent on hardware configuration; the specific interval between frames for detection and 3D simulation can be reasonably adjusted by the user based on the existing hardware conditions.

[0064] The basic principles and main features of the present invention and the advantages of the present invention are shown and described above. For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention.

[0065] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A three-dimensional dynamic simulation method of a vehicle driven by high-speed high-definition video, characterized in that: The following steps are involved: S1. Register and access various cameras on the highway. General IoT management platforms have this capability to achieve integrated management of camera devices and access real-time video through HTTP, HTTPS, RTP, RTSP and other protocols. S2. Build streaming media processing services to extract vehicle information based on real-time video and output it in a standardized manner; S3. Combining the basic highway digital twin scene and the vehicle three-dimensional model, driven by the vehicle information (JSON format) data generated in real time in step S2, a three-dimensional dynamic real-time simulation of the vehicle is performed through the simulation system.

2. The vehicle three-dimensional dynamic simulation method under high-speed high-definition video drive according to claim 1 is characterized in that: In S1, the frustum ranges of the cameras are combined to cover the entire highway area, and the devices are numbered according to the direction of the road.

3. The vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video according to claim 2 is characterized in that: Cameras are installed at key locations, including toll booths, intersections, bridges, tunnels, etc. The coverage area of ​​each camera's frustum (tetrahedron) is calculated based on the camera's installation position, height, and viewing angle. Then the union of all camera areas is taken, and only the union covers the entire highway.

4. The vehicle three-dimensional dynamic simulation method under high-speed high-definition video drive according to claim 1 is characterized in that: The processing process of step S2 is as follows: S2.1, read a frame of the video; S2.

2. Use the YOLOv8 model to detect vehicles, identify the type of vehicle, the pixel position of the vehicle, and segment the image containing the vehicle; S2.3, based on the vehicle image segmented and output in the previous step, perform algorithm analysis on the vehicle color; S2.4, calculate the vehicle's geographic location, and combine the lane-level "high-precision navigation data" to output the vehicle's location, driving direction, and lane number; S2.

5. Define specifications and standardize the output of vehicle information.

5. The vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video according to claim 4 is characterized in that: The image processing process in S2.2 includes loading images, target detection, drawing target bounding boxes, outputting vehicle types, pixel box positions, segmenting images containing vehicles, saving segmented vehicle images, and license plate recognition. License plate recognition uses OCR (Optical Character Recognition) for character recognition and finally outputs the license plate number.

6. The vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video according to claim 5 is characterized in that: The specific process analyzed in S2.2 includes color space conversion, from RGB to HSV. The HSV color space (hue, saturation, brightness) is more suitable for color recognition because it separates color information from brightness and saturation; color segmentation, according to the HSV range of the target color, the image is thresholded to extract the target color area; at the same time, morphological operations such as dilation and erosion can be used to remove noise and connect adjacent color areas; color classification, using the K-means clustering algorithm to classify colors and determine the main color of the vehicle; color output, the output main color is the vehicle color.

7. The vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video according to claim 1 is characterized by: In the S2.5, the detection time, license plate number, vehicle position, vehicle color, vehicle type, driving direction, driving speed (combined with the position distance of the previous and next frames / the time taken for the previous and next frames), lane number, and pile number are obtained through real-time video and output in JSON format.

8. The vehicle three-dimensional dynamic simulation method driven by high-speed high-definition video according to claim 1 is characterized in that: In step S3, the highway digital twin scene and vehicle three-dimensional model are constructed by combining programmatic art with manual modeling. The vehicle three-dimensional model includes four categories: truck, car, bus, and motorcycle, and each type of vehicle model supports external setting of vehicle color.

9. The simulation system used in the simulation method as claimed in claim 1, characterized in that: The simulation system includes camera equipment, high-speed high-definition road network model, vehicle asset model and Internet of Things management platform. The camera equipment extracts vehicle information from the collected video based on streaming media processing service, and the streaming media processing service is used to realize vehicle identification in real-time video, vehicle geographic location calculation, and standardized output of vehicle information.

10. The simulation system according to claim 9, characterized in that: The vehicle asset model is simulated in real time by a three-dimensional rendering engine, and the camera device is connected to the Internet of Things management platform and registered.

Images