Sora-based driving environment video stream generation method and simulation test system

The Sora-based method generates realistic driving environment video streams to enhance autonomous driving simulation tests, addressing the inefficiencies and inaccuracies of traditional methods by providing comprehensive and reliable simulation environments.

CN119066845APending Publication Date: 2024-12-03CHINA FAW CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411092136.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing methods for autonomous driving simulation testing are time-consuming and costly, and struggle to fully simulate the complexity and diversity of real-world driving scenarios, leading to inaccurate and unreliable test results, especially in extreme or rare situations.

Method used

A method and system using Sora technology to generate a driving environment video stream by obtaining comprehensive travel environment data, constructing virtual driving environment information, and generating a video stream using simulation environment construction rules, which includes weather, time, road type, and traffic conditions, and integrating road vehicle, pedestrian, and obstacle data.

Benefits of technology

This approach enhances the realism and reliability of autonomous driving simulation tests, reducing test cycles and costs while improving efficiency and coverage, and supporting intelligent driving systems and traffic management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066845A_ABST
    Figure CN119066845A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a Sora-based driving environment video stream generation method and a simulation test system, and belongs to the technical field of vehicle automatic driving. The method comprises the steps of obtaining travel environment data; according to the travel environment data, obtaining driving internal scene data; constructing virtual driving environment information according to the travel environment data and the driving internal scene data; and generating and obtaining a driving environment video stream by using a Sora technology according to the virtual driving environment information. According to the invention, the scene richness of the automatic driving simulation test can be further verified, the test period and cost can be greatly reduced, the efficiency is improved, and the test coverage and the test reliability are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicle autonomous driving technology, and in particular to a method for generating a driving environment video stream based on Sora and a simulation test system. Background Art

[0002] In autonomous driving virtual simulation testing, the traditional method relies on road data collection to build a simulation environment, which requires a lot of time and cost. The process of collecting, processing and preparing data, as well as building a simulation environment based on the data is very time-consuming and expensive.

[0003] Existing methods often have difficulty in fully simulating the complexity and diversity of the real world when generating simulation environments, which may cause the test results to deviate from the performance in actual scenarios, affecting the accuracy and reliability evaluation of the autonomous driving system.

[0004] Traditional methods may not cover all possible driving scenarios and situations, especially when faced with extreme or rare situations. This limits the ability to fully validate the autonomous driving system. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a method for generating a driving environment video stream based on Sora and a simulation test system.

[0006] The technical solution adopted by the present invention is:

[0007] On the one hand, an embodiment of the present invention provides a method for generating a driving environment video stream based on Sora, the method comprising the following steps:

[0008] Obtain travel environment data;

[0009] Obtaining driving interior scene data according to the travel environment data;

[0010] Constructing virtual driving environment information according to the travel environment data and the driving internal scene data;

[0011] According to the virtual driving environment information, a driving environment video stream is generated using Sora technology.

[0012] Furthermore, the obtaining of travel environment data comprises the following steps:

[0013] Acquire weather condition data; the weather condition data includes sunny day information, rainy day information, and foggy day information;

[0014] Acquire time period data; the time period data includes daytime information and nighttime information;

[0015] Acquire road type data; the road type data includes highway information, city street information, and rural road information;

[0016] Acquiring traffic condition data; the traffic condition data includes traffic congestion information, traffic flow information, and traffic accident scene information;

[0017] The weather condition data, the time period data, the road type data, and the traffic condition data are used as travel environment data.

[0018] Furthermore, the obtaining of travel environment data comprises the following steps:

[0019] Use the vehicle-mounted camera to record the travel environment and obtain a recorded video;

[0020] The recorded video is preprocessed to obtain travel environment data.

[0021] Furthermore, obtaining the driving interior scene data according to the travel environment data comprises the following steps:

[0022] Extracting features and modeling the travel environment data to obtain road vehicle object information, pedestrian object information, and obstacle object information;

[0023] The road vehicle object information includes road vehicle motion trajectory information, road vehicle speed information, and road vehicle acceleration information;

[0024] The pedestrian object information includes pedestrian motion trajectory information, pedestrian speed information, and pedestrian acceleration information;

[0025] The obstacle object information includes obstacle position information and obstacle movement information;

[0026] The road vehicle object information, the pedestrian object information, and the obstacle object information are used as driving internal scene data.

[0027] Furthermore, constructing virtual driving environment information according to the travel environment data and the driving internal scene data comprises the following steps:

[0028] Obtaining interaction data according to the travel environment data and the driving internal scene data; the interaction data includes interaction data between road layout and traffic flow, and interaction data between weather conditions and traffic flow;

[0029] The interaction data is converted into virtual driving environment information.

[0030] Furthermore, obtaining a driving environment video stream according to the virtual driving environment information comprises the following steps:

[0031] Get the simulation environment construction rules;

[0032] According to the simulation environment building rules and the virtual driving environment information, the driving environment video stream is generated using Sora technology.

[0033] On the other hand, an embodiment of the present invention further provides a simulation test system based on Sora's driving environment video stream, the simulation test system comprising:

[0034] The first module is used to obtain travel environment data;

[0035] The second module is used to obtain driving internal scene data according to the travel environment data;

[0036] A third module is used to construct virtual driving environment information according to the travel environment data and the driving internal scene data;

[0037] The fourth module is used to obtain a driving environment video stream according to the virtual driving environment information.

[0038] Furthermore, the simulation test system also includes a driving environment video stream generation unit based on Sora, a video injection unit, an automatic driving control module, a simulation model and a host computer interface;

[0039] The driving environment video stream generation unit based on Sora is used to generate a camera video stream as simulation data according to the simulation environment construction rules;

[0040] The Sora-based driving environment video stream generation unit transmits the camera video stream to the video injection unit through an interface;

[0041] The autonomous driving control module serves as the device under test;

[0042] The video injection unit receives the camera video stream from the Sora-based driving environment video stream generation unit, encapsulates the camera video stream and injects it into the automatic driving control module in series;

[0043] The simulation model is connected to the host computer interface via Ethernet communication;

[0044] The automatic driving control module is connected to the simulation model via Ethernet communication;

[0045] The automatic driving control module is connected to the host computer interface via Ethernet communication.

[0046] Furthermore, the video injection unit includes a video player and a camera module.

[0047] Further, the automatic driving control module calculates the camera video stream of the video injection unit and the input signal of the simulation model to obtain an output signal;

[0048] The host computer interface displays the output signal.

[0049] The embodiments of the present application include at least the following beneficial effects: The present application provides a method for generating a driving environment video stream based on Sora and a simulation test system. The present invention obtains travel environment data; obtains driving internal scene data based on the travel environment data; constructs virtual driving environment information based on the travel environment data and the driving internal scene data; and generates a driving environment video stream using Sora technology based on the virtual driving environment information. The present invention can further verify the scene richness of autonomous driving simulation testing, greatly reduce test cycles and costs, improve concerted efficiency, and increase test coverage and test reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is a flow chart of a method for generating a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0051] Figure 2 is a flow chart of a method for generating a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0052] Figure 3 is a flow chart of a method for generating a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0053] Figure 4 is a flow chart of a method for generating a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0054] Figure 5 is a flow chart of a method for generating a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0055] Figure 6 is a flow chart of a method for generating a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0056] Figure 7 is a schematic diagram of a simulation test system for a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0057] Figure 8 is a schematic diagram of a simulation test system for a driving environment video stream based on Sora provided in an embodiment of the present invention;

[0058] Fig. 9It is a structural diagram of an autonomous driving virtual simulation test system based on Sora's driving environment video stream provided by an embodiment of the present invention;

[0059] Fig.10 It is a structural diagram of an autonomous driving virtual simulation test system based on Sora's driving environment video stream provided in an embodiment of the present invention.

[0060] The annotations of the various structural parts in the accompanying drawings are as follows: Sora-based video stream generation unit 1, video injection unit 2, automatic driving control module 3, simulation model 4, host computer interface 5, video player 21 and camera module 22. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0062] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0063] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0066] 1) Sora, an AI-powered instant video generator.

[0067] 2) L2 level autonomous driving system refers to a partial autonomous driving system, also known as a "partially automated system", which is between fully manual driving and fully autonomous driving in terms of degree of automation.

[0068] 3) L3 and above autonomous driving technologies refer to higher-level automation systems that have higher automation capabilities and less need for human intervention compared to L2 levels.

[0069] 4) CAN bus (Controller Area Network) is a communication protocol and bus standard used for real-time control systems.

[0070] 5) Ethernet card is a network interface card (NIC) used to connect a computer to an Ethernet local area network (LAN) or wide area network (WAN).

[0071] 6) SPI board (Serial Peripheral Interface) is a serial peripheral interface used for short-distance communication between microcontrollers or other digital integrated circuits.

[0072] The present invention takes into account that with the development of autonomous driving technology, more and more cars are equipped with autonomous driving systems. The L2 level autonomous driving solution has been fully rolled out on cars. With the advancement of technology and the update of regulations, L3 and higher level autonomous driving technologies will also be tested and installed on cars. With the implementation of high-level autonomous driving technology, there is an important issue facing everyone, how to ensure the safe and stable operation of the car's autonomous driving system on the road. This requires multi-form, multi-scenario, long-term and long-mileage verification of the autonomous driving system before it is officially put into production.

[0073] As one of the important test links of autonomous driving, the virtual simulation test of autonomous driving will implement the test of the autonomous driving system through video injection and signal simulation. It has the ability to repeat verification for a long time, long mileage, and multiple scenarios, and can effectively verify the functions of autonomous driving technology. For the hardware-in-the-loop simulation test link of high-level autonomous driving technology, it is necessary to prepare dynamic scene data in the form of video to inject into the camera, so as to realize the closed-loop simulation test of autonomous driving. At present, the simulation environment is built based on road acquisition data, and driving videos are produced for simulation testing according to the simulated camera model parameters in the built simulation environment. This work is time-consuming and costly. The mileage and sample richness of the map of the simulation environment are proportional to the acquisition work. Preparing the simulation environment will take a lot of time and cost.

[0074] The present invention provides an automatic driving virtual simulation test system and method based on Sora to generate driving environment video injection, which can generate high-quality video for virtual simulation test video injection. According to the road environment rules, combined with certain real environment collection data, Sora can be used to generate rich mileage driving videos that meet the requirements as test scenes. Under the condition that the test environment is well built, continuous long-term, multi-scene, and long-mileage simulation verification can be achieved.

[0075] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0076] On the one hand, the embodiment of the present invention provides a method for generating a driving environment video stream based on Sora, specifically, referring to Figure 1 , the method comprises the following steps:

[0077] S100, obtaining travel environment data;

[0078] S200, obtaining driving interior scene data according to the travel environment data;

[0079] S300, constructing virtual driving environment information according to the travel environment data and the driving internal scene data;

[0080] S400: Generate a driving environment video stream using Sora technology based on the virtual driving environment information.

[0081] As an optional implementation, S100 of the embodiment of the present invention obtains travel environment data. In this step, the system collects relevant travel environment data, which may include information such as the vehicle's location, speed, and surrounding traffic conditions.

[0082] As an optional implementation, S200 of the embodiment of the present invention obtains driving interior scene data based on the travel environment data. The system further analyzes and extracts the driver's interior scene data, such as the driver's perspective, the state of the dashboard in the car, the position of the passengers, etc., based on the data obtained in the S100 stage.

[0083] As an optional implementation, S300 of the embodiment of the present invention constructs virtual driving environment information based on the travel environment data and the driving internal scene data. In this step, the system integrates and processes the data from the S100 and S200 stages to generate a comprehensive virtual driving environment model that reflects the environment and internal conditions actually faced by the driver.

[0084] As an optional implementation, in the embodiment S400 of the present invention, the driving environment video stream is generated using the Sora technology based on the virtual driving environment information. Finally, the system uses the Sora technology to generate a high-quality driving environment video stream in real time based on the constructed virtual driving environment information. These video streams can be used to simulate driving situations, driving behavior analysis, and the development and testing of driving assistance systems.

[0085] Through these steps, the embodiment of the present invention combines advanced data acquisition and processing technologies to provide an efficient, accurate and practical solution for generating driving environment video streams, which helps to improve driving safety and the efficiency of technology development.

[0086] refer to Figure 2 The embodiment of the present invention discloses step S100 of acquiring travel environment data, including the following steps:

[0087] S110, obtaining weather condition data; the weather condition data includes sunny day information, rainy day information, and foggy day information;

[0088] S120, obtaining time period data; the time period data includes daytime information and nighttime information;

[0089] S130, obtaining road type data; the road type data includes highway information, city street information, and rural road information;

[0090] S140, obtaining traffic condition data; the traffic condition data includes traffic congestion information, traffic flow information, and traffic accident scene information;

[0091] S150: Using weather condition data, time period data, road type data, and traffic status data as travel environment data.

[0092] An embodiment of the present invention relates to a method for acquiring travel environment data, which aims to effectively capture and analyze multiple factors of the driving environment to support applications such as intelligent driving systems, traffic management, and road condition prediction.

[0093] The following is a detailed extended description of step S100:

[0094] In modern traffic management and intelligent driving systems, accurate travel environment data is crucial for safe driving and effective route planning. The present invention obtains comprehensive travel environment data through the following key steps:

[0095] As an optional implementation, step S110 of the present invention obtains weather condition data:

[0096] Weather conditions directly affect the driver's field of vision, the performance of the vehicle, and the safety of the road. Therefore, the present invention first obtains a variety of weather condition data including sunny days, rainy days, and foggy days. Sunny days usually mean good visibility and high road friction, which is suitable for high-speed driving and long-distance driving. Rainy days may cause the road to be slippery and reduce traction, requiring the driver to adopt a more cautious driving strategy. Foggy days may cause visibility to drop sharply, increasing the risk of driving. By obtaining these detailed weather condition data, the present invention can provide the driver with real-time road condition information so that he can make safe driving decisions.

[0097] As an optional implementation, step S120 of the present invention obtains time period data:

[0098] The time period has a significant impact on the driving environment and road conditions. The driving conditions during the day and at night are very different, so the present invention acquires and analyzes data from different time periods. The daytime usually provides good lighting conditions and higher visibility, which is suitable for long-distance visual navigation and fast-response driving. At night, due to the dim light, the driver needs to rely on car lights and street lights for lighting, and the driving risk is relatively increased. By collecting and analyzing data during the day and at night, the present invention can provide the driver with targeted driving suggestions and safer and more efficient route planning.

[0099] As an optional implementation, step S130 of the present invention obtains road type data:

[0100] Different types of roads have a significant impact on driving conditions and driving strategies. The present invention obtains detailed data on multiple road types including highways, city streets and country roads. Highways are usually designed for high-speed driving and long-distance driving, requiring drivers to maintain a stable speed and a safe following distance. City streets may have frequent traffic signals and intersections, requiring drivers to frequently change speeds and pay attention to pedestrians and other vehicles around them. Country roads may be narrow and uneven, requiring drivers to have higher concentration and control skills. By obtaining data on these different road types, the present invention can provide personalized driving suggestions and real-time road condition warnings based on specific circumstances.

[0101] As an optional implementation, step S140 of the present invention obtains traffic condition data:

[0102] Traffic conditions directly affect the smoothness and safety of driving. The present invention obtains a variety of traffic condition data including traffic congestion, smooth traffic and traffic accident scenes. Traffic congestion may lead to slower driving speed and increased driving pressure, requiring drivers to adjust their driving strategies to avoid unnecessary delays. Smooth traffic provides a smooth driving experience, and drivers can more easily control the vehicle and respond to changes in road conditions. Traffic accident scene data helps drivers avoid dangerous areas and choose safe routes. By comprehensively analyzing these traffic condition data, the present invention can provide drivers with real-time traffic and road condition updates to support intelligent driving decisions and efficient travel planning.

[0103] As an optional implementation, step S150 of the present invention integrates travel environment data:

[0104] The present invention integrates and analyzes the detailed travel environment data obtained from each step from S110 to S140. The integrated data provides a comprehensive and detailed driving environment information model, reflecting the comprehensive impact of multiple factors in the actual driving process. This comprehensive analysis can not only help the driver make more informed driving decisions, but also support the intelligent driving system to achieve more efficient and safer autonomous driving.

[0105] Through the above steps, the embodiment of the present invention provides a comprehensive and highly accurate method for acquiring travel environment data, which provides strong support for the development of driving safety and intelligent traffic management systems.

[0106] refer to Figure 3 The embodiment of the present invention discloses step S100 of acquiring travel environment data, including the following steps:

[0107] S160, using a vehicle-mounted camera to record the travel environment to obtain a recorded video;

[0108] S170: Pre-process the recorded video to obtain travel environment data.

[0109] In order to fully capture and analyze the environmental conditions during driving, the present invention first uses a vehicle-mounted camera to record the entire travel process in real time. The vehicle-mounted camera is located inside or outside the vehicle and can capture real-time images from the driver's perspective. These recorded videos not only include the real-time situation of the road ahead, but may also cover the traffic conditions around the vehicle, pedestrian activities, and other important environmental factors. Through video recording, the present invention can obtain high-definition visual information, providing a sufficient material basis for subsequent data processing and analysis.

[0110] After obtaining the recorded videos, the next key step is to preprocess them to extract valuable travel environment data. Preprocessing work includes but is not limited to the following aspects:

[0111] Image processing and analysis: Image processing technology is used to extract information about weather conditions, road types, traffic conditions, etc. from the video. For example, by analyzing the light, slippery road conditions, and road surface conditions in the video, the current weather conditions and road surface conditions can be determined.

[0112] Object detection and tracking: Using computer vision technology, various objects in the video are detected and tracked, including vehicles, pedestrians, traffic signs, etc. The location, speed and behavior of these objects can provide important real-time data support for intelligent driving decisions.

[0113] Behavior analysis and event detection: Through video analysis algorithms, the driver's behavior and various events in the surrounding environment are identified and analyzed, such as traffic accidents, pedestrians suddenly crossing the road, etc. These analysis results can help drivers predict potential dangerous situations and optimize driving strategies and route planning.

[0114] Data integration and processing: Integrate and organize various types of data extracted from the video to form a complete travel environment data set. These data sets can be structured, such as timestamps and geographic location information, or unstructured, such as image frames and video streams. Through effective data processing and management, the present invention ensures that the data provided to users and intelligent driving systems is accurate, reliable and real-time.

[0115] Through the above steps, the embodiment of the present invention not only realizes the comprehensive recording and analysis of the travel environment, but also provides accurate data support for intelligent driving systems and traffic management. This comprehensive method can not only improve driving safety, but also bring new possibilities and application scenarios to the development of the intelligent transportation field.

[0116] refer to Figure 4 The embodiment of the present invention discloses that step S200 obtains driving interior scene data according to travel environment data, including the following steps:

[0117] S210, extracting features and modeling the travel environment data to obtain road vehicle object information, pedestrian object information, and obstacle object information;

[0118] S220, the road vehicle object information includes road vehicle motion trajectory information, road vehicle speed information, and road vehicle acceleration information;

[0119] S230, pedestrian object information includes pedestrian motion trajectory information, pedestrian speed information, and pedestrian acceleration information;

[0120] S240, obstacle object information includes obstacle position information and obstacle movement information;

[0121] S250: Use road vehicle object information, pedestrian object information, and obstacle object information as driving internal scene data.

[0122] First, the present invention conducts in-depth analysis and processing of the travel environment data obtained in step S100. The travel environment data may include videos obtained from the vehicle-mounted camera, information recorded by sensors, and other environmental perception data. In step S210, the system extracts features and models these data through advanced image processing and computer vision technology. Specifically:

[0123] Road vehicle object information: The system identifies and extracts relevant information about all vehicles on the road in the video. This includes dynamic information such as the vehicle's position, movement trajectory, speed, and acceleration. Through precise target detection and tracking algorithms, the position and status of each vehicle can be updated in real time, thereby building a detailed model of the vehicle object.

[0124] Pedestrian object information: Similarly, the system analyzes pedestrian activities in the video and extracts the movement trajectory, speed and acceleration information of each pedestrian. This data not only helps the system understand the movement patterns and behaviors of pedestrians, but also provides important clues about the intentions of pedestrians to the intelligent driving system.

[0125] Obstacle object information: The system also detects and records the location and movement information of obstacles in the video, such as roadblocks, engineering facilities, or other obstructions. This data is critical for path planning and obstacle avoidance during driving.

[0126] In step S220, the system records and analyzes the motion characteristics of vehicles on the road in detail. The motion trajectory information describes the movement path of each vehicle from one location to another, which is modeled by the change of the vehicle position over time. The speed information reflects the speed of the vehicle at a specific moment, while the acceleration information provides dynamic data on the acceleration or deceleration of the vehicle. This information not only helps to monitor traffic flow and vehicle behavior in real time, but also provides necessary input for vehicle control systems, such as collision warning and adaptive cruise control.

[0127] Step S230 focuses on recording and analyzing the movement patterns of pedestrians in the video. The movement trajectory information of pedestrians describes their movement path and position changes within a specific time period. The speed information reflects the movement rate of pedestrians at different time points, while the acceleration information provides dynamic data about the acceleration or deceleration of pedestrians. This information is particularly important for intelligent driving systems, which can help the system predict pedestrian behavior and take corresponding safety measures, such as pedestrian detection and automatic emergency braking.

[0128] In step S240, the system records and analyzes the position and movement information of various obstacles appearing in the video. Obstacles can be static, such as roadblocks or signs, or dynamic, such as pedestrians or other vehicles. By accurately identifying and tracking these obstacles, the system can promptly alert the driver or the autonomous driving system and effectively avoid potential collision risks.

[0129] By integrating and processing the data obtained in steps S210 to S240, the present invention obtains complete driving internal scene data. These data not only include detailed information of all vehicles, pedestrians and obstacles on the road, but also reflect their dynamic changes and mutual relationships. These scene data provide key real-time environmental perception capabilities for intelligent driving systems, and can support advanced driving assistance functions and autonomous driving decisions, such as traffic flow management, collision warning and path planning.

[0130] In summary, the embodiments of the present invention realize the transformation from raw travel environment data to highly refined driving internal scene data through meticulous data processing and analysis steps. These data not only improve driving safety and driving experience, but also provide strong support for the development and application of intelligent transportation systems.

[0131] refer to Figure 5 The embodiment of the present invention discloses that step S300 constructs virtual driving environment information according to travel environment data and driving internal scene data, including the following steps:

[0132] S310, obtaining interaction data according to the travel environment data and the driving internal scene data; the interaction data includes interaction data between road layout and traffic flow, and interaction data between weather conditions and traffic flow;

[0133] S320: Convert the interaction data into virtual driving environment information.

[0134] In an embodiment of the present invention, step S300 involves a process of constructing virtual driving environment information according to travel environment data and driving internal scene data, and the specific steps are as follows:

[0135] In step S310, the system uses the driving internal scene data obtained from step S200 and the travel environment data from step S100 to perform in-depth analysis and extract interactive data. The core of these data interactions includes:

[0136] Interaction data between road layout and traffic flow: The system analyzes the structure of the road layout and the distribution of traffic flow. This includes road characteristics such as road type, number of lanes, curves, intersections, as well as real-time vehicle density, speed and driving direction. With this data, the system can simulate and predict traffic flow dynamics under different road conditions, providing the driver or autonomous driving system with accurate driving environment awareness.

[0137] Interaction data between weather conditions and traffic flow: Weather conditions have a significant impact on traffic flow. For example, rainy, snowy or foggy weather may cause reduced visibility and slippery roads, thus affecting the safety and efficiency of vehicle driving. By analyzing real-time meteorological data (such as precipitation, wind speed, etc.) and traffic flow data, the system can evaluate the impact of these factors on the driving environment and generate corresponding virtual driving environment information.

[0138] In step S320, the system uses the interaction data obtained in step S310 and converts the data into complete virtual driving environment information through mathematical modeling and simulation technology. Specifically:

[0139] Road layout and traffic flow data conversion: The system uses mathematical models and computer simulation technology to convert the interaction data of road layout and traffic flow into a virtual road network and real-time traffic flow simulation. This includes generating the movement trajectory of virtual vehicles, speed changes, and control of traffic signals to simulate driving environments in different scenarios.

[0140] Weather condition and traffic flow data conversion: The system generates virtual weather impact models based on the interaction data of weather conditions and traffic flow. These models can simulate the slipperiness of roads, visibility and changes in road conditions under different weather conditions, thereby helping drivers or autonomous driving systems make safe driving decisions in a virtual environment.

[0141] In summary, through steps S310 and S320, the embodiment of the present invention effectively utilizes the travel environment data and the driving internal scene data to construct highly accurate and realistic virtual driving environment information. This information can not only provide key environmental perception capabilities for the driving assistance system and the automatic driving system, but also support safe and efficient driving decisions, and promote the further development and application of intelligent transportation technology.

[0142] refer to Figure 6 The embodiment of the present invention discloses that step S400 obtains a driving environment video stream according to virtual driving environment information, including the following steps:

[0143] S410, obtaining simulation environment construction rules;

[0144] S420, according to the simulation environment construction rules and the virtual driving environment information, use Sora technology to generate a driving environment video stream.

[0145] In an embodiment of the present invention, a method for generating a driving environment video stream according to virtual driving environment information is disclosed, which specifically includes the following steps:

[0146] S410: Acquire simulation environment construction rules. In this step, the system acquires detailed rules and requirements for defining the virtual driving environment, including the geometric structure of the environment, the layout of objects, weather conditions, etc. These rules are crucial for simulating real driving scenarios.

[0147] S420: Based on the simulation environment construction rules and virtual driving environment information, the driving environment video stream is generated using Sora technology. In this step, the system uses Sora technology to combine the previously acquired simulation rules and real-time driving environment information to generate a high-quality driving environment video stream. Sora technology may include advanced technologies such as image synthesis and real-time rendering to ensure that the generated video stream is visually realistic and meets the requirements of actual driving scenarios.

[0148] Through these steps, this method can effectively use virtual environment technology to generate video streams that are highly similar to actual driving scenes, providing strong support for driving simulation and other virtual reality applications.

[0149] refer to Figure 7 On the other hand, a simulation test system based on Sora's driving environment video stream, the simulation test system includes:

[0150] The first module is used to obtain travel environment data;

[0151] The second module is used to obtain driving internal scene data based on travel environment data;

[0152] The third module is used to construct virtual driving environment information based on travel environment data and driving internal scene data;

[0153] The fourth module is used to obtain a driving environment video stream according to the virtual driving environment information.

[0154] refer to Figure 8 , the simulation test system disclosed in the embodiment of the present invention also includes a driving environment video stream generation unit based on Sora, a video injection unit, an automatic driving control module, a simulation model and a host computer interface;

[0155] The driving environment video stream generation unit based on Sora is used to generate camera video stream as simulation data according to the simulation environment construction rules;

[0156] The driving environment video stream generation unit based on Sora transmits the camera video stream to the video injection unit through the interface;

[0157] The autonomous driving control module is used as the device under test;

[0158] The video injection unit receives the camera video stream from the driving environment video stream generation unit based on Sora, encapsulates the camera video stream and injects it into the automatic driving control module in series;

[0159] The simulation model is connected to the host computer interface via Ethernet communication;

[0160] The autonomous driving control module is connected to the simulation model via Ethernet communication;

[0161] The automatic driving control module is connected to the host computer interface via Ethernet communication.

[0162] The video injection unit disclosed in the embodiment of the present invention includes a video player and a camera module.

[0163] The automatic driving control module disclosed in the embodiment of the present invention calculates the camera video stream of the video injection unit and the input signal of the simulation model to obtain an output signal;

[0164] The host computer interface will output the signal for display.

[0165] The present invention has the following beneficial effects: testers do not have to repeatedly build multiple simulation environment scenes, but only need to maintain the simulation environment building rules and generate video streams for review. The scene richness of autonomous driving simulation tests is further verified, which can greatly reduce the test cycle and cost, improve the efficiency of the test, and increase the test coverage and test reliability.

[0166] The technical solution adopted by the present invention is:

[0167] A Sora-based driving environment video stream generation automatic driving virtual simulation test system and method, comprising a Sora-based driving environment video stream generation unit, a video injection unit, an automatic driving control module, a simulation model and a host computer interface.

[0168] Driving environment video stream generation unit based on Sora: Generates camera video stream as simulation data according to the simulation environment construction rules.

[0169] The Sora driving environment video stream generation unit is a software module, and the generated video stream is transmitted to the video injection unit through an interface.

[0170] The video injection unit receives the video from the video stream generation unit, encapsulates the Sora-based driving environment video stream and injects it serially into the automatic driving control module of the device under test.

[0171] The simulation model and the host computer interface will be connected via Ethernet communication.

[0172] The autonomous driving control module will be hard-wired to the simulation model and the host computer interface through the CAN bus, Ethernet board and SPI board.

[0173] The automatic driving control module will perform calculations based on the video stream injected by the video injection unit and the input signal of the simulation environment, and the output signal will be displayed through the host computer interface.

[0174] The automatic driving control module will display the output signal on the host computer interface according to the continuous injection of video stream and signal changes of the simulation environment.

[0175] The present invention has the following beneficial effects:

[0176] Testers can generate a large number of driving environment video streams that meet the requirements based on the existing driving environment videos. They only need to complete part of the simulation environment construction work in the early stage and maintain the simulation driving environment establishment rules. These video streams can be used for virtual simulation testing of autonomous driving, which can achieve long-term, long-mileage, and multi-scenario testing. Compared with real vehicle testing, it greatly reduces the test cycle and cost, improves test efficiency, and increases test coverage and test depth.

[0177] In order to make the explanation of the present invention more clear, the present invention is further described in detail below in conjunction with implementation examples:

[0178] Fig. 9 This is a structural diagram of an autonomous driving virtual simulation test system based on Sora's driving environment video stream for an implementation example of the present invention. Please refer to Fig. 9 .

[0179] This example provides a video stream autonomous driving virtual simulation test system based on Sora's high-precision map, which includes a Sora-based video stream generation unit 1 (a driving environment video stream generation unit based on Sora), a video injection unit 2, an autonomous driving control module 3, a simulation model 4, and a host computer interface 5. The Sora-based video stream generation unit 1 will be transmitted to the video injection unit 2 in a video format, and the video injection unit 2 will be packaged and serially injected into the autonomous driving control module 3 as the device under test. The autonomous driving control module 3 will be hard-wired with the simulation model 4 and the host computer interface 5 through the CAN bus, Ethernet board and SPI board.

[0180] The key points of the invention are: a method for generating a high-precision map video stream that meets the requirements of autonomous driving virtual simulation testing and an autonomous driving control module simulation test system using video stream injection.

[0181] In addition to the above-mentioned technical solutions, there are alternative solutions for the automatic driving control module simulation test system with video stream injection: the video injection unit 2 can be replaced by a video player 21 and a camera module 22. For specific solutions, see Fig.10 .

[0182] On the other hand, an embodiment of the present invention provides a device for generating a driving environment video stream based on Sora, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating a driving environment video stream based on Sora as described above is implemented.

[0183] The device for generating a driving environment video stream based on Sora according to an embodiment of the present invention includes a memory and a processor.

[0184] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0185] The memory may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage. Among them, ROM can store static data or instructions required by the processor or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0186] The memory stores executable codes, and when the executable codes are processed by the processor, the processor can execute part or all of the methods described above.

[0187] On the other hand, an embodiment of the present invention provides a vehicle, the vehicle including a device for generating a driving environment video stream based on Sora as described above.

[0188] The vehicle of the embodiment of the present invention includes the electric drive assembly of the above-mentioned Sora-based driving environment video stream generation device. Specifically, the vehicle can be a private car, such as a sedan, SUV, MPV or pickup truck. The vehicle can also be an operating vehicle, such as a van, bus, small truck or large trailer. The vehicle can be a gasoline vehicle or a new energy vehicle. When the vehicle is a new energy vehicle, it can be a hybrid vehicle or a pure electric vehicle.

[0189] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for generating a driving environment video stream based on Sora, characterized in that: The method comprises the following steps: Obtain travel environment data; Obtaining driving interior scene data according to the travel environment data; Constructing virtual driving environment information according to the travel environment data and the driving internal scene data; According to the virtual driving environment information, a driving environment video stream is generated using Sora technology.

2. The method according to claim 1, characterized in that The step of obtaining the travel environment data comprises the following steps: Acquire weather condition data; the weather condition data includes sunny day information, rainy day information, and foggy day information; Acquire time period data; the time period data includes daytime information and nighttime information; Acquire road type data; the road type data includes highway information, city street information, and rural road information; Acquiring traffic condition data; the traffic condition data includes traffic congestion information, traffic flow information, and traffic accident scene information; The weather condition data, the time period data, the road type data, and the traffic condition data are used as travel environment data.

3. The method according to claim 1, characterized in that: The step of obtaining the travel environment data comprises the following steps: Use the vehicle-mounted camera to record the travel environment and obtain a recorded video; The recorded video is preprocessed to obtain travel environment data.

4. The method according to claim 1, characterized in that: The step of obtaining driving interior scene data according to the travel environment data comprises the following steps: Extracting features and modeling the travel environment data to obtain road vehicle object information, pedestrian object information, and obstacle object information; The road vehicle object information includes road vehicle motion trajectory information, road vehicle speed information, and road vehicle acceleration information; The pedestrian object information includes pedestrian motion trajectory information, pedestrian speed information, and pedestrian acceleration information; The obstacle object information includes obstacle position information and obstacle movement information; The road vehicle object information, the pedestrian object information, and the obstacle object information are used as driving internal scene data.

5. The method according to claim 1, characterized in that: The step of constructing virtual driving environment information according to the travel environment data and the driving internal scene data comprises the following steps: Obtaining interaction data according to the travel environment data and the driving internal scene data; the interaction data includes interaction data between road layout and traffic flow, and interaction data between weather conditions and traffic flow; The interaction data is converted into virtual driving environment information.

6. The method according to claim 1, characterized in that The step of obtaining a driving environment video stream according to the virtual driving environment information comprises the following steps: Get the simulation environment construction rules; According to the simulation environment building rules and the virtual driving environment information, the driving environment video stream is generated using Sora technology.

7. A driving environment video stream simulation test system based on Sora, characterized in that: The simulation test system comprises: The first module is used to obtain travel environment data; The second module is used to obtain driving internal scene data according to the travel environment data; A third module is used to construct virtual driving environment information according to the travel environment data and the driving internal scene data; The fourth module is used to obtain a driving environment video stream according to the virtual driving environment information.

8. The simulation test system according to claim 7, characterized in that:The simulation test system also includes a driving environment video stream generation unit based on Sora, a video injection unit, an automatic driving control module, a simulation model and a host computer interface; The driving environment video stream generation unit based on Sora is used to generate a camera video stream as simulation data according to the simulation environment construction rules; The Sora-based driving environment video stream generation unit transmits the camera video stream to the video injection unit through an interface; The autonomous driving control module serves as the device under test; The video injection unit receives the camera video stream from the Sora-based driving environment video stream generation unit, encapsulates the camera video stream and injects it into the automatic driving control module in series; The simulation model is connected to the host computer interface via Ethernet communication; The automatic driving control module is connected to the simulation model via Ethernet communication; The automatic driving control module is connected to the host computer interface via Ethernet communication.

9. The simulation test system according to claim 8, characterized in that: The video injection unit includes a video player and a camera module.

10. The simulation test system according to claim 8, characterized in that: The automatic driving control module calculates the camera video stream of the video injection unit and the input signal of the simulation model to obtain an output signal; The host computer interface displays the output signal.

Citation Information

Cited By

  • Vehicle testing method and device, processor, vehicle and storage medium

    CN120741005A