A sytem and method for scenario generation for autonomous vehicles
Patent Information
- Application Number
- IN202511024182
- Authority / Receiving Office
- IN · IN
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2025-03-18
- Publication Date
- 2026-09-23
- Estimated Expiration
- 2045-03-18
AI Technical Summary
Current methods for generating diverse scenarios for autonomous vehicle (AV) and advanced driver-assistance systems (ADAS) testing are expensive, laborious, and time-consuming, requiring manual data collection and extensive human effort, especially in simulation script creation and 3D modeling.
An end-to-end system and method that generates scenarios from monocular RGB videos without camera pose information, utilizing a perception stack, trajectory reconstruction, code generation, and simulation refinement to create synthetic data, eliminating the need for expensive sensor setups.
Provides an efficient and cost-effective solution for scenario generation, ensuring accurate and complete synthetic data creation with minimal human intervention, adhering to industry standards and allowing for iterative refinement until differences fall below a specified threshold.
Abstract
Description
FIELD OF INVENTION:The present invention relates to a system and method which provides an end-to-endsolution to generate scenarios from reference monocular RGB videos and from wildvideos without camera pose information and eliminating the need for expensive sensorsetups in vehicles for data acquisition.BACKGROUND OF THE INVENTION:The following background discussion includes information that may be useful inunderstanding the present invention. It is not an admission that any of the informationprovided herein is prior art or relevant to the presently claimed invention, or that anypublication expressly or implicitly referenced is prior art.Currently, there are two processes adopted to obtain diverse scenarios for AV andADAS testing. The first process is to manually collect real-world data using a varietyof sensors attached on top of a vehicle that is operated to drive for long periods of time.The collected data may also require curation and post-processing to ensure that the dataacquired is of desired quality. Although this is a direct approach, it is expensive,laborious and time-consuming.The second process requires significant time and effort from humans. For example, togenerate a scenario in a simulation, a script writer must create a scenario script fromreference data and run it in the simulation. The script's variables-such as vehiclevelocities, positions, acceleration, and orientation-must be manually adjusted througha trial-and-error process until the desired scenario plays out as intended. Additionally,the road and lane layouts must be hardcoded into the script. A 3D modelling artist mustensure that the entities within the scenario, including the environment and vehicles aredesigned accurately to match with the reference data.The present invention provides a system and a method that provides an end-to-endpipeline to generate scenarios from reference monocular RGB videos and from wildvideos without camera pose information, thus eliminating the need for expensive sensorsetups in vehicles for data acquisition.OBJECTIVES OF THE INVENTION:The primary object of the present invention is to overcome the problem stated in theprior art.Another object of the present invention is to provide a system and a method thatprovides an end-to-end solution to generate scenarios from reference monocular RGBvideos and wild videos without camera pose information, thus eliminating the need forexpensive sensor setups in vehicles for data acquisition.SUMMARY OF THE INVENTION:The present invention provides a system for scenario generation for autonomousvehicles comprising:a) a user interface where a user uploads a monocular RGB video;b) a perception stack unit comprises plurality of components, where eachcomponent is configured to extract a specific feature such as depth,camera pose from the input monocular RGB video, thereafter thecomponents run perception tasks such as semantic segmentation,localization, object tracking, lane detection etc to form a scenario;c) a trajectory reconstruction unit receives data from the perception stackunit is configured to reconstructs trajectories from a beginning to anend of the scenario;d) a code generator unit uses the trajectories data from the trajectoryreconstruction unit to construct OD and OS scripts and ensuring therequired standards and syntaxes are followed; ande) a simulation refinement unit receives the OD and OS scripts from thecode generator and runs a simulation, where during the simulation, oneor more sensors are attached to an ego vehicle, capturing the scenariofrom their perspectives and generating a synthetic data;wherein the generated synthetic data given to the perception stack unit toextract a specific feature such as depth, camera pose from the syntheticdata and comparing the data from the original simulation by the simulationrefinement module, where the total difference is treated as an error andcorrected directly in the OS and OD files by the simulation refinementmodule.In an embodiment, the supplementary data such as an IMU sensor data, a depth cameradata if available to the user and its metadata such as a camera intrinsics, a focal lengthis provided through the user interface.In an embodiment, the components share an internal representation but predict only asingle feature type from the monocular RGB video.In an embodiment, the components are configured to use the supplementary data topredict a more accurate output.In an embodiment, the simulation, one or more sensors such as cameras, depth cameras,IMU sensors are attached to the ego vehicle, capturing the scenario from theirperspectives and generating a synthetic data. Further, the sensors that record data insimulation can be of any type, physical or virtual and can be programmed in simulationas per the need.In an embodiment, the corrected OS and OD files are then replayed in the simulation,and this process is repeated until the difference falls below a specified threshold.In an embodiment, the output from each the components are stored as an intermediateoutput in a staging database and may be stored in different formats.In an embodiment, the intermediate outputs from the staging database, are stored in alogs database where the logs database is configured to provide option debugging,evaluation and fine-tuning of scenario.The present invention provides a method for scenario generation for autonomousvehicles comprising steps of:a) providing a monocular RGB video through a user interface by a user;b) extracting a specific feature such as depth, camera pose from the inputmonocular RGB video by a perception stack unit, thereafter thecomponents run perception tasks such as semantic segmentation,localization, object tracking, lane detection etc to form a scenario;c) receiving scenario data from the perception stack unit by a trajectoryreconstruction unit, where the trajectory reconstruction unitreconstructs trajectories from a beginning to an end from the scenariodata;d) constructing OD and OS scripts by a code generator unit based on thetrajectories data it received from the trajectory reconstruction; ande) running a simulation by a simulation refinement unit based on the ODand OS scripts it receives from the code generator, where during thesimulation, one or more sensors are attached to an ego vehicle,capturing the scenario from their perspectives and generating asynthetic data;wherein the generated synthetic data given to the perception stack unit toextract a specific feature such as depth, camera pose from the syntheticdata and comparing the data from the original simulation by the simulationrefinement module, where the total difference is treated as an error andcorrected directly in the OS and OD files by the simulation refinementmodule.DETAILED DESCRIPTION OF DRAWINGS:To further clarify advantages and features of the present invention, a more particulardescription of the invention will be rendered by reference to specific embodimentsthereof, which is illustrated in the appended drawings. It is appreciated that thesedrawings depict only typical embodiments of the invention and are therefore not to beconsidered limiting of their scope. The invention will be described and explained withadditional specificity and detail with the accompanying drawings in which:Fig. 1: illustrates the system and method of the present invention.DETAILED DESCRIPTION:For the purpose of promoting an understanding of the principles of the invention,reference will now be made to the embodiment illustrated in the drawings and specificlanguage will be used to describe the same. It will nevertheless be understood that nolimitation of the scope of the invention is thereby intended, such alterations and furthermodifications in the illustrated system, and such further applications of the principlesof the invention as illustrated therein being contemplated as would normally occur toone skilled in the art to which the invention relates.It will be understood by those skilled in the art that the foregoing general descriptionand the following detailed description are exemplary and explanatory of the inventionand are not intended to be restrictive thereof.The terms "comprises", "comprising", "includes", or any other variations thereof, areintended to cover a non-exclusive inclusion, such that a setup, device or method thatcomprises a list of components or steps does not include only those components or stepsbut may include other components or steps not expressly listed or inherent to such setupor device or method. In other words, one or more elements in a system or apparatusproceeded by "comprises... a" does not, without more constraints, preclude theexistence of other elements or additional elements in the system or method.The present invention discloses a provide a system and a method that provides an end20to-end solution to generate scenarios from reference monocular RGB videos.In an embodiment, the present invention system for scenario generation for autonomousvehicles comprising:a) a user interface where a user uploads a monocular RGB video;b) a perception stack unit comprises plurality of components, where eachcomponent is configured to extract a specific feature such as depth,camera pose from the input monocular RGB video, thereafter thecomponents run perception tasks such as semantic segmentation,localization, object tracking, lane detection etc to form a scenario;c) a trajectory reconstruction unit receives data from the perception stackunit is configured to reconstructs trajectories from a beginning to anend of the scenario;d) a code generator unit uses the trajectories data from the trajectoryreconstruction unit to construct OD and OS scripts and ensuring therequired standards and syntaxes are followed; ande) a simulation refinement unit receives the OD and OS scripts from thecode generator and runs a simulation, where during the simulation, oneor more sensors are attached to an ego vehicle, capturing the scenariofrom their perspectives and generating a synthetic data;wherein the generated synthetic data given to the perception stack unit toextract a specific feature such as depth, camera pose from the syntheticdata and comparing the data from the original simulation by the simulationrefinement module, where the total difference is treated as an error andcorrected directly in the OS and OD files by the simulation refinementmodule.In an embodiment, the supplementary data such as an IMU sensor data, a depth cameradata if available to the user and its metadata such as a camera intrinsics, a focal lengthis provided through the user interface.In an embodiment, the components share an internal representation but predict only asingle feature type from the monocular RGB video.In an embodiment, the components are configured to use the supplementary data topredict a more accurate output.In an embodiment, the simulation, one or more sensors such as cameras, depth cameras,IMU sensors are attached to the ego vehicle, capturing the scenario from theirperspectives and generating a synthetic data. Further, the sensors that record data insimulation can be of any type, physical or virtual and can be programmed in simulationas per the need.In an embodiment, the corrected OS and OD files are then replayed in the simulation,and this process is repeated until the difference falls below a specified threshold.In an embodiment, the output from each the components are stored as an intermediateoutput in a staging database and may be stored in different formats.In an embodiment, the intermediate outputs from the staging database, are stored in alogs database where the logs database is configured to provide option debugging,evaluation and fine-tuning of scenario.In an embodiment, the present invention provides a method for scenario generation forautonomous vehicles comprising steps of:a) providing a monocular RGB video through a user interface by a user;b) extracting a specific feature such as depth, camera pose from the inputmonocular RGB video by a perception stack unit, thereafter thecomponents run perception tasks such as semantic segmentation,localization, object tracking, lane detection etc to form a scenario;c) receiving scenario data from the perception stack unit by a trajectoryreconstruction unit, where the trajectory reconstruction unitreconstructs trajectories from a beginning to an end from the scenariodata;d) constructing OD and OS scripts by a code generator unit based on thetrajectories data it received from the trajectory reconstruction; ande) running a simulation by a simulation refinement unit based on the ODand OS scripts it receives from the code generator, where during thesimulation, one or more sensors are attached to an ego vehicle,capturing the scenario from their perspectives and generating asynthetic data;wherein the generated synthetic data given to the perception stack unit toextract a specific feature such as depth, camera pose from the syntheticdata and comparing the data from the original simulation by the simulationrefinement module, where the total difference is treated as an error andcorrected directly in the OS and OD files by the simulation refinementmodule.As shown in the figure 1, the present invention provides a system and method whichutilizes a deep learning-based perception stack to extract scenario-specific informationfrom a monocular RGB video and generate scenario files that can be played in asimulation. The process flow and its sequence are shown in figure 1, with the numberingindicating each step.In an embodiment, the user primarily uploads a RGB video, supplementary data (likeIMU sensor data, depth camera data etc., if available) and its metadata (for e.g. cameraintrinsics, focal length) through an interface or an API call. Providing supplementarydata is optional however in some case meta data can be also optional it depends uponcamera configuration with which video was taken can be of any configuration, type etc.Further, the uniqueness of the pipeline is to function without it, which has not been doneyet by any. It may be noted that the largest repositories of vehicle-bound RGB videoscome without supplementary data.In an embodiment, the perception stack unit comprises a set of components, each ofwhich is designed to extract a specific feature (like depth, camera pose etc.) from aninput RGB video. The components then run perception tasks such as semanticsegmentation, localization, object tracking, lane detection etc. Each of the componentsmay share internal representations but predict only a single feature type from the video.Additionally, the components are designed to use supplementary information to predictmore accurate outputs if available.In an embodiment, the final outputs from each model / component are stored asintermediate outputs in the staging database and may be stored in different formats. Forinstance, the metric depth estimation model generates per-frame depth maps with pixelvalues ranging from 0 to 200 meters, representing real-world distances from the camerato objects corresponding to the pixels. These maps are stored as 16-bit image files inthe png format. In contrast, the object tracking model produces a text file containingobject IDs and image position coordinates for each frame.Further, the intermediate outputs from the staging database, complete trajectories andsimulator log files are stored in a logs database. This database may be used laterdebugging, evaluation and fine-tuning of parameters throughout the entire architecture.In an embodiment, the trajectory reconstruction module / unit makes vehicle trajectoriesform the core of a scenario. However, the trajectory data generated by the perceptionstack-part of the intermediate outputs-is often noisy and incomplete, as vehiclepositions and orientations are computed frame by frame. To ensure completeness,temporal consistency, and adherence to vehicle kinematics, the trajectory reconstructionmodule reconstructs trajectories from the beginning to the end of the scenario for allvehicles. The resulting outputs are fully reconstructed trajectories containing essentialinformation such as ego velocities, lane offset values and ego-relative positions.In an embodiment, the code generator unit uses the complete trajectories andintermediate outputs from the staging data base to construct OD and OS scripts,ensuring the required standards and syntaxes are followed. In addition to these scripts,miscellaneous data like vehicle models, environment maps and other simulationparameters are generated.In an embodiment, the simulator and simulation refinement module / unit performsimulation and refinement process. due to the probabilistic nature of deep learningmodels, the generated trajectories may not match with the ones observed from the videoand may need validation and refinement. The compiled OD and OS scripts, along withmiscellaneous data and user settings, are passed as input to the simulation engine, whichruns the simulation. During the simulation, one or more sensors (cameras, depthcameras, IMU sensors etc.) are attached to the ego vehicle, capturing the scenario fromtheir perspectives.Further, the synthetic data generated is then processed in the perception stack. Theintermediate outputs corresponding to these synthetically generated data are comparedwith those from the original simulation in the simulation refinement module. The totaldifference is treated as an error and corrected directly in the OS and OD files. Thecorrected OS and OD files are then replayed in the simulation, and this process isrepeated until the difference falls below a specified threshold.Further, Once the OS and OD files are refined, the data is made available to the user viaAPI or a user interface. The user may then choose to run the scenario simulation withthe refined OS and OD files. The OS and OD file format is the current industry standardfor sharing scenarios, but the method of the present invention work for any structuredformat.The foregoing description of the invention has been set merely to illustrate the inventionand is not intended to be limiting. Since modifications of the disclosed embodimentsincorporating the substance of the invention may occur to person skilled in the art, theinvention should be construed to include everything within the scope of the invention.
Claims
1. A system for scenario generation for autonomous vehicles comprising: a) a user interface where a user uploads a monocular RGB video; b) a perception stack unit comprises plurality of components, where each component is configured to extract a specific feature such as depth, camera pose from the input monocular RGB video, thereafter the components run perception tasks such as semantic segmentation, localization, object tracking, lane detection etc to form a scenario; c) a trajectory reconstruction unit receives data from the perception stack unit is configured to reconstructs trajectories from a beginning to an end of the scenario; d) a code generator unit uses the trajectories data from the trajectory reconstruction unit to construct OD and OS scripts and ensuring the required standards and syntaxes are followed; and e) a simulation refinement unit receives the OD and OS scripts from the code generator and runs a simulation, where during the simulation, one or more sensors are attached to an ego vehicle, capturing the scenario from their perspectives and generating a synthetic data; wherein the generated synthetic data given to the perception stack unit to extract a specific feature such as depth, camera pose from the synthetic data and comparing the data from the original simulation by the simulation refinement module, where the total difference is treated as an error and corrected directly in the OS and OD files by the simulation refinement module.
2. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein a supplementary data such as an IMU sensor data, a depth camera data if available to the user and its metadata such as a camera intrinsics, a focal length is provided through the user interface.
3. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein each of the components share an internal representation but predict only a single feature type from the monocular RGB video.
4. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein each of the components is configured to use the supplementary data to predict a more accurate output, where supplementary data includes alternate view videos, such as stereo / depth (two-view videos) or multiple-view (360-degree videos).
5. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein during the simulation, one or more sensors such as cameras, depth cameras, IMU sensors are attached to the ego vehicle, capturing the scenario from their perspectives and generating a synthetic data.
6. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein the corrected OS and OD files are then replayed in the simulation, and this process is repeated until the difference falls below a specified threshold.
7. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein an output from each the components are stored as an intermediate output in a staging database and may be stored in different formats.
8. The system for scenario generation for autonomous vehicles as claimed in claim 1, wherein the intermediate outputs from the staging database, are stored in a logs database where the logs database is configured to provide option debugging, evaluation and fine-tuning of scenario.
9. A method for scenario generation for autonomous vehicles comprising steps of: a) providing a monocular RGB video through a user interface by a user; b) extracting a specific feature such as depth, camera pose from the input monocular RGB video by a perception stack unit, thereafter the components run perception tasks such as semantic segmentation, localization, object tracking, lane detection etc to form a scenario; c) receiving scenario data from the perception stack unit by a trajectory reconstruction unit, where the trajectory reconstruction unit reconstructs trajectories from a beginning to an end from the scenario data; d) constructing OD and OS scripts by a code generator unit based on the trajectories data it received from the trajectory reconstruction; and e) running a simulation by a simulation refinement unit based on the OD and OS scripts it receives from the code generator, where during the simulation, one or more sensors are attached to an ego vehicle, capturing the scenario from their perspectives and generating a synthetic data; wherein the generated synthetic data given to the perception stack unit to extract a specific feature such as depth, camera pose from the synthetic data and comparing the data from the original simulation by the simulation refinement module, where the total difference is treated as an error and corrected directly in the OS and OD files by the simulation refinement module.
