Method, device and equipment for controlling simulation and medium

Abstract

The embodiment of the invention relates to a method and device for controlling simulation, equipment and a medium. The method includes acquiring a pixel-level simulation image, wherein the pixel-level simulation image is generated based on a test case and simulates a driving environment of the vehicle during driving. The method further includes acquiring simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on the test case and simulate a driving state of the vehicle during driving. The method further includes controlling a first speed at which the pixel-level simulation image is generated, a second speed at which the simulation parameters are generated, and a third speed at which the pixel-level simulation image and the simulation parameters are consumed such that the generated first and second speeds match the consumed third speed. In this way, simulation details can be enriched, and the driving environment and performance of the vehicle can be truly simulated. Meanwhile, the balance between the generation and consumption of the simulation data is realized, and the problem of storage resource occupation caused by large data volume is solved.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of simulation, and in particular to methods, devices, apparatuses, and media for controlling simulation. Background Art

[0002] Vehicle driving simulation refers to the use of computer technology, virtual reality technology and other means to simulate various environments and parameters of the vehicle in the actual driving process, such as roads, traffic signals, other vehicles and pedestrians, as well as the vehicle's response and behavior, so as to train, test, and further verify and optimize algorithm-based systems such as advanced driver assistance systems (ADAS) and autonomous driving systems (ADS).

[0003] By simulating various complex road conditions and traffic situations in a virtual environment, various functions of the vehicle can be tested, and potential safety hazards can be discovered to ensure the safety of the vehicle. The research and development of vehicle-related products requires a lot of testing. If it relies on real people to conduct it on actual roads, it will take a lot of time and manpower costs, resulting in redundant and long development cycles and low efficiency. In addition, some long-tail scenarios and dangerous scenarios are difficult to reproduce in a real environment, resulting in an inability to guarantee their reliability. In addition, there are still many technical problems that need to be solved in the simulation of vehicle driving. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, device, and medium for controlling simulation.

[0005] According to a first aspect of the present disclosure, a method for controlling simulation is provided. The method includes acquiring a pixel-level simulation image, wherein the pixel-level simulation image is generated based on a test case and simulates the driving environment of a vehicle during driving. The method also includes acquiring simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on a test case and simulate the driving state of the vehicle during driving. The method also includes controlling a first speed for generating a pixel-level simulation image, a second speed for generating simulation parameters, and a third speed for consuming the pixel-level simulation image and the simulation parameters, so that the first speed and the second speed match the third speed.

[0006] According to a second aspect of the present disclosure, a device for controlling simulation is provided. The device includes a simulation image acquisition unit, which is configured to acquire a pixel-level simulation image, wherein the pixel-level simulation image is generated based on a test case and simulates the driving environment of a vehicle during driving, and the device also includes a simulation parameter acquisition unit, which is configured to acquire simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on a test case and simulate the driving state of the vehicle during driving. The device also includes a simulation control unit, which is configured to control a first speed for generating a pixel-level simulation image, a second speed for generating simulation parameters, and a third speed for consuming the pixel-level simulation image and the simulation parameters, so that the first speed and the second speed match the third speed.

[0007] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, causes the device to perform the steps of the method in the first aspect of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored on the computer-readable storage medium, wherein the computer-executable instructions are executed by a processor to implement the steps of the method in the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0010] Figure 1 A schematic diagram illustrating an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0011] Figure 2 A flow chart of a method for controlling simulation according to an embodiment of the present disclosure is illustrated;

[0012] Figure 3 A diagram illustrating a simulation control process according to an embodiment of the present disclosure;

[0013] Figure 4 A schematic diagram illustrating an example of a pixel-level simulation image according to an embodiment of the present disclosure;

[0014] Figure 5 A diagram illustrating a simulation data synchronization process according to an embodiment of the present disclosure;

[0015] Figure 6 A diagram illustrating a process of forming a pixel-level simulated video stream according to an embodiment of the present disclosure;

[0016] Fig. 7A A diagram illustrating a simulation control framework according to an embodiment of the present disclosure is illustrated;

[0017] Figure 7B A schematic diagram illustrating the use of simulation data according to an embodiment of the present disclosure;

[0018] Figure 8 A schematic diagram of an apparatus for controlling simulation according to an embodiment of the present disclosure is illustrated;

[0019] Fig. 9 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is illustrated.

[0020] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0022] In the description of the embodiments of the present disclosure, the term "including" and its variations should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly indicated to be different.

[0023] As mentioned above, simulation of vehicle driving is an important part of vehicle product development. By simulating the environment and parameters of the vehicle during driving, it is possible to evaluate and optimize the vehicle's performance, test and verify vehicle-related algorithms and systems, etc., thereby improving driving safety and comfort, while also shortening its development cycle and reducing costs.

[0024] Although vehicle driving simulation has been continuously developed in recent years as an indispensable part of vehicle-related product development, there are still some problems with the current simulation solutions that prevent them from fully meeting the needs of vehicle-related product development.

[0025] There are still certain differences between the synthetic data (i.e., the results of simulation) generated by traditional solutions and the original data or real data. For example, the synthetic data may have problems such as insufficient information and insufficient details, which makes it unable to fully meet the needs of practical applications. Among them, some traditional solutions can only provide object-level synthetic data, which can only schematically indicate the driving process, and cannot achieve the same or approximate results as, for example, those actually sensed by various sensors on the vehicle. In addition, the synthetic data generated simulates the real-time data or parameters sensed by various sensors on the vehicle. In other words, such synthetic data is a time series. Due to the lack of synchronization mechanism in traditional solutions, the synthetic data cannot be aligned in time. For example, the simulated speed simulates the speed of the vehicle at a certain moment, but the simulated acceleration at that moment cannot be found.

[0026] In addition, the amount of data is also a challenge for traditional solutions, which is due to the need to develop and operate complex real-car measurement systems. From the perspective of physical storage devices, it is difficult to store this amount of data for testing, etc., not to mention the burden on cache during operation. It is also necessary to consider that traditional solutions usually involve very complex system-level projects with huge consumption in hardware infrastructure or software development, which limits their ability to develop advanced algorithms.

[0027] At least to solve the above and other potential problems, an embodiment of the present disclosure provides a scheme for controlling simulation. The scheme includes obtaining a pixel-level simulation image, wherein the pixel-level simulation image is generated based on a test case and simulates the driving environment of the vehicle during driving. The scheme also includes obtaining simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on the test case and simulate the driving state of the vehicle during driving. The scheme also includes controlling a first speed for generating a pixel-level simulation image, a second speed for generating simulation parameters, and a third speed for consuming pixel-level simulation images and simulation parameters, so that the first speed and the second speed match the third speed. In this way, the simulation details can be enriched and the driving environment and performance of the vehicle can be realistically simulated. At the same time, a balance between the generation and consumption of simulation data is achieved, solving the waste of storage and computing resources caused by large amounts of data. Below, the embodiments of the present disclosure will be further described in detail in conjunction with the accompanying drawings.

[0028] Figure 1 1 is a schematic diagram of an example environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 1As shown in , a test case 110 including a scenario 101 and a trajectory 102 may be transmitted to a simulator 120 (hereinafter, also referred to as a simulation engine 120 ). The simulator 120 may then feed the simulation result to the system 130 for testing the system 130 , for example.

[0029] According to an embodiment of the present disclosure, the test case 110 for simulation may be editable (e.g., by the user himself or automatically by the simulator 120), and may include a scenario 101 and a track 102. The scenario 101 may indicate the driving condition, driving conditions, etc. of the vehicle to be simulated while driving. In some embodiments, the scenario may, for example, include long-tail scenarios and dangerous scenarios such as damaged traffic lights, sudden vehicle failures, collisions with adjacent vehicles, etc. It should be understood that the embodiments of the present disclosure are not limited thereto, and may also include examples of more different scenarios. For example, a vehicle is driving on a city road, driving at night, parking in a parking lot, etc. The scenario 110 in the test case 110 may be predetermined according to specific usage needs (e.g., test purposes, etc.).

[0030] In addition, the trajectory 102 in the test case 110 may indicate the driving behavior, driving route, etc. of the vehicle to be simulated while driving. In some embodiments, the trajectory may include, for example, starting, accelerating, braking, uphill, downhill, U-turn, reversing, lane keeping, etc. It should be understood that the embodiments of the present disclosure are not limited thereto, and may also include examples of more different scenarios. For example, changing lanes, turning, following a vehicle, etc. Similarly, the trajectory 102 in the test case 110 may be predetermined according to specific usage needs (e.g., development purposes, etc.).

[0031] Alternatively or additionally, the test case 110 may include, for example, simulation time, simulation target, simulation conditions, etc., in addition to the scene 101 and the trajectory 102, to constrain or limit the simulation process according to the embodiment of the present disclosure. Among them, the simulation time may indicate the time when the simulation starts and ends, or may include information at a specific time point in the simulation process. The simulation target may indicate the object to be simulated and its attributes (such as type, size, etc.), etc. The simulation condition may indicate the restrictions in the driving process to be simulated, such as the maximum speed, maximum acceleration, etc. It should be understood that these are also only exemplary and non-restrictive examples. In addition, the test case 110 may include one or more or all of the above items, for example, the test case 110 may only include the scene 101. Moreover, the embodiments of the present disclosure do not limit the types of these items, for example, the scene 101 may be in the form of an image or video, etc.

[0032] According to an embodiment of the present disclosure, after receiving the test case 110 from a remote or local source, the simulator 120 can generate simulation data based on the test case 110. As described above, the simulator 120 can be configured to simulate the environment and state of the vehicle during driving to test and improve vehicle-related algorithms and systems. The simulator 120 according to an embodiment of the present disclosure can be a simulation component or system based on software implementation, and can be run on a device with computing power, can simulate the driving condition, driving behavior, etc. of the vehicle, and can simulate the hardware components of the vehicle without actually using them in the simulation process using the simulator 120.

[0033] According to an embodiment of the present disclosure, the system 130 may be a vehicle-related driving system, which may be trained, tested, etc. using simulation data from the simulator 120, so as to further verify and optimize its functions. Such a development process using simulation data can effectively solve the problems of difficulty in actual data collection, long-tail scenarios, and difficult reproduction of dangerous scenarios, and a large amount of simulation data can further ensure the reliability of the system 130. In some embodiments, the system 130 may include an algorithm-based driving system, such as an advanced driver assistance system ADAS, an automatic driving system ADS, etc. By using simulation data from the simulator 120, the functions of the system 130 can be further improved, thereby providing better driving performance and experience. Through the simulation space according to an embodiment of the present disclosure, the speed of generating simulation data at the simulator 120 and the speed of consuming simulation data at the system 120 can be controlled, so that the generation speed and the consumption speed match to improve the problem of storage resource occupation.

[0034] like Figure 1 As shown in , the simulator 120 can be coupled with the system 130 to perform data interaction. According to an embodiment of the present disclosure, the simulator 120 and the system 130 can be deployed locally, for example, deployed at a device with computing capabilities such as a personal computer, a smart phone, a tablet computer, a laptop computer, a server, a wearable electronic device, etc. In addition, the simulator 120 and the system 130 can also be distributedly arranged on a cloud platform, for example, the simulator 120 receives a simulation request from the system 130 via a network and sends a simulation result to it.

[0035] Combined with the above Figure 1 An example environment 100 is described in which methods and / or processes according to embodiments of the present disclosure may be implemented. Figure 2 A flowchart of a method 200 for controlling simulation according to an embodiment of the present disclosure is described. Through the method 200 for controlling simulation according to an embodiment of the present disclosure, more details can be provided in the simulated driving process, and a balance between the generation and consumption of simulation data is achieved.

[0036] At box 210, a pixel-level simulation image is obtained, wherein the pixel-level simulation image is generated based on the test case 110 and simulates the driving environment of the vehicle during driving. As described above, the test case 100 may include a scene 101 and a track 102. In some embodiments, by parsing the test case 100, the simulator 120 may generate a pixel-level simulation image based on image processing techniques such as pixelization, color matching, edge detection, etc., and with the help of neural networks and deep learning techniques. Compared with general object-level simulation images, the generated pixel-level simulation images can achieve higher realism and accuracy, while providing better visualization effects, so that the simulated image is close to or even the same as what the camera sensor on the vehicle actually senses.

[0037] At box 220, simulation parameters associated with the pixel-level simulation image are obtained, wherein the simulation parameters are generated based on the test case 110 and simulate the driving state of the vehicle during driving. In some embodiments, the simulation parameters can be generated by the simulator 120, and can include at least one of a distance simulation parameter, a direction simulation parameter, and a vehicle dynamics simulation parameter. The distance simulation parameter can simulate the distance parameters of the vehicle during driving sensed by the ultrasonic sensor on the vehicle, such as the distance to the center line of the lane, the spacing with the adjacent vehicle, the distance to the target position, etc. The direction simulation parameter can simulate the direction parameters of the vehicle during driving sensed by the inertial measurement unit on the vehicle, such as the speed, acceleration, steering angle, etc. of the vehicle. The vehicle dynamics simulation parameter can simulate the vehicle dynamics parameters of the vehicle during driving sensed by the vehicle dynamics sensor on the vehicle, such as lateral force, longitudinal force, yaw moment, etc. It should be noted that the embodiments of the present disclosure are not limited to this, and the simulation parameters can include other different simulation parameters, such as obstacle profiles, etc.

[0038] At frame 230, a first speed of generating pixel-level simulation images, a second speed of generating simulation parameters, and a third speed of consuming pixel-level simulation images and simulation parameters are controlled so that the first speed and the second speed match the third speed. Simulation data is generally used for complex systems 130, and the amount of data of simulation data is extremely large. In order not to increase the storage and cache burden, the speed of generation and consumption of simulation data is controlled, and such coordination enables a balance to be achieved between generation and consumption of simulation data. Below, the simulation control process according to an embodiment of the present disclosure will be further described in detail.

[0039] Figure 3 FIG. 3 is a diagram illustrating a simulation control process 300 according to an embodiment of the present disclosure. Figure 3As exemplarily shown in the figure, a mechanism for controlling simulation is provided between the simulator 120 and the system 130 to implement the simulation control process 300. The mechanism can be abstracted as a controller 310 for simulation, which can be a component or system for controlling simulation based on software implementation and can be run on a device with computing capabilities. Through the controller 310 for simulation according to an embodiment of the present disclosure, a dynamic balance can be achieved between the simulator 120 generating simulation data and the system 130 consuming simulation data, so as to reduce the storage and cache burden and save computing resources and transmission resources.

[0040] In order to achieve a dynamic balance between the simulator 120 generating simulation data and the system 130 consuming simulation data, the simulation data generation situation on the simulation side and the simulation data consumption situation on the system side can be determined. According to an embodiment of the present disclosure, the determination module 311 in the controller 310 for simulation can be configured to determine the difference between the sum of the first number of pixel-level simulation images generated within a predetermined time period and the second number of simulation parameters generated within a predetermined time period and the third number of pixel-level simulation images and simulation parameters consumed within a predetermined time period. The determination module 311 can respectively determine the first number of pixel-level simulation images generated within a specific time period and the second number of simulation parameters generated within the specific time period to count the simulation data generation situation on the simulation side. At the same time, the determination module 311 can consume the third number of pixel-level simulation images and simulation parameters within the specific time period to count the simulation data consumption situation on the system side. In this way, based on the difference between the first number, the second number and the third number, both sides can be accurately grasped.

[0041] According to an embodiment of the present disclosure, the adjustment module 312 in the controller 310 for simulation can be configured to adjust the first speed of generating pixel-level simulation images, the second speed of generating simulation parameters, and the third speed of consuming pixel-level simulation images and simulation parameters based on the difference between the first number and the second number and the third number. In some embodiments, based on the difference between the first number and the second number and the third number being positive, this indicates that the amount of simulation data generated on the simulation side is greater than the amount of simulation data consumed on the system side. If this continues, the generated simulation data will not be consumed in time, thereby increasing the storage and cache burden. In this case, the first speed and the second speed can be reduced to reduce the generation of simulation data, or the third speed can be increased to increase the consumption of simulation data.

[0042] In some embodiments, based on the difference between the first number, the second number and the third number being negative, this indicates that the amount of simulation data generated on the simulation side is less than the amount of simulation data consumed on the system side, which, if continued, will result in insufficient data being available on the system side. In this case, the first speed and the second speed may be increased to increase the generation of simulation data, or the third speed may be decreased to reduce the consumption of simulation data.

[0043] In some embodiments, based on the difference between the first number and the second number and the third number being zero, that is, the sum of the first number and the second number is equal to the third number, this indicates that the amount of simulation data generated on the simulation side is equal to the amount of simulation data consumed on the system side. In other words, a balance has been reached between the simulation data generation on the simulation side and the simulation data consumption on the system side, and the first speed and the second speed at this time match the third speed. Alternatively, differences within a certain range are also acceptable. For example, when the absolute value of the difference between the first number and the second number and the third number is less than a difference threshold, the situation can also be regarded as the first speed and the second speed matching the third speed at this time.

[0044] According to an embodiment of the present disclosure, the first speed and the second speed of generating simulation data may be adjusted based on the third speed of consuming simulation data. Alternatively or additionally, the third speed of consuming simulation data may be adjusted based on the first speed and the second speed of generating simulation data.

[0045] Figure 4 A schematic diagram of an example 400 of a pixel-level simulated image according to an embodiment of the present disclosure is illustrated. As exemplarily shown in example 400, the pixel-level simulated image thus simulated can be close to or even identical to what is actually sensed by a camera sensor on a vehicle. By using pixel-level simulated images, the authenticity and accuracy of the simulation results can be effectively improved, while also bringing about better visualization effects.

[0046] In some embodiments, the pixel-level simulation image is adjusted to simulate the driving environment during driving sensed by the camera sensor on the vehicle from the perspective of the main vehicle by performing image processing on the pixel-level simulation image. In this way, the simulation target can be more clearly defined, and the driving environment from the perspective of the driver driving the vehicle can be simulated. Figure 4 As shown in , example 400 simulates the driving process from the perspective of the main vehicle. In some embodiments, the image processing for the pixel-level simulation image may include camera positioning, image rotation, image cropping, or at least one of target detection and tracking. It should be understood that this is only exemplary and not restrictive, and may also include feature extraction, image segmentation, multimodal fusion, etc. Appropriate image processing means may be selected according to specific needs.

[0047] As mentioned above, the simulation parameters may include distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters. Figure 5 To describe the simulation data synchronization process according to an embodiment of the present disclosure.

[0048] Figure 5 A diagram of a simulation data synchronization process 500 according to an embodiment of the present disclosure is illustrated. Each of the simulation data may have its own timestamp to record the time of the scene or state simulated by the simulation data, etc. For example, each of the generated multiple pixel-level simulation images may have a corresponding timestamp.

[0049] According to an embodiment of the present disclosure, in order to synchronize these simulation data for subsequent system purposes, a timeline for controlling the simulation can be determined. In some embodiments, a main timeline of the simulator 120 can be determined for the simulation data synchronization process 500. The timestamps of the pixel-level simulation images, distance simulation parameters, direction simulation parameters, and vehicle dynamics simulation parameters can be determined. Based on the determined timeline, the determined timestamps of the various simulation data can be aligned in time.

[0050] exist Figure 5 In the example of , by way of example, for example, simulation data 501 is a pixel-level simulation image and has a timestamp of “1”, simulation data 502 is a distance simulation parameter and has a timestamp of “2”, simulation data 503 is a direction simulation parameter and has a timestamp of “3”, and simulation data 504 is a vehicle dynamics simulation parameter and has a timestamp of “-2”. Based on the time axis, the timestamps of simulation data 501, 502, 503, and 504 may be modified to align them in time.

[0051] As described above, system 130 can be an algorithm-based driving system. In some embodiments, consuming simulation data can include feeding pixel-level simulation images, distance simulation parameters, direction simulation parameters, and vehicle dynamics simulation parameters to the algorithm-based driving system. After receiving these simulation data, these simulation data can be used to test (or train) the algorithm-based driving system to improve or optimize its function. Here, the algorithm-based driving system can include an advanced driver assistance system (ADAS) or an automatic driving system (ADS).

[0052] According to an embodiment of the present disclosure, the distance simulation parameter, the direction simulation parameter and the vehicle dynamics simulation parameter can be converted into a format corresponding to the algorithm by performing data processing on the distance simulation parameter, the direction simulation parameter and the vehicle dynamics simulation parameter. For example, some algorithms (e.g., assisted driving (AD) algorithms) may have certain requirements on the format of the input data. In order to make the input simulation data compatible with the format requirements of the target algorithm, data processing is performed on these simulation data. In some embodiments, the data processing for the distance simulation parameter, the direction simulation parameter and the vehicle dynamics simulation parameter may include at least one of data cleaning, feature extraction, data conversion and mapping, or data standardization. It should be understood that this is only exemplary and non-restrictive, and for example, data smoothing, data interpolation, data classification, etc. may also be included. Suitable data processing means can be selected according to specific needs.

[0053] Figure 6 A diagram of a formation process 600 of a pixel-level simulation video stream 620 according to an embodiment of the present disclosure is illustrated. The generated pixel-level simulation images can form a simulation image set 610. As described above, the timestamp of each pixel-level simulation image can be determined. Thus, the pixel-level simulation images in the simulation image set 610 can be sorted in chronological order based on the timestamp of each pixel-level simulation image, and the sorted pixel-level simulation images can be converted into a pixel-level simulation video stream 620. In this way, the real-time driving state of the vehicle during driving can be simulated in the form of a video stream.

[0054] Fig. 7AThe diagram of the simulation control framework 700A according to an embodiment of the present disclosure is illustrated. On the simulation side, the test case 110 for simulation can be transmitted via the simulation graphical user interface (GUI) 710. After receiving the request from the simulation graphical user interface (GUI) 710, the simulation backend 720 can initialize the simulation. Through the analysis of the test case 110, the camera data can be rendered and generated by the (multiple) simulation graphics processing units (GPUs) 730 in the host, and obtained by the simulation engine 120 through the TCP / IP protocol. Among them, the GPU camera sensor component 750 can perform the above-mentioned adjustment of the simulation image to the simulation of the driving environment sensed by the camera sensor on the vehicle from the perspective of the main vehicle during driving, and continue to send it down via the TCP / IP protocol by the sending port 760, wherein the sending port 760 may include a Socket port. Meanwhile, the data acquisition interface 740 can acquire data from above, and can generate distance simulation parameters using the non-GPU sensor ultrasonic radar (USS) component 770, generate direction simulation parameters using the non-GPU sensor inertial measurement unit (IMU) component 780, and generate vehicle dynamics simulation parameters using the vehicle dynamics component 790. The generated simulation parameters can be processed as described above, and can be sent further down through the sending interface 7100 via the TCP / IP protocol.

[0055] On the data player side, the video stream receiving client 7110 can receive the camera data and form a video stream, wherein the camera data can be synchronized with the parameter data based on its timestamp, and the synchronization information can be sent to the simulation backend 720. After receiving the camera data and parameter data respectively, the video receiving client 7110 and the receiving client 7120 can send them to the algorithm 7160 via the virtual camera driver component 7130 and the raw data gateway 7140, wherein the sent camera data can be a synthetic video stream. After receiving the camera data and parameter data, the algorithm 7160 can run based on the algorithm GPU 7150.

[0056] Through the simulation space according to the embodiment of the present disclosure, it is possible to control the speed at which the simulation GPU 730 generates simulated camera data, control the speed at which the non-GPU sensor USS component 770, the non-GPU sensor IMU component 780, and the vehicle dynamics component 790 generate simulated parameter data, and control the speed at which the algorithm 7160 consumes these simulated camera data and parameter data. The generation speed and the consumption speed are matched to improve the problem of storage resource occupation.

[0057] Figure 7BA schematic diagram of a use example 700B of simulation data according to an embodiment of the present disclosure is illustrated. According to an embodiment of the present disclosure, simulation data (including pixel-level simulation images, simulation parameters, etc.) controlled by simulation can be fed to an algorithm-based driving system for various purposes, such as training and testing the driving system. As shown in use example 700B, the driving system can use such simulation data for road recognition, etc., to improve the performance and experience of the driving system, etc.

[0058] Figure 8 A schematic diagram of an apparatus 800 for controlling simulation according to an embodiment of the present disclosure is further shown. The apparatus 800 may include multiple units or modules for performing the following steps: Figure 2 The corresponding steps in the method 200 discussed in Figure 8 As shown in , the device 800 includes: a simulation image acquisition unit 810, configured to acquire a pixel-level simulation image, wherein the pixel-level simulation image is generated based on a test case and simulates the driving environment of a vehicle during driving; a simulation parameter acquisition unit 820, configured to acquire simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on a test case and simulate the driving state of the vehicle during driving; and a simulation control unit 830, configured to control a first speed of generating a pixel-level simulation image, a second speed of generating simulation parameters, and a third speed of consuming the pixel-level simulation image and the simulation parameters, so that the generated first speed and the second speed match the consumed third speed.

[0059] In some embodiments, the device 800 may also include an image processing unit configured to adjust the pixel-level simulation image to simulate the driving environment during driving sensed by a camera sensor on the vehicle from the perspective of the main vehicle by performing image processing on the pixel-level simulation image.

[0060] In some embodiments, the image processing for the pixel-level simulation image may include at least one of the following: camera positioning; image rotation; image cropping; or target detection and tracking.

[0061] In some embodiments, the simulation parameters may include at least one of the following: a distance simulation parameter, the distance simulation parameter simulates the distance parameter of the vehicle during driving sensed by an ultrasonic sensor on the vehicle; a direction simulation parameter, the direction simulation parameter simulates the direction parameter of the vehicle during driving sensed by an inertial measurement unit on the vehicle; and a vehicle dynamics simulation parameter, the vehicle dynamics simulation parameter simulates the vehicle dynamics parameter of the vehicle during driving sensed by a vehicle dynamics sensor on the vehicle.

[0062] In some embodiments, the device 800 may also include: a timeline determination unit, configured to determine the timeline for controlling the simulation; and a timestamp determination unit, configured to determine the timestamps of the simulation of pixel-level simulation images, distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters; and an alignment unit, respectively configured to align the timestamps of the pixel-level simulation images, distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters based on the timeline.

[0063] In some embodiments, the device 800 may also include: a feeding unit configured to feed pixel-level simulation images, distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters to the algorithm-based driving system; and a using unit configured to test the driving system using pixel-level simulation images, distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters.

[0064] In some embodiments, the driving system may include an advanced driver assistance system ADAS or an automated driving system ADS.

[0065] In some embodiments, the device 800 may also include: a data processing unit configured to convert the distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters into a format corresponding to the algorithm by performing data processing on the distance simulation parameters, direction simulation parameters and vehicle dynamics simulation parameters.

[0066] In some embodiments, data processing for distance simulation parameters, direction simulation parameters, and vehicle dynamics simulation parameters may include at least one of the following: data cleaning; feature extraction; data conversion and mapping; or data standardization.

[0067] In some embodiments, the simulation control unit 830 can be further configured to adjust the first speed, the second speed, and the third speed based on the difference between the sum of a first number of pixel-level simulation images generated within a predetermined time period and a second number of simulation parameters generated within a predetermined time period and a third number of pixel-level simulation images and simulation parameters consumed within a predetermined time period.

[0068] In some embodiments, matching the first rate and the second rate of generation with the third rate of consumption may include causing the sum of the first number and the second number to equal the third number.

[0069] In some embodiments, the test cases may be editable and may include scenarios and trajectories, and wherein: scenarios may include damaged traffic lights, sudden vehicle failures, collisions with adjacent vehicles, and trajectories may include starting, accelerating, braking, uphill, downhill, U-turns, reversing, and lane keeping.

[0070] Fig. 9 A schematic block diagram of an example device 900 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The electronic device 104 in the embodiment can be implemented by using the device 900. As shown in the figure, the device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0071] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a memory 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0072] The various processes and processing described above, such as method 200 and processes 500, 600, and 700, may be performed by processing unit 901. For example, in some embodiments, method 200 and processes 500, 600, and 700 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of method 200 and processes 500, 600, and 700 described above may be performed.

[0073] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0074] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0075] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0076] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0077] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0078] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0079] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0080] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0081] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for controlling simulation, comprising: Acquire a pixel-level simulation image, wherein the pixel-level simulation image is generated based on the test case and simulates a driving environment of the vehicle during driving; acquiring simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on the test case and simulate a driving state of the vehicle during driving; as well as A first speed of generating the pixel-level simulation image, a second speed of generating the simulation parameters, and a third speed of consuming the pixel-level simulation image and the simulation parameters are controlled so that the first speed of generation and the second speed of generation match the third speed of consumption.

2. The method according to claim 1, further comprising: The pixel-level simulation image is adjusted to simulate a driving environment during driving sensed by a camera sensor on the vehicle from a main vehicle perspective by performing image processing on the pixel-level simulation image.

3. The method according to claim 2, wherein the image processing for the pixel-level simulation image comprises at least one of the following: Camera positioning; Image rotation; Image cropping; or Object detection and tracking.

4. The method according to claim 2, wherein the simulation parameters include at least one of the following: A distance simulation parameter, wherein the distance simulation parameter simulates a distance parameter of the vehicle during driving sensed by an ultrasonic sensor on the vehicle; Direction simulation parameters, wherein the direction simulation parameters simulate direction parameters of the vehicle during driving sensed by an inertial measurement unit on the vehicle; as well as The vehicle dynamics simulation parameters simulate vehicle dynamics parameters of the vehicle during driving sensed by a vehicle dynamics sensor on the vehicle.

5. The method according to claim 4, further comprising: Determine the timeline used to control the simulation; Determining timestamps for simulating the pixel-level simulation image, the distance simulation parameter, the direction simulation parameter, and the vehicle dynamics simulation parameter; as well as Based on the time axis, the timestamps of the pixel-level simulation image, the distance simulation parameter, the direction simulation parameter, and the vehicle dynamics simulation parameter are aligned.

6. The method of claim 4, wherein consuming the pixel-level simulated image and the simulated parameters comprises: feeding the pixel-level simulation image, the distance simulation parameter, the direction simulation parameter, and the vehicle dynamics simulation parameter to an algorithm-based driving system; as well as The driving system is tested using the pixel-level simulation image, the distance simulation parameter, the direction simulation parameter, and the vehicle dynamics simulation parameter.

7. The method according to claim 6, wherein: The driving system includes an advanced driving assistance system ADAS or an automatic driving system ADS.

8. The method according to claim 6, further comprising: The distance simulation parameter, the direction simulation parameter, and the vehicle dynamics simulation parameter are converted into a format corresponding to the algorithm by performing data processing on the distance simulation parameter, the direction simulation parameter, and the vehicle dynamics simulation parameter.

9. The method according to claim 8, wherein the data processing for the distance simulation parameter, the direction simulation parameter and the vehicle dynamics simulation parameter comprises at least one of the following: Data cleaning; Feature extraction; Data transformation and mapping; or Data standardization.

10. The method of claim 1, wherein controlling the first speed, the second speed, and the third speed comprises: The first speed, the second speed, and the third speed are adjusted based on a difference between a sum of a first number of the pixel-level simulation images generated within a predetermined time period and a second number of the simulation parameters generated within the predetermined time period and a third number of the pixel-level simulation images and the simulation parameters consumed within the predetermined time period.

11. The method of claim 10, wherein matching the first and second generated speeds with the third consumed speed comprises: The generated first speed and the second speed are adjusted based on the consumed third speed.

12. The method of claim 1, wherein the test case is editable and includes a scenario and a trajectory, and wherein: The scenarios include traffic light damage, sudden failure of the vehicle, collision of adjacent vehicles, and The trajectory includes starting, accelerating, braking, going uphill, going downhill, turning around, reversing, and keeping lane.

13. An apparatus for collecting training data, comprising: a simulation image acquisition unit configured to acquire a pixel-level simulation image, wherein the pixel-level simulation image is generated based on a test case and simulates a driving environment of a vehicle during driving; a simulation parameter acquisition unit configured to acquire simulation parameters associated with the pixel-level simulation image, wherein the simulation parameters are generated based on the test case and simulate a driving state of the vehicle during driving; as well as A simulation control unit is configured to control a first speed of generating the pixel-level simulation image, a second speed of generating the simulation parameters, and a third speed of consuming the pixel-level simulation image and the simulation parameters, so that the first speed and the second speed of generation match the third speed of consumption.

14. An electronic device, comprising: at least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the apparatus to perform the method according to any one of claims 1-12.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 12.

Images