Autonomous driving subjective experience evaluation method, system, terminal and medium

Abstract

The present invention provides a method and system for evaluating the subjective experience of autonomous driving, wherein the evaluation method includes: constructing a test scene including a test vehicle and its surrounding traffic environment, obtaining multi-view video around the vehicle and vehicle dynamic information data; splicing the multi-view video around the vehicle into 360-degree video data; synchronizing the vehicle dynamic information data with the 360-degree video data on the time axis, using VR tools to load the simulated video of the vehicle's surrounding environment, using a multi-axis simulator platform to load the vehicle motion simulation data, simulating the vehicle's surrounding environment information and the vehicle's six-axis posture changes; repeating the above steps to complete the subjective evaluation of autonomous driving. The present invention can greatly reduce the understanding deviation in the subjective experience demand / problem communication, effectively shorten the research and development cycle of creating a more personalized autonomous driving function; no actual test site is required, which is more convenient; different vehicle characteristics can be simulated.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving subjective experience evaluation method, system, terminal and medium. Background Art

[0002] Virtual Reality (VR) technology is gaining more and more attention in the market. It can create an immersive effect by using special display glasses to construct a three-dimensional world. VR technology is mainly a display technology. It uses binocular glasses to construct a virtual scene, combined with the changes in the state of the human head, to change the perspective in the virtual scene and create an immersive effect. As VR technology matures, more interactions are also introduced into VR technology solutions. At present, VR technology combines various input settings to build virtual characters with the main perspective in a virtual environment to make different actions to achieve interaction. It is currently mainly used in games and special technology training.

[0003] In automobile manufacturing and racing events, driving simulators are used to perform axle dynamic simulations to simulate the dynamic response of the vehicle on different axes in order to accelerate automobile research and development or improve the driving level of the driver.

[0004] The autonomous driving function requires controlling the vehicle to ensure not only the safety of the function, but also the comfort of the driver. In the current autonomous driving research and development, the evaluation of subjective experience is mainly done through experience in real scenes on real vehicles. This method is time-consuming and cannot repeat the quantitative analysis of the scenes that need to be evaluated. However, if the above-mentioned VR technology or driving simulation technology is applied to the subjective experience evaluation of autonomous driving, the following technical problems will exist:

[0005] Pure VR cannot reflect the physical feedback of the function; a simple vehicle driving simulator obtains the driver's input and simulates the dynamic information of the vehicle. The data of the two cannot be synchronized, and cannot directly reflect the dynamic response of the vehicle in a specific functional scenario, and cannot complete the subjective evaluation in this scenario.

[0006] After searching, we found:

[0007] The Chinese invention patent application "Intelligent Driving Evaluation System and Evaluation Method" with application number 201910482252.5 includes: a positioning device, a pedestrian simulation device, a vehicle simulation device, a test site and a controller, wherein the positioning device is installed on the test vehicle and is used to feed back the position of the test vehicle to the controller in real time, the pedestrian simulation device is used to receive and execute instructions from the controller to simulate pedestrian movement, the vehicle simulation device is used to receive and execute instructions from the controller to simulate vehicle movement, and a traffic light is set on the test site. The traffic light receives and executes instructions from the controller to change the light color of the light, and the controller includes a signal generator for sending control instructions to the pedestrian simulation device, the vehicle simulation device and the traffic light in the test site. The evaluation system and evaluation method need to be deployed in an actual test site and need to have sufficient test space; they need to rely on actual vehicles, and each deployment requires switching an actual vehicle.

[0008] The Chinese invention patent application with application number 201910684034.X, entitled "A Traffic Scene Simulation Experiment Platform and Method Based on Virtual Reality", comprises: the simulation scene control module (1), the human-computer interaction mechanism decision module (2), the network communication simulation module (3) and the control input interface (4), wherein the simulation scene control module (1), the network communication simulation module (3) and the control input interface (4) are connected to the human-computer interaction mechanism decision module (2), and the human-computer interaction mechanism decision module (2) has pre-packaged virtual driver characteristics; the human-computer interaction mechanism decision module (2) simulates the behavioral decisions of real drivers or virtual drivers and other traffic elements for various scenarios, and performs virtual simulation of human-vehicle-road joint operation, and controls the virtual vehicle to accelerate, decelerate, brake, overtake and / or change lanes in a virtual environment based on the dynamic properties of the vehicle in the real environment. The simulation experiment platform and method, through constructing a full virtual scene, conducts a simulation experiment, mainly using the simulation scene to simulate the virtual scene of the main vehicle and other traffic vehicles, traffic facilities and the driver's interaction; using pure simulation, the vehicle dynamic information cannot be simulated, and the driver can only use the steering wheel and pedals as input. Summary of the invention

[0009] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method, system, terminal and medium for evaluating the subjective experience of autonomous driving.

[0010] According to one aspect of the present invention, a method for evaluating subjective experience of autonomous driving is provided, comprising:

[0011] Constructing a test scenario including a test vehicle and its surrounding traffic environment, and acquiring multi-view videos around the vehicle and vehicle dynamic information data based on the test scenario;

[0012] Splicing the multi-view videos around the vehicle into 360-degree video data;

[0013] The vehicle dynamic information data is synchronized with the 360-degree video data on a time axis to obtain a vehicle surrounding environment simulation video and vehicle motion simulation data at the same time; the vehicle surrounding environment simulation video is loaded using a VR tool, and the vehicle motion simulation data is loaded using a multi-axis simulator platform to simulate the vehicle surrounding environment information and the vehicle six-axis posture changes;

[0014] Repeat the above steps to complete the subjective evaluation of autonomous driving.

[0015] Preferably, the test scenario includes: a real road scenario and a virtual simulation scenario; wherein:

[0016] For the real road scene, construct the test scene, including:

[0017] Based on the ISO vehicle coordinate system, multiple cameras are used to synchronously capture multi-angle video images along the xy axis plane to obtain the surrounding vehicle video;

[0018] In the ISO vehicle coordinate system, the vehicle dynamic information data on the three axes of xyz is obtained through the on-board IMU sensor, including: acceleration a along the x-axis x , angle ω around the x-axis x , acceleration along the y-axis a y , angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axis y ;

[0019] Reproducing a test scene including a test vehicle and its traffic environment based on the surrounding vehicle video and the vehicle dynamic information data;

[0020] For the virtual simulation scenario, constructing the test scenario includes:

[0021] By using the virtual scene editor and manually setting the surrounding vehicle video and vehicle dynamic information data, a test scene including the current test and its traffic environment is constructed.

[0022] Preferably, stitching the multi-view videos around the vehicle into a 360-degree video includes:

[0023] The multi-view videos of the vehicle are spliced ​​together, and the features and key points in each frame of the video are locally detected using a scale-invariant feature transformation matching algorithm;

[0024] A random sampling consensus method is used to exclude the matches of adjacent videos that do not conform to geometric changes, and the transformation matrix H is calculated using the remaining matching points;

[0025] For each frame of adjacent videos to be spliced, the transformation matrix H is used to perform projection transformation, and the pixels of the overlapping parts after transformation are weighted average fused;

[0026] Through the above steps, the multi-view videos around the vehicle at the same time are stitched together to generate a continuous 360-degree video image based on the vehicle's xy plane.

[0027] Preferably, the step of synchronizing the vehicle dynamic information data with the 360-degree video data on a time axis to obtain a vehicle surrounding environment simulation video and vehicle motion simulation data at the same time includes:

[0028] The 360-degree video data and the vehicle dynamic information data are synchronized through the time axis in the vehicle operation data recorded by the real vehicle or generated by simulation, and the 360-degree video data and the vehicle dynamic information data at the same time are retrieved;

[0029] At the same time, the visual field direction of the evaluation personnel's head posture input is collected, and the viewing angle is set as yaw; through the viewing angle yaw input by the evaluation personnel, the visual range of the specified 360-degree video data in the visual field direction is loaded in real time, and the simulated video of the vehicle surrounding environment is obtained as follows:

[0030] Video o =f(timestamps,yaw,FOV)

[0031] Where:

[0032] timestamps is the time of synchronization;

[0033] Video o Simulate video of the vehicle's surroundings;

[0034] Yaw is the head posture of the evaluator, the deflection angle along the vehicle's travel direction;

[0035] FOV is the visible range of VR video;

[0036] The vehicle dynamic information data at the same time as the 360-degree video data is retrieved to obtain the vehicle motion simulation data:

[0037] Pose o =f(timestamps)

[0038] Where:

[0039] Pose o Simulation data for vehicle motion, including: acceleration a along the x-axis x , angle ω around the x-axis x , acceleration along the y-axis a y, angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axis y ;

[0040] The vehicle surrounding environment simulation video and vehicle motion simulation data obtained at the same time are expressed as:

[0041] [Video o ,Pose o ]=f(timestamps,,,pause)

[0042] Where:

[0043] Pause is an interface for pausing the 360-degree video and vehicle dynamic information data. It is used to facilitate pausing at any time to evaluate the dynamic response of the test vehicle in the current test scenario.

[0044] Preferably, the method of using a VR tool to load a simulated video of the vehicle's surrounding environment, using a multi-axis simulator platform to load vehicle motion simulation data, and simulating the vehicle's surrounding environment information and the vehicle's six-axis posture changes includes:

[0045] Output the obtained simulated video of the vehicle's surrounding environment to the VR headset of the evaluator;

[0046] Through CAN communication, the obtained vehicle motion simulation data is responded to on the multi-axis simulator platform to obtain the postures in different axes; among which:

[0047] The multi-axis simulator platform controls the thrust of the electric push rod of the multi-axis simulator to simulate the acceleration on different axes according to the postures on different axes requested by the platform; and simulates the deflection angle around different axes through the combination of different push rod strokes.

[0048] According to another aspect of the present invention, a system for evaluating subjective experience of autonomous driving is provided, comprising:

[0049] A test scenario construction module, which is used to construct a test scenario including a test vehicle and its surrounding traffic environment, and based on the test scenario, obtain multi-view videos around the vehicle and vehicle dynamic information data;

[0050] A video stitching module, which is used to stitch the multi-view videos around the vehicle into 360-degree video data;

[0051] A simulation data acquisition module, which is used to synchronize the vehicle dynamic information data with the 360-degree video data on a time axis to obtain a vehicle surrounding environment simulation video and vehicle motion simulation data at the same time;

[0052] The simulation display module utilizes the VR module and the multi-axis simulator module to load the vehicle surrounding environment simulation video and the vehicle motion simulation data respectively, observe the vehicle surrounding environment information and feel the vehicle six-axis posture changes, and complete the subjective evaluation.

[0053] Preferably, the system also includes: a data playback tool module, which is used to arrange the simulation data acquisition module, control the timeline synchronization, and be connected to the simulation display module control, and is used to control the VR module to play, pause or replay the vehicle surrounding environment simulation video and control the multi-axis simulator module to execute the vehicle motion simulation data.

[0054] According to a third aspect of the present invention, there is provided a computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any of the above-described methods or to run the above-described system when executing the program.

[0055] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the above-mentioned methods, or to run the above-mentioned system.

[0056] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0057] The autonomous driving subjective experience evaluation method, system, terminal and medium provided by the present invention provide a quantitative analysis tool for subjective evaluation, which can effectively and quantitatively and repeatedly evaluate the subjective experience in different scenarios, polish product details, and solve the problem of non-reproducibility of subjective evaluation problems in the testing process and the problem that quantitative data cannot be reproduced with actual physical sensations. It can greatly reduce the understanding deviation in the communication of subjective experience needs / problems, and effectively shorten the research and development cycle for creating more personalized autonomous driving functions.

[0058] The autonomous driving subjective experience evaluation method, system, terminal and medium provided by the present invention do not need to be deployed in an actual test site, but only need to be deployed in a laboratory. Compared with traditional autonomous driving subjective experience tests that rely on different road environments or professional test sites, they are more convenient.

[0059] The autonomous driving subjective experience evaluation method, system, terminal and medium provided by the present invention only need to build a multi-axis degree of freedom simulation platform to simulate different vehicle characteristics and further simulate different vehicles.

[0060] The autonomous driving subjective experience evaluation method, system, terminal and medium provided by the present invention focus on reconstructing data collected from real vehicles into virtual scenes, which are directly generated from the data without relying on artificially constructed virtual scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0062] Figure 1 The present invention is a flowchart of a method for evaluating the subjective experience of autonomous driving in a preferred embodiment of the present invention.

[0063] Figure 2 It is a schematic diagram of a vehicle surrounding environment simulation video obtained in a preferred embodiment of the present invention.

[0064] Figure 3 This is a working principle diagram of the autonomous driving subjective experience evaluation method in a preferred embodiment of the present invention.

[0065] Figure 4 Schematic diagram of the components of the autonomous driving subjective experience evaluation system in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0066] The following is a detailed description of the embodiments of the present invention: This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

[0067] An embodiment of the present invention provides a method for evaluating the subjective experience of autonomous driving, which reproduces the environment in which the autonomous driving vehicle is located and the dynamic response of the vehicle by post-processing the video of the environmental information around the vehicle into a VR video and presenting the dynamic information of the vehicle on a multi-degree-of-freedom vehicle dynamic simulation platform. The scene information can be constructed from a pure simulation platform or recorded from a real vehicle. By reproducing the surrounding environment and the dynamic response of the vehicle, a simulation system is established that can reproduce the data that affects the subjective experience in autonomous driving.

[0068] like Figure 1 As shown, the autonomous driving subjective experience evaluation method provided in this embodiment may include:

[0069] S1, construct a test scene including a test vehicle and its surrounding traffic environment, and based on the test scene, obtain multi-view video of the surrounding vehicle and vehicle dynamic information data;

[0070] S2, stitching the multi-view videos around the vehicle into 360-degree video data;

[0071] S3, synchronize the vehicle dynamic information data with the 360-degree video data on the time axis to obtain the vehicle surrounding environment simulation video and vehicle motion simulation data at the same time; use VR tools to load the vehicle surrounding environment simulation video, use a multi-axis simulator platform to load the vehicle motion simulation data, and simulate the vehicle surrounding environment information and the vehicle's six-axis posture changes;

[0072] Repeat the above steps to complete the subjective evaluation of autonomous driving.

[0073] The technical solutions provided by the above embodiments of the present invention are further described in detail below.

[0074] The autonomous driving subjective experience evaluation method provided in the above embodiment of the present invention comprises the following steps:

[0075] Step 1: Construct a test scenario including the test vehicle and its surrounding traffic environment, and obtain multi-view videos around the vehicle and vehicle dynamic information data based on the test scenario;

[0076] Step 2: stitch the multi-view videos around the vehicle into a 360-degree video;

[0077] Step 3: Synchronize the vehicle dynamic information data with the 360-degree video data on the time axis to obtain the vehicle surrounding environment simulation video and vehicle motion simulation data at the same time; use VR tools to load the vehicle surrounding environment simulation video, use the multi-axis simulator platform to load the vehicle motion simulation data, and simulate the vehicle surrounding environment information and the vehicle's six-axis posture changes;

[0078] Step 4: Testers use the above steps to conduct multiple evaluations and finally obtain subjective evaluation results. The subjective evaluation results can be used to guide the next step of product development.

[0079] In a preferred embodiment of step 1, the test scene includes: a real road scene and a virtual simulation scene; wherein:

[0080] Build test scenarios for real road scenarios, including:

[0081] Step 1.1.1, based on the ISO vehicle coordinate system, use multiple cameras to synchronously capture multi-angle video images along the xy axis plane to obtain a surrounding vehicle video;

[0082] Step 1.1.2, in the ISO vehicle coordinate system, obtain the vehicle dynamic information data on the three axes xyz through the on-board IMU sensor, including: acceleration a along the x-axis x , angle ω around the x-axis x , acceleration along the y-axis a y , angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axisy ;

[0083] Step 1.1.3, reproducing the test scene including the test vehicle and its traffic environment based on the surrounding vehicle video and vehicle dynamic information data;

[0084] Build test scenarios for virtual simulation scenarios, including:

[0085] Step 1.2.1, using the virtual scene editor, by manually setting the surrounding vehicle video and vehicle dynamic information data, construct a test scene including the current test and its traffic environment.

[0086] In this step, if the scene to be evaluated is generated by a real vehicle, the scene to be evaluated can be constructed using the collected vehicle data; if a virtually constructed scene needs to be evaluated, a virtual scene needs to be constructed, such as the dynamic information of the vehicle and the movement information of other traffic participants.

[0087] In a preferred embodiment of step 1, the constructed data needs to meet the following requirements:

[0088] Based on the ISO vehicle coordinate system, a 360-degree video image along the xy-axis plane (needs to be composed of data collected synchronously by multiple cameras);

[0089] The dynamic information of the vehicle, in the ISO vehicle coordinate system, dynamic information on three axes, including acceleration along the x-axis (a x ), the angle around the x-axis (ω x ), the acceleration along the y-axis (a y ), the angle around the y-axis (θ y ), the acceleration along the z-axis (a z ), the angle around the z-axis (ψ y ).

[0090] In a preferred embodiment of step 2, stitching the multi-view videos around the vehicle into 360-degree video data includes the following steps:

[0091] Step 2.1, stitch the multi-view videos of the surrounding vehicle, and use SIFT (Scale Invariant Feature Transform, scale-invariant feature transform matching algorithm) local description film to detect the features and key points in each frame of the video;

[0092] Step 2.2, RANSAC (RANdom SAmple Consensus) method is used to exclude the matches that do not conform to most geometric changes in adjacent videos, and the transformation matrix H is estimated using the remaining matching points;

[0093] Step 2.3: For each frame of the adjacent videos that needs to be spliced, the transformation matrix H is used to perform projection transformation, and the pixels of the overlapping parts after transformation are weighted average fused.

[0094] In this step, the multi-view videos of the vehicle at Monday can be spliced ​​together through the above three steps, that is, a continuous video image based on the 360-degree view of the vehicle's XY plane can be generated.

[0095] In this step, the multi-view video around the vehicle can come from data recorded by the real vehicle or vehicle operation data constructed by the virtual scene. The data may include: the absolute time corresponding to the vehicle at each moment, the vehicle speed, acceleration, etc.

[0096] In a preferred embodiment of step 3, the vehicle dynamic information data and the 360-degree video data are synchronized in time axis to obtain the vehicle surrounding environment simulation video and vehicle motion simulation data at the same time, including the following steps:

[0097] Step 3.1: Synchronize the 360-degree video data and the vehicle dynamic information data through the time axis in the vehicle operation data generated by real vehicle recording or simulation, and retrieve the 360-degree video data and the vehicle dynamic information data at the same time (timestamps);

[0098] Step 3.2: At the same time, collect the visual field direction of the evaluator's head posture input, and set the viewing angle as yaw; the input video should satisfy the following relationship. Through the yaw input by the evaluator, load the visual range of the video in the visual field direction in real time to obtain the simulated video of the vehicle's surrounding environment:

[0099] Video o =f(timestamps,yaw,FOV)

[0100] Where:

[0101] Timestamps: synchronization time;

[0102] Video o : Simulate video of the vehicle's surrounding environment;

[0103] yaw: the head posture of the evaluator, the deflection angle along the direction of vehicle travel;

[0104] FOV: VR video viewing range;

[0105] The principle of obtaining the simulated video of the vehicle's surrounding environment, such as Figure 2 As shown;

[0106] Step 3.3: Retrieve the vehicle dynamic information data at the same time as the 360-degree video data to obtain the vehicle motion simulation data:

[0107] Pose o =f(timestamps)

[0108] Where:

[0109] Pose o Simulation data for vehicle motion, including: acceleration a along the x-axis x , angle ω around the x-axis x , acceleration along the y-axis a y , angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axis y ;

[0110] The vehicle surrounding environment simulation video and vehicle motion simulation data obtained at the same time are expressed as:

[0111] [Video o ,Pose o ]=f(timestamps,,,pause)

[0112] Where:

[0113] Pause is an interface for pausing the 360-degree video and vehicle dynamic information data. It is used to facilitate pausing at any time to evaluate the dynamic response of the test vehicle in the current test scenario.

[0114] In another preferred embodiment of step 3, a VR tool is used to load a simulated video of the vehicle's surrounding environment, and a multi-axis simulator platform is used to load vehicle motion simulation data, so as to simulate the vehicle's surrounding environment information and the vehicle's six-axis posture changes, including the following steps:

[0115] Step 3.01: Output the obtained simulated video of the vehicle's surrounding environment to the VR headset of the evaluator;

[0116] Step 3.02: Through CAN communication, the obtained vehicle motion simulation data is responded to on the multi-axis simulator platform to obtain the postures on different axes; wherein:

[0117] The multi-axis simulator platform controls the electric push rod of the simulator according to the postures in different axes requested by the platform, and simulates the acceleration on different axes by controlling the thrust of the push rod; through the combination of different push rod strokes, the deflection angles around different axes are simulated.

[0118] In a preferred embodiment, the above steps further include:

[0119] Use the data playback tool to control the simultaneous playback of video and posture data. It also supports pausing data playback, which can be used for refined analysis. In order to protect the display delay between the body sense and the actual data, the overall device response delay should be controlled within 100ms.

[0120] In a preferred embodiment of step 3 and / or step 4, an evaluator is required to perform an in-loop test and make a subjective evaluation of the vehicle response.

[0121] The basic working principle of the evaluation method provided by the above embodiment of the present invention is as follows: Figure 3 shown.

[0122] The evaluation method provided in the above embodiment of the present invention generates a VR video based on real vehicle test video data or a constructed virtual scene. The VR video can present the perspectives of the vehicle at different positions, so as to simulate the subjective experience of any perspective of the vehicle after obtaining the data.

[0123] In the evaluation method provided by the above embodiment of the present invention, while playing back the VR video, the dynamic data of the vehicle is played back on the multi-axis simulator to reproduce the posture of the vehicle in the data. The subjective experience evaluator can make a more detailed subjective experience analysis on the simulator by combining the VR video and the dynamic posture of the vehicle.

[0124] An embodiment of the present invention provides an autonomous driving subjective experience evaluation system.

[0125] like Figure 4 As shown, the autonomous driving subjective experience evaluation system provided by this embodiment may include:

[0126] A test scenario construction module, which is used to construct a test scenario including a test vehicle and its surrounding traffic environment, and obtain multi-view videos around the vehicle and vehicle dynamic information data based on the test scenario;

[0127] A video stitching module, which is used to stitch multi-view videos around the vehicle into 360-degree video data;

[0128] A simulation data acquisition module, which is used to synchronize the vehicle dynamic information data with the 360-degree video data on a time axis to obtain the vehicle surrounding environment simulation video and vehicle motion simulation data at the same time;

[0129] The simulation display module uses the VR module and the multi-axis simulator module to load the vehicle's surrounding environment simulation video and vehicle motion simulation data respectively, observe the vehicle's surrounding environment information and feel the vehicle's six-axis posture changes to complete the subjective evaluation.

[0130] It should be noted that the steps in the method provided by the present invention can be implemented by using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system, that is, the embodiments in the method can be understood as preferred examples for constructing the system, which will not be elaborated here.

[0131] An embodiment of the present invention provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the processor can be used to execute any method of the above-mentioned embodiments of the present invention, or to run any system of the above-mentioned embodiments of the present invention.

[0132] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0133] The above-mentioned computer programs, computer instructions, etc. may be stored in one or more memories in partitions, and the above-mentioned computer programs, computer instructions, data, etc. may be called by a processor.

[0134] The processor is used to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0135] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0136] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0137] Those skilled in the art know that, in addition to implementing the system and its various devices provided by the present invention in a purely computer-readable program code, the system and its various devices provided by the present invention can be made to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component; the devices for implementing various functions can also be considered as both software modules for implementing the method and structures within the hardware component.

[0138] The autonomous driving subjective experience evaluation method, system, terminal and medium provided by the above-mentioned embodiments of the present invention provide a quantitative analysis tool for subjective evaluation, which can effectively and quantitatively repeatedly evaluate the subjective experience in different scenarios, polish product details, and solve the problem of the non-reproducibility of subjective evaluation problems in the test process and the inability to reproduce the quantified data and the actual physical sensation. It can greatly reduce the understanding deviation in the communication of subjective experience needs / problems, and effectively shorten the research and development cycle for creating more personalized autonomous driving functions; there is no need for actual test sites, only need to be deployed in the laboratory, which is more convenient than traditional autonomous driving subjective experience tests that rely on different road environments or professional test sites; only a multi-axis degree of freedom simulation platform needs to be built to simulate different vehicle characteristics, and then simulate different vehicles; it focuses on reconstructing data collected by real vehicles, reconstructing them into virtual scenes, and generating them directly from the data without relying on artificially constructed virtual scenes.

[0139] The autonomous driving subjective experience evaluation method, system, terminal and medium provided in the above embodiments of the present invention construct a set of data control technology, which can control the simultaneous playback of VR video and the dynamic response of the vehicle for subjective evaluation through existing data; or control the simultaneous generation of VR video and the dynamic response of the vehicle for subjective evaluation through a constructed simulation scene.

[0140] All matters not covered in the above embodiments of the present invention are well known in the art.

[0141] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for evaluating subjective experience of autonomous driving, characterized in that: include: S1, constructing a test scene including a test vehicle and its surrounding traffic environment, and acquiring multi-view videos around the vehicle and vehicle dynamic information data based on the test scene; S2, stitching the multi-view videos around the vehicle into 360-degree video data; S3, synchronizing the vehicle dynamic information data with the 360-degree video data on a time axis to obtain a vehicle surrounding environment simulation video and vehicle motion simulation data at the same time; using a VR tool to load the vehicle surrounding environment simulation video, using a multi-axis simulator platform to load the vehicle motion simulation data, and simulating the vehicle surrounding environment information and the vehicle six-axis posture changes; Repeat S1 to S3 above to complete the subjective evaluation of autonomous driving; in: The step of synchronizing the vehicle dynamic information data with the 360-degree video data on a time axis to obtain a vehicle surrounding environment simulation video and vehicle motion simulation data at the same time includes: The 360-degree video data and the vehicle dynamic information data are synchronized through the time axis in the vehicle operation data recorded by the real vehicle or generated by simulation, and the 360-degree video data and the vehicle dynamic information data at the same time are retrieved; At the same time, the visual field direction of the evaluation personnel's head posture input is collected, and the viewing angle is set as yaw; through the viewing angle yaw input by the evaluation personnel, the visual range of the specified 360-degree video data in the visual field direction is loaded in real time, and the simulated video of the vehicle surrounding environment is obtained as follows: Video o =f(timestamps,yaw,FOV) Where: timestamps is the time of synchronization; Video o Simulate video of the vehicle's surroundings; Yaw is the head posture of the evaluator, the deflection angle along the vehicle's travel direction; FOV is the visible range of VR video; The vehicle dynamic information data at the same time as the 360-degree video data is retrieved to obtain the vehicle motion simulation data: Pose o =f(timestamps) Where: Pose o Simulation data for vehicle motion, including: acceleration a along the x-axis x , angle around the x-axis ω x , acceleration along the y-axis a y , angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axis y ; The vehicle surrounding environment simulation video and vehicle motion simulation data obtained at the same time are expressed as: [Video o ,Pose o ]=f(timestamps,yaw,FOV,pause) Where: Pause is an interface for pausing the 360-degree video and vehicle dynamic information data. It is used to facilitate pausing at any time to evaluate the dynamic response of the test vehicle in the current test scenario. The method of using a VR tool to load a simulated video of the vehicle's surrounding environment, using a multi-axis simulator platform to load vehicle motion simulation data, and simulating the vehicle's surrounding environment information and the vehicle's six-axis posture changes includes: Output the obtained simulated video of the vehicle's surrounding environment to the VR headset of the evaluator; Through CAN communication, the obtained vehicle motion simulation data is responded to on the multi-axis simulator platform to obtain the postures in different axes; among which: The multi-axis simulator platform controls the thrust of the electric push rod of the multi-axis simulator to simulate the acceleration on different axes according to the postures on different axes requested by the platform; and simulates the deflection angle around different axes through the combination of different push rod strokes.

2. The method for subjective experience evaluation of autonomous driving according to claim 1, characterized in that: The test scenarios include: real road scenarios and virtual simulation scenarios; wherein: For the real road scene, construct the test scene, including: Based on the ISO vehicle coordinate system, multiple cameras are used to synchronously capture multi-angle video images along the xy axis plane to obtain the surrounding vehicle video; In the ISO vehicle coordinate system, the vehicle dynamic information data on the three axes of xyz is obtained through the on-board IMU sensor, including: acceleration a along the x-axis x , angle around the x-axis ω x , acceleration along the y-axis a y , angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axis y ; Reproducing a test scene including a test vehicle and its traffic environment based on the surrounding vehicle video and the vehicle dynamic information data; For the virtual simulation scenario, constructing the test scenario includes: By using the virtual scene editor and manually setting the surrounding vehicle video and vehicle dynamic information data, a test scene including the current test and its traffic environment is constructed.

3. The method for subjective experience evaluation of autonomous driving according to claim 1, characterized in that: The step of stitching the multi-view videos around the vehicle into 360-degree video data includes: The multi-view videos of the surrounding vehicle are spliced, and the features and key points in each frame of the video are locally detected using a scale-invariant feature transformation matching algorithm; A random sampling consensus method is used to exclude the matches of adjacent videos that do not conform to geometric changes, and the transformation matrix H is calculated using the remaining matching points; For each frame of adjacent videos to be spliced, the transformation matrix H is used to perform projection transformation, and the pixels of the overlapping parts after transformation are weighted average fused; Through the above steps, the multi-view videos around the vehicle at the same time are stitched together to generate continuous 360-degree video data based on the vehicle's xy plane.

4. An autonomous driving subjective experience evaluation system, characterized in that: include: A test scenario construction module, which is used to construct a test scenario including a test vehicle and its surrounding traffic environment, and based on the test scenario, obtain multi-view videos around the vehicle and vehicle dynamic information data; A video stitching module, which is used to stitch the multi-view videos around the vehicle into 360-degree video data; A simulation data acquisition module, which is used to synchronize the vehicle dynamic information data with the 360-degree video data on a time axis to obtain a vehicle surrounding environment simulation video and vehicle motion simulation data at the same time; A simulation display module, which uses a VR module and a multi-axis simulator module to load the vehicle surrounding environment simulation video and the vehicle motion simulation data, respectively, observe the vehicle surrounding environment information and feel the vehicle six-axis posture changes, and complete the subjective evaluation; in: The simulation data acquisition module synchronizes the vehicle dynamic information data with the 360-degree video data on a time axis to obtain the vehicle surrounding environment simulation video and vehicle motion simulation data at the same time, including: The 360-degree video data and the vehicle dynamic information data are synchronized through the time axis in the vehicle operation data recorded by the real vehicle or generated by simulation, and the 360-degree video data and the vehicle dynamic information data at the same time are retrieved; At the same time, the visual field direction of the evaluation personnel's head posture input is collected, and the viewing angle is set as yaw; through the viewing angle yaw input by the evaluation personnel, the visual range of the specified 360-degree video data in the visual field direction is loaded in real time, and the simulated video of the vehicle surrounding environment is obtained as follows: Video o =f(timestamps,yaw,FOV) Where: timestamps is the time of synchronization; Video o Simulate video of the vehicle's surroundings; Yaw is the head posture of the evaluator, the deflection angle along the vehicle's travel direction; FOV is the visible range of VR video; The vehicle dynamic information data at the same time as the 360-degree video data is retrieved to obtain the vehicle motion simulation data: Pose o =f(timestamps) Where: Pose o Simulation data for vehicle motion, including: acceleration a along the x-axis x , angle around the x-axis ω x , acceleration along the y-axis a y , angle θ around the y-axis y , acceleration along the z-axis a z and the angle ψ around the z-axis y ; The vehicle surrounding environment simulation video and vehicle motion simulation data obtained at the same time are expressed as: [Video o ,Pose o ]=f(timestamps,yaw,FOV,pause) Where: Pause is an interface for pausing the 360-degree video and vehicle dynamic information data. It is used to facilitate pausing at any time to evaluate the dynamic response of the test vehicle in the current test scenario. The simulation display module uses the VR module and the multi-axis simulator module to load the vehicle surrounding environment simulation video and the vehicle motion simulation data respectively, observe the vehicle surrounding environment information and feel the vehicle six-axis posture changes, including: Output the obtained simulated video of the vehicle's surrounding environment to the VR headset of the evaluator; Through CAN communication, the obtained vehicle motion simulation data is responded to on the multi-axis simulator platform to obtain the postures in different axes; among which: The multi-axis simulator platform controls the thrust of the electric push rod of the multi-axis simulator to simulate the acceleration on different axes according to the postures on different axes requested by the platform; and simulates the deflection angle around different axes through the combination of different push rod strokes.

5. The autonomous driving subjective experience evaluation system according to claim 4, characterized in that: Also includes: A data playback tool module, which is used to arrange the simulation data acquisition module, control the timeline synchronization, and be connected to the simulation display module control module, and is used to control the VR module to play, pause or replay the vehicle surrounding environment simulation video and control the multi-axis simulator module to execute the vehicle motion simulation data.

6. A computer terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 3, or to run the system described in any one of claims 4 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 3, or to run the system described in any one of claims 4 to 5.

Images