Driver-in-the-loop simulation test method, system and device and storage medium

Abstract

The invention discloses a driver in-the-loop simulation test method, system and device and a storage medium, and relates to the technical field of driving test. According to the method, position action tracking is conducted on a driver through a motion capture position tracking system, driver tracking data are obtained, then the current state of the driver is determined according to the driver tracking data, and when the current state of the driver is a driving cabin entering state, the driver enters the driving cabin. If yes, a virtual cabin entering scene is constructed according to the cockpit in the real environment and the actual cabin entering route, and the virtual cabin entering scene is displayed in virtual reality equipment worn on the driver; and when the current state of the driver is that the driver enters the cockpit, initializing a virtual driving scene according to the virtual cabin entering scene, and displaying the virtual driving scene in virtual reality equipment. The driving simulation immersion of the driver can be improved, and the reliability of the in-the-loop simulation test is improved.

Description

Technical Field

[0001] The present invention relates to the field of driving test technology, and in particular to a driver-in-the-loop simulation test method, system, equipment and storage medium. Background Art

[0002] In-the-loop simulation testing, also known as hardware-in-the-loop (HiL), is a powerful test method for effectively testing embedded control systems. This test method simulates the operating state of the controlled object by running a simulation model on a real-time processor, and connects to the cockpit through an I / O interface. In the field of vehicle testing, the vehicle is used as a controlled object, and driving tests are carried out in a simulation model to reduce development time and cost. Through driving simulation, various test verification scenarios of concern can be created quickly and richly without placing the driving test personnel in danger in the actual operating vehicle environment, and without relying on actual specific traffic conditions, weather, etc., and can ensure that the same test evaluation environment with high consistency is provided to different test personnel. Compared with actual vehicle testing, driving simulation can evaluate and develop improvements in automotive functional performance more safely, efficiently, consistently and reliably.

[0003] At present, the driver-passenger in-the-loop simulation test is basically carried out in the test space through a dynamic driving simulator system. The basic principle scheme is: establish a motion simulation platform and cockpit system in the test space to simulate the dynamic motion state of the vehicle, arrange a screen vision system around the cockpit to simulate and display the dynamic scene around the vehicle, and the driver can simulate driving the vehicle in the simulated cockpit through the steering wheel and pedals and other control systems. In this test method, the driver logs into the simulated cockpit from the test space and then conducts a simulated driving test. The tester will clearly realize that he is using a driving simulator device, which reduces the immersion of simulating real car driving in psychological perception. Under this non-real driving psychological suggestion, the driver's test behavior may be more radical and dangerous, which will be inconsistent with the behavior of driving an actual vehicle, resulting in a decrease in the overall test credibility. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a driver-in-the-loop simulation test method, system, device and storage medium, aiming to improve the driver's immersion in simulated driving and improve the credibility of the in-the-loop simulation test.

[0005] To achieve the above object, one aspect of an embodiment of the present application proposes a driver-in-the-loop simulation test method, comprising the following steps:

[0006] Track the position and movement of the driver through a motion capture position tracking system to obtain driver tracking data;

[0007] determining a current state of a driver based on the driver tracking data;

[0008] When the driver is currently in the state of entering the cockpit, a virtual cockpit entry scene is constructed according to the cockpit in the real environment and the actual cockpit entry route, and the virtual cockpit entry scene is displayed in a virtual reality device, wherein the virtual reality device is used to be worn on the driver;

[0009] When the driver is currently in the cockpit state, a virtual driving scene is initialized according to the virtual cockpit entry scene, and the virtual driving scene is displayed in a virtual reality device.

[0010] In some embodiments, the motion capture position tracking system includes a plurality of cameras for being arranged in the test space, a first tracker for being arranged in the virtual reality device, a second tracker for being arranged in the driver's hand, and a third tracker for being arranged in the cockpit door handle;

[0011] The method of tracking the position and movement of the driver by a motion capture position tracking system to obtain driver tracking data comprises the following steps:

[0012] Obtain the video stream captured by the camera;

[0013] The video stream is subjected to target recognition and extraction of the first tracker, the second tracker, and the third tracker to obtain driver tracking data.

[0014] In some embodiments, determining the current state of the driver according to the driver tracking data comprises the following steps:

[0015] Performing position identification on a first tracker in the driver tracking data to obtain a driver position;

[0016] Analyzing the relative positions of the second tracker and the third tracker in the driver tracking data to identify the driver's door opening action;

[0017] When the driver is located in the test space and the driver's door opening action is not recognized, determining that the driver is currently in the state of entering the cockpit;

[0018] When the driver's door opening action is recognized, it is determined that the driver is currently in a state of preparing to enter the cockpit;

[0019] When the driver is located in the cockpit of the test space, it is determined that the driver is currently in the cockpit state.

[0020] In some embodiments, the step of constructing a virtual cabin entry scene based on the cockpit in the real environment and the actual cabin entry route includes the following steps:

[0021] Initialize the three-dimensional virtual space according to the test space image;

[0022] Constructing a corresponding virtual vehicle in the three-dimensional virtual space according to the position and appearance of the cockpit in the real environment;

[0023] Constructing a virtual driving road in the three-dimensional virtual space according to the position of the virtual vehicle, and constructing a virtual cabin entry road in the three-dimensional virtual space according to the actual cabin entry route between the test space entrance and the cockpit position;

[0024] The blank areas in the three-dimensional virtual space are filled with virtual elements to obtain a virtual cabin entry scene.

[0025] In some embodiments, displaying the virtual cabin entry scene in a virtual reality device includes the following steps:

[0026] determining a driver's viewing angle based on relative positions between marker points of a first tracker in the driver tracking data;

[0027] A picture of a corresponding perspective is selected from the virtual cabin entry scene according to the driver's perspective and displayed in the virtual reality device.

[0028] In some embodiments, the driver-in-the-loop simulation test method further includes the following steps:

[0029] When the driver is currently in a state of preparing to enter the cockpit, constructing a virtual door opening animation according to a door motion model of the virtual vehicle in the virtual cockpit entry scene;

[0030] The virtual door opening animation is displayed in a virtual reality device.

[0031] In some embodiments, the driver-in-the-loop simulation test method further includes the following steps:

[0032] Collect driving instructions from the cockpit;

[0033] The image of the virtual driving scene displayed in the virtual reality device is updated according to the driving instruction.

[0034] To achieve the above object, another aspect of the embodiment of the present application provides a driver-in-the-loop simulation test system, comprising:

[0035] Motion capture position tracking system, used to track the position and motion of the driver and obtain driver tracking data;

[0036] A processing system is used to determine the current state of the driver based on the driver tracking data; when the current state of the driver is entering the cockpit, a virtual cabin entry scene is constructed according to the cockpit in a real environment and the actual cabin entry route, and the virtual cabin entry scene is displayed in a virtual reality device, wherein the virtual reality device is worn on the driver; when the current state of the driver is already in the cockpit, a virtual driving scene is initialized according to the virtual cabin entry scene, and the virtual driving scene is displayed in the virtual reality device.

[0037] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the driver-in-the-loop simulation test method described in the above-mentioned embodiment is realized.

[0038] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the driver-in-the-loop simulation test method described in the above embodiment.

[0039] The driver-in-the-loop simulation test method, system, device and storage medium proposed in the present application track the position and movement of the driver through a motion capture position tracking system to obtain driver tracking data, and then determine the driver's current state based on the driver tracking data. When the driver's current state is entering the cockpit, a virtual cabin entry scene is constructed based on the cockpit in the real environment and the actual cabin entry route, and the virtual cabin entry scene is displayed in a virtual reality device worn by the driver; when the driver's current state is already in the cockpit, the virtual driving scene is initialized according to the virtual cabin entry scene, and the virtual driving scene is displayed in the virtual reality device. The present application allows the driver to wear a virtual reality device to experience the simulated real environment where the cockpit is located before entering the cabin, which can improve the driver's immersion in simulated driving, and initializes the virtual driving scene based on the virtual cabin entry scene, improves the continuity of the virtual scenes of the cabin entry process and the simulated driving process, thereby improving the authenticity of the simulated driving experience, and further improving the credibility of the in-the-loop simulation test. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a driver-in-the-loop simulation test method provided by an embodiment of the present application;

[0041] Figure 2 yes Figure 1Flow chart of step S101 in FIG.

[0042] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0043] Figure 4 yes Figure 1 A flowchart of the step of constructing a virtual approach field according to the cockpit in the real environment and the actual approach route in step S103;

[0044] Figure 5 yes Figure 1 A flowchart of the step of displaying the virtual cabin entry scene in the virtual reality device in step S103;

[0045] Figure 6 is a flow chart of a driver-in-the-loop simulation test method provided by another embodiment of the present application;

[0046] Figure 7 is a flow chart of a driver-in-the-loop simulation test method provided by another embodiment of the present application;

[0047] Figure 8 is a schematic diagram of a driver-in-the-loop simulation test system provided in an embodiment of the present application;

[0048] Fig. 9 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0049] Fig.10 It is a schematic diagram of the driver-in-the-loop simulation test process provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0053] The driver-in-the-loop simulation test method, system, device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the driver-in-the-loop simulation test method in the embodiments of the present application is described.

[0054] The driver-in-the-loop simulation test method provided in the embodiment of the present application relates to the field of driving test technology. The driver-in-the-loop simulation test method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the driver-in-the-loop simulation test method, etc., but is not limited to the above forms.

[0055] The driver-in-the-loop simulation test method of the embodiment of the present application can be applied to the following scenarios:

[0056] The cockpit for simulated driving is set in the test space, which can refer to a test room or a space formed by the shooting angle of the distributed cameras of the motion capture position tracking system. The motion capture position tracking system mainly includes distributed cameras, which are used to track the position and posture of the driver.

[0057] Before entering the test space, the driver wears a virtual reality device, such as VR glasses. The virtual reality device communicates wirelessly with the remote processing system. The content displayed by the virtual reality device is controlled by the processing system. The processing system can execute the driver-in-the-loop testing method of the embodiment of the present application.

[0058] Specifically, the processing system obtains the real-time position and posture of the driver through the motion capture position tracking system. When the driver is identified to have entered the test area, the virtual reality device can be controlled to display a virtual cabin entry scene. The virtual cabin entry scene is constructed according to the cockpit and the actual cabin entry route in the real environment, and can imitate the displayed parking environment in the virtual space, and can guide the driver to walk to the cockpit accurately in the virtual space, immersing the driver in the real parking environment in advance before entering the cockpit, improving the consistency of the driver's subsequent simulated driving operations with the driving operations in the real environment, thereby improving the accuracy of the test. After recognizing that the driver has entered the cockpit, the processing system can control the virtual reality device to display the virtual driving scene, so that the driver can complete the driving test in a simulated real environment.

[0059] Figure 1 is an optional flowchart of the driver-in-the-loop simulation test method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0060] Step S101, tracking the position and movement of the driver through a motion capture position tracking system to obtain driver tracking data;

[0061] Step S102, determining the current state of the driver according to the driver tracking data;

[0062] Step S103, when the driver is currently in the state of entering the cockpit, a virtual cockpit entry scene is constructed according to the cockpit in the real environment and the actual cockpit entry route, and the virtual cockpit entry scene is displayed in a virtual reality device, wherein the virtual reality device is used to be worn by the driver;

[0063] Step S104, when the driver is currently in the cockpit state, a virtual driving scene is initialized according to a virtual cockpit entry scene, and the virtual driving scene is displayed in a virtual reality device.

[0064] In step S101 of some embodiments, the motion capture position tracking system includes a plurality of cameras in different orientations installed around the test space. Specifically, the plurality of cameras are not installed in the same straight line, and the fixed position is stable and does not shake. The number and position of the cameras can be set according to actual needs, such as requiring that each tracked position in the cabin and the driver's entry and boarding route area needs to be in at least three camera perspectives. The motion capture position tracking system can use the target detection model to perform target recognition on the tracked object according to the characteristics of the object to be tracked, thereby tracking the position and motion posture of the driver. Further, the motion capture position tracking system can also include a tracker, specifically including a first tracker for setting in the virtual reality device, a second tracker for setting in the driver's hand, and a third tracker for setting in the cockpit door handle. Specifically, the tracker can be a marking component with prominent features in the image, for example, a solid ball of a material with strong reflection of a specific wavelength. The tracker is arranged at the key motion capture position points of the cockpit door handle, the driver's head, and the hand, which can improve the motion tracking effect. For example, three marking points are arranged at non-collinear positions on the car door to form the third tracker; four marking points are arranged on the VR glasses to form the first tracker; one marking point is arranged at the base of the driver's hand to form the second tracker. The camera in the test space is used to collect images containing the above trackers. In order to ensure the accuracy and real-time performance of motion capture information, the camera acquisition frequency is not less than 50Hz. The camera can use a dedicated specific wavelength to illuminate the imaging camera to achieve image positioning of a solid ball with a strong reflective material of a specific wavelength on the captured object.

[0065] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S202:

[0066] Step S201, obtaining a video stream captured by a camera;

[0067] Step S202 , performing target recognition and extraction of the first tracker, the second tracker, and the third tracker on the video stream to obtain driver tracking data.

[0068] In this embodiment, the camera performs real-time shooting to obtain a video stream, and the video stream is an image sequence collected at a certain frequency. The video stream is input into a target detection model for target recognition and image frames containing the target are extracted to obtain driver tracking data. The target recognition includes a first tracker, a second tracker, and a third tracker. The target detection model can be a neural network model, for example, a yolov model, and a specific target can be recognized by training the model.

[0069] In step S102 of some embodiments, driver tracking data is obtained through a motion capture position tracking system. By performing position and posture analysis on the tracker in the driver tracking data, the driver's current position, the driver's head perspective, and the driver's hand movements can be determined, thereby determining the driver's current state.

[0070] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S305:

[0071] Step S301, performing position identification on the first tracker in the driver tracking data to obtain the driver's position;

[0072] Step S302, analyzing the relative positions of the second tracker and the third tracker in the driver tracking data to identify the driver's door opening action;

[0073] Step S303, when the driver is located in the test space and the driver's door opening action is not recognized, it is determined that the driver is currently in the state of entering the cockpit;

[0074] Step S304, when the driver's door opening action is identified, it is determined that the driver is currently in a state of preparing to enter the cockpit;

[0075] Step S305: When the driver is located in the cockpit of the test space, it is determined that the driver is currently in the cockpit state.

[0076] In this embodiment, an image including the first tracker in the driver tracking data is extracted, and the position of the first tracker in the image is compared according to the reference position in the test space, so as to determine the driver's position in the test space. An image including the second tracker and the third tracker is extracted from the driver tracking data, and the relative position between the second tracker and the third tracker in the image is analyzed to identify the driver's door opening action. Specifically, when the distance between the second tracker and the third tracker is less than a preset value, it indicates that the driver's hand is close to or in contact with the door handle position, and it is considered that the driver's door opening action is recognized at this time. When the driver's position is in the test space and the driver's door opening action is not recognized, it indicates that the driver has entered the test space wearing a VR device and has not entered the cockpit at this time, and it can be determined that the driver's current state is entering the cockpit state. When the driver's door opening action is recognized, it indicates that the driver is near the cockpit and is ready to open the door at this time, and it can be determined that the driver's current state is ready to enter the cockpit state. When the driver's position is in the cockpit of the test space, it is determined that the driver's current state is already in the cockpit state. Specifically, a camera may be arranged in the cockpit. When the camera in the cockpit recognizes the driver, it may be determined that the driver is located in the cockpit of the test space.

[0077] In step S103 of some embodiments, when the driver's current state is entering the cockpit state, which means that the driver has entered the test space and is in the process of boarding the cockpit, the processing system controls the VR device to display a virtual cabin entry scene. The virtual cabin entry scene is constructed based on the cockpit and the actual cabin entry route in the real environment. It can simulate the displayed parking environment in the virtual space, and can guide the driver to walk accurately to the cockpit in the virtual space. Before the driver enters the cockpit, he is immersed in the real parking environment in advance, thereby improving the consistency of the driver's driving operations in subsequent simulated driving operations with those in the real environment, thereby improving the accuracy of the test.

[0078] See also Figure 4 In some embodiments, the step S103 of constructing a virtual cabin entry scene according to the cockpit in the real environment and the actual cabin entry route may include but is not limited to steps S401 to S404:

[0079] Step S401, initializing the three-dimensional virtual space according to the test space image;

[0080] Step S402, constructing a corresponding virtual vehicle in a three-dimensional virtual space according to the position and appearance of the cockpit in the real environment;

[0081] Step S403, constructing a virtual driving road in the three-dimensional virtual space according to the virtual vehicle position, and constructing a virtual cabin entry road in the three-dimensional virtual space according to the actual cabin entry route between the test space entrance and the cockpit position;

[0082] Step S404, fill virtual elements in the blank area in the three-dimensional virtual space to obtain a virtual cabin entry scene.

[0083] In this embodiment, the camera in the test space captures images in the real environment, determines the fixed position of the cockpit in the test space and the boarding route area, and performs virtual reconstruction of the actual test space entry environment according to the actual route from the entrance of the driving simulation test space to the simulated cockpit and the ground conditions. The test space image can be obtained by laser point cloud scanning or elevation mapping to establish a 1:1 test space entry and boarding route and a virtual cabin entry scene where the cockpit is located. The specific construction process is as follows: initialize the three-dimensional virtual space according to the test space image obtained by laser point cloud scanning, replace the simulated cockpit with a 1:1 virtual vehicle model in the three-dimensional virtual space, Among them, the door of the cockpit is separately established as a motion model that can be opened and closed relative to the body of the car; a virtual driving road is constructed according to the position of the virtual vehicle (specifically the position of the contact point of the tire of the vehicle model), the ground route from the entrance of the test space to the cockpit door is mapped and beautified, and a virtual cabin entry route is constructed, wherein the virtual driving road in the three-dimensional virtual space does not overlap or interfere with the virtual cabin entry route; the blank area outside the virtual driving road, virtual vehicle and virtual cabin entry road in the three-dimensional virtual space is filled, and by adding virtual elements of the environmental scenes pre-stored in the system, such as roadbed, curb, grass, bushes, distant view, sky, etc., a three-dimensional virtual cabin entry scene simulating a real parking environment is constructed.

[0084] See also Figure 5 In some embodiments, step S103 of displaying the virtual cabin scene in the virtual reality device may include but is not limited to steps S501 to S502:

[0085] Step S501, determining the driver's viewing angle according to the relative positions between the marker points of the first tracker in the driver tracking data;

[0086] Step S502: Select a picture of a corresponding perspective from the virtual cabin entry scene according to the driver's perspective and display it in the virtual reality device.

[0087] In this embodiment, four marker points that are not in the same straight line are distributed on the VR glasses worn on the driver's head to form a first tracker. The driver's perspective can be determined by the relative position relationship between the four marker points and the distance between the marker points in the collected image. After the processing system recognizes that the driver has entered the test space, it controls the VR glasses to display the virtual cabin entry scene, and analyzes the driver's perspective in real time, creates a scene image rendering from the driver's perspective in the virtual cabin entry scene, and transmits the rendered driver's head perspective image to the VR glasses for display. At this time, the driver can observe the three-dimensional virtual cabin entry scene that moves with the perspective through the VR glasses.

[0088] See also Figure 6In some embodiments, the driver-in-the-loop simulation test method of the embodiment of the present application may also include but is not limited to steps S601 to S602:

[0089] Step S601, when the driver is currently in a state of preparing to enter the cockpit, a virtual door opening animation is constructed according to a door motion model of a virtual vehicle in a virtual cabin entry scene;

[0090] Step S602: Display the virtual door opening animation in the virtual reality device.

[0091] In this embodiment, the door motion model of the virtual vehicle is a door state transition algorithm pre-stored in the processing system to characterize the door opening process. When the processing system recognizes that the driver's hand is close to the door, it determines that the driver is currently preparing to enter the cockpit. At this time, a virtual door opening animation of the virtual vehicle is constructed according to the door motion model of the virtual vehicle, and the virtual door opening animation is displayed in the virtual reality device. In this embodiment, when the driver opens the door, the VR glasses display the virtual door opening animation of the virtual vehicle accordingly, which improves the driver's immersion in the process of boarding the cabin.

[0092] See also Figure 7 In some embodiments, the driver-in-the-loop simulation test method of the embodiment of the present application may also include but is not limited to steps S701 to S702:

[0093] Step S701, collecting driving instructions from the cockpit;

[0094] Step S702: updating the image of the virtual driving scene displayed in the virtual reality device according to the driving instruction.

[0095] In this embodiment, after the processing system recognizes that the driver has entered the cockpit, it controls the VR glasses to switch to a virtual driving scene for display. The virtual driving scene is a virtual scene similar to the actual test environment or specific to the test requirements. The scene may include elements such as roads, traffic signals, other vehicles, pedestrians, buildings, and the interaction rules and physical properties between them. These elements are created by simulation software and simulated based on real physical laws and traffic rules. The controller under test (such as the controller of the autonomous driving system) will generate control instructions according to the current driving instructions, such as acceleration, deceleration, steering, braking and other actions. When the processing system receives the control instruction, it updates the status of each element in the simulation model according to the control instruction to update the displayed virtual driving scene. The updated elements include but are not limited to information such as the position, speed, direction of the vehicle, and changes in the status of traffic signals, other vehicles and pedestrians. The driver's experience is improved by following the changes in the real-time driving scene.

[0096] According to some embodiments of the present application, referring to Fig.10, the driver-in-the-loop simulation test process of the embodiment of the present application is explained.

[0097] At the beginning of the test, the test driver wears VR glasses and stands at the entrance of the laboratory. Then, a virtual cabin entry scene constructed based on the cockpit and actual cabin entry route in the real environment is displayed in the VR glasses. The virtual cabin entry scene is a simulation of the virtual vehicle parking and boarding environment. At this time, the driver will approach the virtual vehicle according to the virtual vehicle position and virtual cabin entry road in the virtual cabin entry scene. The virtual cabin entry scene is constructed according to the actual laboratory, and the driver's movement in the virtual scene will be consistent with the laboratory boarding route, so that the driver can walk to the cockpit smoothly.

[0098] The key positions of the driver's movements are marked according to the movements of the driver's body parts, and the key positions of the simulated cockpit in the laboratory are marked. When the driver walks to the cockpit, the position and motion tracking system tracks the driver according to the key position marking points of the driver's movements, obtains driving status information (including the driver's position and movement posture), and determines the position information of the cabin, i.e., the door, according to the key position marking points of the cockpit. The processing system analyzes the driver's head viewing angle according to the information obtained by the position and motion tracking system, so as to adaptively render the image of the corresponding viewing angle in the virtual cabin entry scene according to the driver's head viewing angle, and display it in the VR glasses. The processing system analyzes the driver's hand movements and the cockpit door movements according to the information obtained by the position and motion tracking system, and displays the virtual door opening animation in the VR glasses accordingly, realizing the cockpit door opening and closing interaction in both the real environment and the virtual environment. When the driver reaches the vehicle door opening position in the virtual scene, the actual driver has reached the door opening position of the simulated cockpit. The driver will extend his hand to open the door according to the door position in the virtual scene. By collecting the hand marker position information and the door marker position information, the door model in the virtual scene is driven to open, allowing the driver to complete the action of entering the cabin.

[0099] In an embodiment of the present application, the laboratory simulator device under test is associated with a virtual simulation scene, and the VR device system and simulation technology are combined to enable the test personnel to enter the simulated cockpit from the driving simulation laboratory, and the visual perception is connected to the virtual scene to be tested, and the fact that the test personnel are in the laboratory conditions is concealed by the visual perception. This embodiment can reduce the test personnel's perception of the laboratory environment, thereby enhancing the sense of immersion during the test, so that the operation of the driver's test can be closer to driving an actual real vehicle, bringing better simulation effects, and at the same time, it can increase the perception and adaptation time of the virtual scene before entering the cockpit test, reducing the time for personnel to adapt to a large amount of immersion on the simulator.

[0100] See also Figure 8The embodiment of the present application also provides a driver-in-the-loop simulation test system, comprising:

[0101] Motion capture position tracking system, used to track the position and motion of the driver and obtain driver tracking data;

[0102] A processing system is used to determine the current state of the driver based on the driver tracking data; when the driver's current state is entering the cockpit, a virtual cabin entry scene is constructed based on the cockpit in the real environment and the actual cabin entry route, and the virtual cabin entry scene is displayed in a virtual reality device, wherein the virtual reality device is used to be worn on the driver; when the driver's current state is already in the cockpit, a virtual driving scene is initialized according to the virtual cabin entry scene, and the virtual driving scene is displayed in the virtual reality device.

[0103] It can be understood that the contents of the above-mentioned driver-in-the-loop simulation test method embodiment are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above-mentioned driver-in-the-loop simulation test method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned driver-in-the-loop simulation test method embodiment.

[0104] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned driver-in-the-loop simulation test method. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0105] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0106] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0107] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the driver-in-the-loop simulation test method of the embodiment of this application;

[0108] Input / output interface 903, used to implement information input and output;

[0109] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0110] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0111] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0112] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned driver-in-the-loop simulation test method.

[0113] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0114] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0115] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0116] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0117] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0118] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, systems or units, which can be electrical, mechanical or other forms.

[0119] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0122] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A driver-in-the-loop simulation test method, characterized in that: The following steps are involved: Track the position and movement of the driver through a motion capture position tracking system to obtain driver tracking data; determining a current state of a driver based on the driver tracking data; When the driver is currently in the state of entering the cockpit, a virtual cockpit entry scene is constructed according to the cockpit in the real environment and the actual cockpit entry route, and the virtual cockpit entry scene is displayed in a virtual reality device, wherein the virtual reality device is used to be worn on the driver; When the driver is currently in the cockpit state, a virtual driving scene is initialized according to the virtual cockpit entry scene, and the virtual driving scene is displayed in a virtual reality device.

2. The driver-in-the-loop simulation test method according to claim 1, characterized in that: The motion capture position tracking system includes a plurality of cameras for being arranged in the test space, a first tracker for being arranged in the virtual reality device, a second tracker for being arranged in the driver's hand, and a third tracker for being arranged in the cockpit door handle; The method of tracking the position and movement of the driver by a motion capture position tracking system to obtain driver tracking data comprises the following steps: Obtain the video stream captured by the camera; The video stream is subjected to target recognition and extraction of the first tracker, the second tracker, and the third tracker to obtain driver tracking data.

3. The driver-in-the-loop simulation test method according to claim 2, characterized in that: Determining the current state of the driver according to the driver tracking data comprises the following steps: Performing position identification on a first tracker in the driver tracking data to obtain a driver position; Analyzing the relative positions of the second tracker and the third tracker in the driver tracking data to identify the driver's door opening action; When the driver is located in the test space and the driver's door opening action is not recognized, determining that the driver is currently in the state of entering the cockpit; When the driver's door opening action is recognized, it is determined that the driver is currently in a state of preparing to enter the cockpit; When the driver is located in the cockpit of the test space, it is determined that the driver is currently in the cockpit state.

4. The driver-in-the-loop simulation test method according to claim 1, characterized in that: The method of constructing a virtual cabin entry scene according to the cockpit in the real environment and the actual cabin entry route includes the following steps: Initialize the three-dimensional virtual space according to the test space image; Constructing a corresponding virtual vehicle in the three-dimensional virtual space according to the position and appearance of the cockpit in the real environment; Constructing a virtual driving road in the three-dimensional virtual space according to the position of the virtual vehicle, and constructing a virtual cabin entry road in the three-dimensional virtual space according to the actual cabin entry route between the test space entrance and the cockpit position; The blank areas in the three-dimensional virtual space are filled with virtual elements to obtain a virtual cabin entry scene.

5. The driver-in-the-loop simulation test method according to claim 2, characterized in that: Displaying the virtual cabin entry scene in a virtual reality device comprises the following steps: determining a driver's viewing angle based on relative positions between marker points of a first tracker in the driver tracking data; A picture of a corresponding perspective is selected from the virtual cabin entry scene according to the driver's perspective and displayed in the virtual reality device.

6. The driver-in-the-loop simulation test method according to claim 3, characterized in that: The driver-in-the-loop simulation test method further comprises the following steps: When the driver is currently in a state of preparing to enter the cockpit, constructing a virtual door opening animation according to a door motion model of the virtual vehicle in the virtual cockpit entry scene; The virtual door opening animation is displayed in a virtual reality device.

7. The driver-in-the-loop simulation test method according to claim 1, characterized in that: The driver-in-the-loop simulation test method further comprises the following steps: Collect driving instructions from the cockpit; The image of the virtual driving scene displayed in the virtual reality device is updated according to the driving instruction.

8. A driver-in-the-loop simulation test system, characterized in that: include: Motion capture position tracking system, used to track the position and motion of the driver and obtain driver tracking data; a processing system for determining a current state of a driver based on the driver tracking data; When the driver is currently in the state of entering the cockpit, a virtual cabin entry scene is constructed according to the cockpit in the real environment and the actual cabin entry route, and the virtual cabin entry scene is displayed in a virtual reality device, wherein the virtual reality device is used to be worn on the driver; when the driver is currently in the state of having entered the cockpit, a virtual driving scene is initialized according to the virtual cabin entry scene, and the virtual driving scene is displayed in the virtual reality device.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the driver-in-the-loop simulation test method as described in any one of claims 1 to 7 are realized.

10. A computer storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the driver-in-the-loop simulation test method described in any one of claims 1 to 7.

Images