High-fidelity man-machine interaction driving test method and device based on digital twinning

Abstract

The invention discloses a high-fidelity man-machine interaction driving test method and device based on digital twinning, and the scheme can comprise the steps: collecting the operation of a driver through a force feedback steering wheel and pedal equipment, generating an operation signal, receiving the operation signal through a data processing and control software environment, and converting the operation signal into a vehicle control instruction; and the digital twin model is sent to the vehicle dynamics simulation software, the simulation software carries out real-time dynamics calculation on the digital twin model according to an instruction and outputs vehicle state data, the data is subjected to byte packaging processing to form a binary data packet, and the binary data packet is synchronously sent to the three-dimensional graphic engine through a UDP protocol. The graphic engine analyzes the data packet and directly drives the vehicle model in the virtual scene to update the position, the posture and the wheel state according to the content of the data packet, and a built-in simple physical engine is not used in the process. According to the scheme, all software modules run at the same ip address or networking computing equipment, data interaction is achieved through a UDP protocol, and a driving test closed-loop platform with the human-machine-in-the-loop is jointly constructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a high-fidelity human-machine interaction driving test method and device based on digital twinning. BACKGROUND

[0002] With the rapid development of automatic driving technology, comprehensive, safe and efficient testing and verification of automatic driving systems are increasingly important. Traditional road testing methods have the problems of long time consumption and high cost, and especially when testing extreme or dangerous scenarios, there are also non-negligible safety risks, so simulation technology has become an indispensable means in the development and testing of autonomous vehicles.

[0003] However, existing simulation solutions have obvious deficiencies. On the one hand, traditional high-precision dynamics software represented by TruckSim can provide detailed physical models and dynamic behavior simulation, but the graphics rendering capability is relatively simple and it is difficult to provide an immersive user experience. On the other hand, high-performance graphics engines such as Unity 3D have excellent graphics expression, flexible scene construction capabilities and rich interactive components, but the built-in vehicle physical model is usually a simplified version and cannot meet the engineering-level precision requirements of vehicle dynamics.

[0004] Currently, some technologies use high-performance real-time controllers to build test platforms, transmit input and output data of dynamics simulation to Unity 3D and realize real-time control operation. Mainstream devices such as NI, dSPACE and SpeedGoat have a minimum simulation step of 1ms, which can meet the real-time requirements of most vehicle chassis simulation tests, but such high-performance real-time controllers as hardware-in-the-loop test necessary equipment greatly increase the economic cost of simulation testing. SUMMARY

[0005] The present application provides a high-fidelity human-machine interaction driving test method and device based on digital twinning to overcome at least one technical problem in the related art.

[0006] According to a first aspect of the embodiments of the present application, a high-fidelity human-machine interaction driving test method based on digital twinning is provided, comprising: capturing physical operations of a driver through force feedback steering wheel and pedal devices to generate operation signals; using a data processing and control software environment to perform the following operations: receiving the operation signals; converting the operation signals into control instructions, the control instructions including at least one of steering angle, throttle opening, brake opening, clutch state and gear signal; sending the control instructions to vehicle dynamics simulation software; receiving the vehicle dynamics state data corresponding to the vehicle to be tested from the vehicle dynamics simulation software; byte-packing the vehicle dynamics state data to form a binary data packet; sending the binary data packet to the three-dimensional graphics engine via a UDP protocol; receiving the control instructions by the vehicle dynamics simulation software, performing real-time dynamics calculation on the digital twin model of the vehicle to be tested based on the control instructions, and outputting the vehicle dynamics state data of the vehicle to be tested, the vehicle dynamics state data including global position coordinates, attitude angle, vehicle body speed, vehicle body angular velocity, and rotation speed and rotation angle of each wheel of the vehicle to be tested; using the three-dimensional graphics engine to perform the following operations: receiving the binary data packet, parsing, in accordance with the order, data type, and byte sequence consistent with the packing process, and restoring the vehicle dynamics state data; based on the restored vehicle dynamics state data, directly driving the digital twin model corresponding to the vehicle to be tested in the three-dimensional virtual scene to update the position, rotation attitude, and wheel state, and performing rendering output, the driving process avoiding using the built-in physics engine of the three-dimensional graphics engine for calculation; wherein the vehicle dynamics simulation software, the data processing and control software environment, and the three-dimensional graphics engine run on the same general-purpose computing device or a networked general-purpose computing device, realize real-time data interaction through the UDP protocol, and jointly construct a human-in-the-loop driving test closed-loop platform.

[0007] In some optional embodiments, the byte-packing process specifically includes: converting a plurality of vehicle dynamics state variables of double-precision floating-point type into a continuous unsigned 8-bit integer array in a predefined order; wherein the total length of the binary data packet is N×8 bytes, and the symbol N represents the number of packed double-precision floating-point variables.

[0008] In some optional embodiments, the process of parsing the binary data packet specifically includes: unpacking the received binary data packet in the same order, same data type, and same byte sequence as predefined in the packing process to restore the original double-precision floating-point type vehicle dynamics state data.

[0009] In some optional embodiments, the update of the rotation attitude specifically includes: The Euler angle or quaternion attitude data in the vehicle dynamics state data is converted into a quaternion format adopted by the three-dimensional graphics engine and directly assigned to a rotation attribute of the digital twin model to drive the attitude change of the digital twin model, thereby avoiding the use of a built-in simple physics engine of the three-dimensional graphics engine for calculation.

[0010] In some optional embodiments, the updating of the wheel state further comprises: creating a sub-object for each wheel of the digital twin model in the three-dimensional virtual scene and independently updating the steering angle local rotation and the rotation state around the axis of each wheel according to the restored wheel rotation angle and rotation speed data.

[0011] In some optional embodiments, the method further comprises: synchronizing the simulation calculation step length of the vehicle dynamics simulation software and the three-dimensional graphics engine by setting the simulation calculation step length, and implementing time synchronization by adding a timestamp in a UDP data packet and performing verification and adjustment.

[0012] In some optional embodiments, in the steps performed by using the data processing and control software environment, the steps further comprise: generating a force feedback signal according to the steering torque or road vibration data calculated by the vehicle dynamics simulation software and sending the force feedback signal to the force feedback steering wheel to provide physical tactile feedback.

[0013] According to a second aspect of the embodiments of the present specification, a high-fidelity human-machine interaction driving test platform based on digital twinning is provided, comprising: A vehicle dynamics simulation module is established by using TruckSim software and is configured to calculate and output vehicle state data including global position coordinates, attitude information, vehicle speed, vehicle angular velocity, rotation speed and rotation angle of each wheel of the vehicle in real time according to input driver control instructions or automatic driving control instructions. a data and control center module, which is built in the MATLAB / Simulink environment, is in communication connection with the vehicle dynamics simulation module, is used for directly calling and reading the vehicle state data output by TruckSim through an S-Function module, performing data format conversion and protocol communication processing on the vehicle state data, packing a plurality of double-type vehicle state variables into a byte array in a predefined order through a Byte Pack module, and then sending the byte array to an IP address and a port consistent with the virtual visualization and interaction module through a UDP Send module via a UDP protocol; meanwhile, the module is also connected with and reads the angle sensor of a Logitech G29 steering wheel, the pressure sensor of an accelerator pedal, the pressure sensor of a brake pedal, and the signal of a gear shifter through a JoystickInput plug-in, maps the physical rotation angle of the steering wheel to the steering input of TruckSim, converts the physical travel of the accelerator pedal and the brake pedal into throttle opening and brake opening, respectively, and inputs them into TruckSim, and transmits the gear signal to TruckSim for gear shifting; in addition, the module also generates a force feedback signal according to the steering torque or road reaction force calculated by TruckSim and sends the force feedback signal to the Logitech G29 steering wheel through a DirectX Force Feedback API; a virtual visualization and interaction module, which is built in the Unity 3D game engine, is in communication connection with the data and control center module through a UDP protocol, is used for creating a vehicle model matched with the vehicle geometry in TruckSim through a C# script, and configuring Wheel Collider, Rigidbody and Box Collider components for the vehicle model; the C# script receives byte array data packets from the UDP protocol in real time, parses the vehicle state data in the same order and data type as the sending end, and directly updates the position and rotation of the Transform component corresponding to the vehicle model in the Unity 3D scene, the vehicle body attitude angle, the vehicle body speed and the steering state of each wheel by using the parsed data, to realize the synchronous visualization of the TruckSim physical model and the Unity 3D virtual vehicle motion trajectory; a human-computer interaction device module, which adopts a Logitech G29 force feedback steering wheel and pedal kit, is connected with the data and control center module, is used for capturing the steering, throttle, brake and gear shifting operations of the driver in real time through the angle sensor and the pressure sensor thereof, and receiving the force feedback signal generated by the data and control center module to provide the driver with real steering torque feeling and road vibration tactile feedback; the vehicle dynamics simulation module, the data and control center module, the virtual visualization and interaction module and the human-computer interaction device module jointly constitute a human-in-the-loop driving test closed-loop platform.

[0014] In some optional embodiments, the platform further comprises a simulation step synchronization module for synchronizing the simulation steps of TruckSim and Unity 3D by uniform setting, and checking and adjusting by the timestamp in the UDP data packet to avoid data delay or jump.

[0015] According to a third aspect of the embodiments of the present specification, a high-fidelity human-in-the-loop driving test device based on digital twinning is provided, comprising: a vehicle dynamics simulation module configured to be implemented by high-precision vehicle dynamics simulation software, for real-time dynamics calculation according to the received control instructions, and outputting the dynamics state data of the vehicle; a data and control center module configured to be implemented by a data processing and control software environment, for receiving operation signals from the human-machine interaction device and converting them into control instructions recognizable by the vehicle dynamics simulation module, receiving vehicle dynamics state data from the vehicle dynamics simulation module, and sending the data after byte packaging through the UDP protocol; a virtual visualization and interaction module configured to be implemented by a three-dimensional graphics engine, for receiving and analyzing data packets through the UDP protocol, restoring the vehicle dynamics state data, and directly driving the vehicle model in the virtual scene to update the motion and synchronize the posture based on the restored data, while rendering the output; a human-machine interaction device, which is a force feedback steering wheel and pedal device, for collecting the physical operation of the driver and converting it into an electrical signal transmitted to the data and control center module; The vehicle dynamics simulation module, the data and control center module, and the virtual visualization and interaction module run on the same or networked general-purpose computing device and interact with each other through the UDP protocol, together forming a human-in-the-loop driving test closed-loop platform.

[0016] The beneficial effects of the embodiments of the present specification are as follows: The technical scheme of the present application accurately converts driving operations into control instructions through clear functional separation and cooperation, drives high-fidelity digital models to perform dynamics calculations, and directly drives the model motion in the virtual scene after efficiently encoding the precise state data generated by the calculations, ultimately successfully constructing a human-in-the-loop real-time test closed-loop platform running on general-purpose hardware, which can ensure high consistency between the virtual environment and the physical simulation, and provides a low-cost, high-precision solution for high-fidelity driving tests that does not rely on special real-time controller hardware. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to make the technical solutions in the embodiments or the related art clearer, the accompanying drawings needed in the embodiments or the related art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other accompanying drawings can be obtained by those skilled in the art without any creative effort.

[0018] Figure 1 The system architecture diagram of the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 2 The schematic diagram of the separate configuration of the wheels in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 3 The schematic diagram of setting the simulation step in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 4 The schematic diagram of the UDP data communication Simulink model in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 5 The schematic diagram of the related configuration of the UDP Send module in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 6 The schematic diagram of the related configuration of the Byte Pack module in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 7 The test schematic diagram of the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application in the city road, water-involved road surface, desert and snow-covered ground scenes respectively; Figure 8 The schematic diagram of the steering wheel reading Simulink model in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 9 The steering angle diagram in the automobile dynamics model simulation software TruckSim in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 10 The speed diagram in the automobile dynamics model simulation software TruckSim in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 11 The steering angle diagram in the Unity 3D software in the high-fidelity human-computer interaction driving test method based on digital twinning proposed in the technical solutions of the present application; Figure 12 Speed map in Unity 3D software in the high-fidelity human-machine interaction driving test method based on digital twinning proposed for the technical solutions of the present application; Figure 13 The specific flowchart of the technical solutions of the present application in the actual scene of the multi-axle all-wheel drive vehicle. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of one or more embodiments of the present application clearer, the technical solutions of one or more embodiments of the present application will be described clearly and completely below in conjunction with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of one or more embodiments of the present application.

[0020] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.

[0021] The present application provides a high-fidelity human-machine interaction driving test method based on digital twinning, which can include: Capturing the physical operation of the driver through the force feedback steering wheel and pedal device to generate an operation signal; Using the data processing and control software environment to perform the following operations: Receiving the operation signal; Converting the operation signal into a control instruction, the control instruction including at least one of a steering angle, an accelerator opening degree, a brake opening degree, a clutch state and a gear signal; Sending the control instruction to the vehicle dynamics simulation software; Receiving the dynamics state data corresponding to the vehicle to be tested from the feedback of the vehicle dynamics simulation software; Performing byte packing processing on the vehicle dynamics state data to form a binary data packet; Sending the binary data packet to the three-dimensional graphics engine through the UDP protocol; Receiving the control instruction using the vehicle dynamics simulation software, performing real-time dynamics calculation on the digital twinning model of the vehicle to be tested based on the control instruction, and outputting the dynamics state data of the vehicle to be tested, the dynamics state data including the global position coordinates, attitude angle, vehicle body speed, vehicle body angular velocity and rotational speed and angle of each wheel of the vehicle to be tested; Using the three-dimensional graphics engine to perform the following operations: receiving the binary data packet, parsing, restoring the vehicle dynamics state data in the order consistent with the packaging process, data type and byte sequence; Based on the restored vehicle dynamics state data, directly drive the digital twin model corresponding to the vehicle to be tested in the three-dimensional virtual scene to update the position, rotation attitude and wheel state, and render and output. The driving process avoids using the built-in physics engine of the three-dimensional graphics engine to calculate. Among them, the vehicle dynamics simulation software, the data processing and control software environment and the three-dimensional graphics engine run on the same or networked general computing device, realize real-time data interaction through UDP protocol, and jointly construct a human-in-the-loop driving test closed-loop platform.

[0022] The overall content of the foregoing technical solution will be described in detail below. First, the physical operation actions of the driver are captured in real time through the force feedback steering wheel and pedal devices, and converted into corresponding electrical signals, i.e. operation signals. These operation signals can accurately reflect the driver's control intention for the vehicle, including steering, acceleration, braking, etc. Then, the captured operation signals are processed by the data processing and control software environment. The software environment receives the above operation signals and converts them into control instructions that can be recognized by the vehicle dynamics simulation software. These control instructions can include at least one of steering angle, throttle opening, brake opening, clutch state and gear signal. The converted control instructions are sent to the vehicle dynamics simulation software in real time to drive subsequent simulation calculations. The vehicle dynamics simulation software receives the control instructions from the data processing and control software environment and performs high-precision real-time dynamics calculations on the pre-established digital twin model corresponding to the vehicle to be tested based on these instructions. After the calculation is completed, the software outputs the dynamics state data reflecting the real-time state of the vehicle. These data can include the global position coordinates (X, Y, Z) of the vehicle, the attitude angle (pitch, roll, yaw), the vehicle body speed (Vx, Vy, Vz), the vehicle body angular velocity (Wx, Wy, Wz), and the rotation speed and angle of each wheel.

[0023] After receiving the dynamics state data feedback from the vehicle dynamics simulation software, the data processing and control software environment will convert these data into a format suitable for network transmission. Specifically, it will perform byte packing processing on multiple double-precision floating-point type vehicle state variables in a strictly defined order, converting them into a continuous, unsigned 8-bit integer array, i.e. forming a compact binary data packet. Thereafter, the binary data packet is sent to the three-dimensional graphics engine through the UDP protocol. UDP protocol is chosen mainly for its connectionless and low-latency features, which can meet the stringent requirements of real-time simulation for data transmission speed.

[0024] The three-dimensional graphics engine receives the binary data packet transmitted through the UDP protocol, and unpacks the data packet in the same order, data type and byte sequence as the packing process, so as to accurately restore the original vehicle dynamics state data. Then, based on the restored high-precision vehicle dynamics state data, the three-dimensional graphics engine directly drives the digital twin model corresponding to the vehicle to be tested in the virtual scene to update the state, and the driving process is completely based on the received real physical calculation results, including updating the position coordinates and rotation attitude of the vehicle model (by converting Euler angle or quaternion data into the four quaternion format suitable for the engine and directly assigning values), and independently updating the steering angle and rotation state of each wheel. It is crucial that this whole process completely avoids the use of the simple physics engine built-in the three-dimensional graphics engine, ensuring that the motion of the virtual vehicle is highly consistent with the calculation results of the vehicle dynamics simulation software, thereby guaranteeing the engineering accuracy and physical reality of the simulation. Finally, the engine renders the updated virtual scene in real time and synchronously outputs the current state parameters of the vehicle, providing immersive visual feedback for the driver.

[0025] During the implementation of the technical solution of the present application, the vehicle dynamics simulation software, the data processing and control software environment, and the three-dimensional graphics engine can all run on the same general-purpose computing device or through network connection, they interact with each other through the above-mentioned UDP protocol for efficient and real-time data exchange, and together form a complete human-in-the-loop driving test closed-loop platform. The solution can enable the physical operation of the driver to affect the calculation results of the high-precision vehicle model in real time, and immediately reflect in the immersive visualization environment, thereby providing a high-fidelity, cost-effective and flexible solution for testing, verification and evaluation of autonomous driving systems.

[0026] Based on the foregoing technical solution, the present application also provides some more specific technical solutions, which are described below.

[0027] In an optional embodiment, the byte packing process can specifically include: Converting a plurality of vehicle dynamics state variables of double-precision floating-point type into a continuous unsigned 8-bit integer array in a predefined order; wherein the total length of the binary data packet is N×8 bytes, and the symbol N represents the number of packed double-precision floating-point variables.

[0028] In the technical solution of the embodiment, firstly, the vehicle dynamics state variables to be packed are determined, which are all double-precision floating-point numbers, and are derived from vehicle dynamics simulation software, such as TruckSim, to calculate key state data in real time, including but not limited to global position coordinates (X, Y, Z) of the vehicle, attitude angles (pitch angle, roll angle, yaw angle), vehicle body speed (Vx, Vy), vehicle body angular velocity (Wx, Wy, Wz), and the rotational speed and angle of each wheel, etc. The specific variables need to be determined according to the state parameters to be transmitted in actual testing.

[0029] Secondly, the selected double-precision floating-point variables are arranged in a predefined fixed order, which needs to be preset and fixed in a data processing and control software environment, such as MATLAB / Simulink, for example, the order can be arranged as “global position X→global position Y→global position Z→pitch angle→roll angle→yaw angle→vehicle body speed Vx→vehicle body speed Vy→vehicle body angular velocity Wx→vehicle body angular velocity Wy→vehicle body angular velocity Wz→left front wheel angle→right front wheel angle→…→left rear wheel speed→right rear wheel speed”, and the order needs to be strictly consistent with the order when the subsequent three-dimensional graphics engine analyzes the data packet, to ensure the correspondence of data transmission and analysis.

[0030] Then, a byte packing tool in the data processing and control software environment is used, such as the Byte Pack module of MATLAB / Simulink, to perform conversion operation, which will process each double-precision floating-point variable after sorting. Since each double-precision floating-point number occupies 8 bytes when stored in a computer, the tool will extract 8 bytes of data for each variable in the form of binary storage in its memory, and then concatenate these byte data to form a complete unsigned 8-bit integer array. For example, if a double-precision floating-point number is stored as 8 consecutive byte data in the memory, such as 0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC, 0xDE, 0xF0, the tool will directly extract these 8 bytes and include them in the array, and the byte data of multiple variables are concatenated in order according to the arrangement order, to finally form a continuous byte stream.

[0031] Finally, the total length of the binary data packet is determined, since each double-precision floating-point variable corresponds to 8 bytes, the total length of the data packet is the product of the number N of packed double-precision floating-point variables and 8 bytes, i.e. the total length is N×8 bytes. For example, when 11 double-precision floating-point variables need to be transmitted, the total length of the binary data packet formed after the above packing process is 11×8=88 bytes, which can be sent by the data processing and control software environment to the three-dimensional graphics engine through the UDP protocol, to provide a basis for subsequent analysis and vehicle state synchronization.

[0032] The entire process above integrates the scattered vehicle dynamics state data into a compact continuous byte stream through fixed sequential arrangement, byte-by-byte extraction and splicing, which not only adapts to the real-time transmission requirement of the UDP protocol, but also provides clear format basis for accurate analysis of the receiving end.

[0033] In an optional embodiment, the process of analyzing the binary data packet can specifically include: The received binary data packet is unpacked in the same order, same data type and same byte sequence as predefined in the packing process to restore the original vehicle dynamics state data of the double-precision floating-point type.

[0034] In an optional embodiment, the process of analyzing the binary data packet can be implemented in the following way: The binary data packet transmitted by the UDP protocol is received by using the network communication function of a three-dimensional graphics engine such as Unity 3D, which is in the form of an unsigned 8-bit integer array, and its total length strictly corresponds to the setting when packing. After receiving the data packet, its integrity is first confirmed to ensure that no transmission loss or damage has occurred, and the length conforms to the preset specification of N x 8 bytes. Then, the parsing operation is performed by the pre-written C# script in the three-dimensional graphics engine: first, split by fixed byte length, specifically split the received unsigned 8-bit integer array by 8 bytes as a unit, and each 8 bytes corresponds to one double-precision floating-point variable before packing. Second, maintain the consistency of the order, that is, the parsing order of the split 8-byte data is strictly consistent with the arrangement order of the double-precision floating-point variable before packing. Third, restore the data type, that is, convert each 8-byte unsigned 8-bit integer data to a double-precision floating-point type by data conversion methods such as BitConverter class in C#, and restore its original numerical value, which can include the global position coordinates of the vehicle, attitude angles (pitch angle, roll angle, yaw angle), vehicle body speed, vehicle body angular velocity, and wheel rotation speed and angle, etc. Fourth, maintain the consistency of the byte sequence, strictly follow the same byte sequence as the packing process in the parsing process, that is, the storage order of the data in the memory, so that the high and low bits of the multi-byte data are arranged without deviation.

[0035] Through the above parsing operation, the binary data packet is restored to the original double-precision floating-point type vehicle dynamics state data, which is used to drive the vehicle model in the three-dimensional virtual scene to update the motion and synchronize the attitude.

[0036] In an optional embodiment, the update of the rotation attitude can specifically include: Convert the Euler angle or quaternion attitude data in the vehicle dynamics state data into a quaternion format adopted by the three-dimensional graphics engine, and directly assign it to the rotation attribute of the digital twin model to drive its attitude change, thereby avoiding the use of the built-in physics engine of the three-dimensional graphics engine for calculation.

[0037] In an optional embodiment, the updating of the rotation attitude is specifically achieved by the following method: First, extract the original data for describing the vehicle attitude from the dynamics state data output by the vehicle dynamics simulation software such as TruckSim. These data can include Euler angles such as pitch angle, roll angle, yaw angle or quaternion parameters, which directly reflect the inclination and rotation state of the vehicle in three-dimensional space, such as the pitch angle of the vehicle head when climbing uphill, the roll degree of the vehicle body when turning, etc.

[0038] Subsequently, in a three-dimensional graphics engine such as Unity 3D, a pre-written program script is used to convert the extracted attitude data. Specifically, if the original data is Euler angle, it needs to be converted into the quaternion format adopted by the three-dimensional graphics engine according to the coordinate system specifications of the engine, including the direction of the coordinate axis, the definition of the angle sign, etc. In the conversion process, the mathematical calculation rules of the engine must be strictly followed to ensure that the angle values and directions of pitch, roll and yaw correspond accurately to the original data after conversion. If the original data is quaternion, it is directly adjusted according to the quaternion format requirements of the engine to ensure that the data structure conforms to the parsing rules of the engine.

[0039] Then, the converted quaternion data is directly assigned to the rotation attribute of the digital twin model corresponding to the vehicle to be tested in the three-dimensional virtual scene, such as the rotation parameter of the model Transform component. By directly modifying the attribute parameter, the digital twin model is driven to produce corresponding attitude changes, such as real-time adjustment of the pitch, roll or yaw angle of the vehicle body according to the converted quaternion.

[0040] Throughout the process of the present application, the built-in physics engine of the three-dimensional graphics engine is not enabled for attitude calculation, i.e. the attitude change of the digital twin model is completely controlled by the converted quaternion data above, rather than being generated by the physics engine based on force analysis and other simulation calculations. In this way, the attitude of the virtual vehicle is highly synchronized with the physical calculation results output by the vehicle dynamics simulation software, avoiding deviations caused by the simplification of the engine's own physical model.

[0041] In an optional embodiment, the method can further include synchronizing the simulation calculation step length of the vehicle dynamics simulation software and the three-dimensional graphics engine by setting them synchronously, and implementing time synchronization by adding a timestamp in the UDP data packet and performing verification and adjustment.

[0042] In an optional embodiment, the process of achieving time synchronization in the method can be implemented in the following way: On the one hand, the basic time alignment is achieved by synchronizing the simulation calculation step length of the vehicle dynamics simulation software and the three-dimensional graphics engine. In the three-dimensional graphics engine, such as Unity 3D, enter the time configuration interface of its project settings, and set the fixed time step to a preset fixed value, for example, 0.005. This step directly determines the time interval of vehicle state update and physical calculation in the three-dimensional graphics engine. At the same time, in the vehicle dynamics simulation software, set the time step of its simulation calculation to the same value as the three-dimensional graphics engine, such as also set to 0.005 seconds, or set to an integer multiple of the fixed step length, to ensure that the calculation frequency and state update frequency of the two in unit time are consistent, fundamentally reducing the time deviation caused by the difference in calculation period, and matching the rhythm of vehicle dynamics calculation and virtual scene rendering with each other.

[0043] On the other hand, the time synchronization accuracy is further optimized by adding a timestamp in the UDP data packet and performing dynamic checking and adjustment. In the data processing and control software environment, such as MATLAB / Simulink, when the vehicle dynamics state data is byte packed, the precise time information of the data generation is additionally encapsulated as part of the data packet, so that each UDP data packet contains not only the position, attitude and other state data of the vehicle, but also the specific time of the data generation. The three-dimensional graphics engine receives the data packet, extracts the timestamp through the pre-written program script, and compares it with the local simulation time in real time. If the deviation between the timestamp and the local time is within the preset range, such as less than 5 milliseconds, the data is applied to update the state of the virtual vehicle in normal time sequence. If the deviation is too large, such as more than 5 milliseconds, the data usage time is adjusted through interpolation calculation or delayed application to avoid jump or lag of the virtual vehicle. If the timestamp is later than the local time, the data is applied after a short waiting period to ensure that the update rhythm of the vehicle state in the virtual scene and the calculation rhythm of the vehicle dynamics simulation software are highly synchronized.

[0044] In the technical solution of the present application, the above two methods are combined for implementation. Through the dual mechanism of fixed step control and dynamic time checking, the consistency of the vehicle dynamics simulation software and the three-dimensional graphics engine in the time dimension can be ensured, and the real-time synchronization of the motion trajectory, attitude change and physical simulation result of the virtual vehicle can be guaranteed, thereby reducing the visual deviation or control delay caused by time asynchronization.

[0045] In an optional embodiment, in the steps executed by the data processing and control software environment, a force feedback signal can also be generated according to the steering torque or road vibration data calculated by the vehicle dynamics simulation software, and sent to the force feedback steering wheel to provide physical tactile feedback.

[0046] In the embodiment, first, the key mechanical data calculated by the vehicle dynamics simulation software is obtained. The vehicle dynamics simulation software synchronously outputs mechanical parameters related to the interaction between the steering system and the road surface during real-time dynamics calculation based on control instructions, including but not limited to steering torque, such as the reaction torque transmitted by the steering mechanism when the driver turns the steering wheel, and road vibration data, such as the vibration frequency and amplitude generated by the tire when the vehicle is driving on a bumpy road, gravel road, or joint, which are transmitted in real time to the data processing and control software environment, such as MATLAB / Simulink, through a data interaction link.

[0047] Secondly, the signal for the force feedback steering wheel is generated in the data processing and control software environment. After receiving the mechanical data, the data processing and control software environment can convert and process the data through pre-written algorithms, such as proportional amplification and frequency filtering. For steering torque data, according to the force feeling output range of the force feedback steering wheel, such as the maximum torque range of Logitech G29, it is mapped to the current or pulse signal recognizable by the steering wheel, ensuring that the torque size is proportional to the feedback force intensity. For example, when the calculated steering torque increases during sharp turns, the generated signal makes the steering wheel feedback force increase, simulating the tactile feeling of "the faster the steering, the heavier the steering wheel" in real driving. For road vibration data, the vibration frequency and amplitude are converted into periodic pulse signals to make the steering wheel vibrate at corresponding frequencies and amplitudes. For example, when driving through a speed bump, the vibration data triggers a high-frequency pulse signal, and the steering wheel produces a short vibration.

[0048] Then, the force feedback signal is sent to the force feedback steering wheel through a special interface. The data processing and control software environment transmits the generated force feedback signal to the force feedback steering wheel, such as Logitech G29, through an interface that supports force feedback communication, such as DirectX Force Feedback API, which ensures the real-time transmission of the signal and makes the steering wheel respond to the changes in mechanical data within milliseconds.

[0049] Finally, the force feedback steering wheel produces physical touch according to the received signals. The motor inside the steering wheel adjusts the output torque or vibration frequency according to the signal parameters, that is, when receiving the steering torque signal, the motor outputs the corresponding torque to make the steering wheel generate resistance, and the driver can feel the "heavy feeling" matching the vehicle dynamics state when turning, and when receiving the road vibration signal, the motor generates vibration through fast forward and reverse rotation to transmit the touch of road unevenness to the driver. In the technical solution of the application, the driver can intuitively perceive the steering resistance and road conditions of the vehicle, and the immersion and authenticity of the driving simulation are enhanced.

[0050] It should be understood that in the method described in one or more embodiments of the present specification, the order of some steps can be adjusted according to actual needs, or some steps can be omitted.

[0051] The application also provides a high-fidelity human-computer interaction driving test platform based on digital twinning to solve the technical difficulties that high-precision vehicle dynamics simulation software, three-dimensional graphics engine, human-computer interaction device and data processing tool are difficult to organically cooperate in the prior art, that is, in the prior art, TruckSim can provide engineering-level dynamics calculation, but cannot directly realize low-latency data interaction with Unity 3D, Unity 3D has immersive visualization capability, but its built-in physical model cannot match the precision of TruckSim, force feedback devices such as Logitech G29 often have interface incompatibility problems with simulation software, and multi-module synchronization is prone to data delay or jump, and needs to rely on high-priced real-time controllers to force linkage. The platform, through modular design, data bidirectional closed-loop circulation and lightweight synchronization mechanism, organically integrates the vehicle dynamics simulation module TruckSim, the data and control center module MATLAB / Simulink, the virtual visualization and interaction module Unity 3D, the human-computer interaction device module Logitech G29 and the simulation step synchronization module, without a special real-time controller, the full-link closed loop of driver operation, dynamics calculation, visualization feedback and haptic feedback can be realized. The platform can include the following modules: The vehicle dynamics simulation module is established by using TruckSim software, and is used to calculate and output vehicle state data including global position coordinates, attitude information, vehicle speed, vehicle angular velocity, wheel speed and wheel angle of the vehicle according to the input driver control instruction or automatic driving control instruction in real time; a data and control center module, which is built in the MATLAB / Simulink environment, is in communication connection with the vehicle dynamics simulation module, is used for directly calling and reading the vehicle state data output by TruckSim through an S-Function module, performing data format conversion and protocol communication processing on the vehicle state data, packing a plurality of double-type vehicle state variables into a byte array in a predefined order through a Byte Pack module, and then sending the byte array to an IP address and a port consistent with the virtual visualization and interaction module through a UDP Send module via a UDP protocol; meanwhile, the module is also connected with and reads the angle sensor of a Logitech G29 steering wheel, the pressure sensor of an accelerator pedal, the pressure sensor of a brake pedal, and the signal of a gear shifter through a JoystickInput plug-in, maps the physical rotation angle of the steering wheel to the steering input of TruckSim, converts the physical travel of the accelerator pedal and the brake pedal into throttle opening and brake opening, respectively, and inputs them into TruckSim, and transmits the gear signal to TruckSim for gear shifting; in addition, the module also generates a force feedback signal according to the steering torque or road reaction force calculated by TruckSim and sends the force feedback signal to the Logitech G29 steering wheel through a DirectX Force Feedback API; a virtual visualization and interaction module, which is built in the Unity 3D game engine, is in communication connection with the data and control center module through a UDP protocol, is used for creating a vehicle model matched with the vehicle geometry in TruckSim through a C# script, and configuring Wheel Collider, Rigidbody and Box Collider components for the vehicle model; the C# script receives byte array data packets from the UDP protocol in real time, parses the vehicle state data in the same order and data type as the sending end, and directly updates the position and rotation of the Transform component corresponding to the vehicle model in the Unity 3D scene, the vehicle body attitude angle, the vehicle body speed and the steering state of each wheel by using the parsed data, to realize the synchronous visualization of the TruckSim physical model and the Unity 3D virtual vehicle motion trajectory; a human-machine interaction device module, which adopts a Logitech G29 force feedback steering wheel and pedal kit, is connected with the data and control center module, is used for capturing the steering, throttle, brake and gear shifting operations of the driver in real time through the angle sensor and the pressure sensor thereof, and receiving the force feedback signal generated by the data and control center module to provide the driver with real steering torque feeling and road vibration tactile feedback; The vehicle dynamics simulation module, the data and control center module, the virtual visualization and interaction module and the human-machine interaction device module jointly constitute a human-in-the-loop driving test closed-loop platform.

[0052] The platform can also include a simulation step synchronization module for synchronizing the simulation steps of TruckSim and Unity 3D by uniformly setting the simulation steps, and checking and adjusting through the timestamp in the UDP packet to avoid data delay or jump.

[0053] In the high-fidelity human-machine interaction driving test platform based on digital twinning provided in the above embodiment, in terms of the vehicle dynamics simulation module, the embodiment adopts TruckSim software to construct a digital twinning model of a vehicle to be tested, configures physical parameters such as vehicle mass, wheelbase, and tire attributes, and dynamics characteristic models such as suspension deformation and tire-pavement mechanical interaction, sets input variables such as steering angle and throttle opening to receive driver or automatic driving control instructions, and outputs state data such as global position coordinates, attitude information, vehicle body speed, and wheel rotational speed and angle, and the output frequency and data are matched with the sampling frequency of the data and control center module. At the same time, in view of the problem that TruckSim data in the prior art is mostly stored in local files, is difficult to transmit in real time, and is difficult to be compatible in format, in the embodiment, the API interface is directly called through the S-Function module of MATLAB / Simulink, and real-time data reading is realized. The data and control center module is built in the MATLAB / Simulink environment, on the one hand, vehicle state data is read through the TruckSim S-Function module, a plurality of double type variables are packed into a uint8 byte array in a pre-defined order through a Byte Pack module, and then are sent to Unity 3D through a UDP Send module at an IP address and a port consistent with the virtual visualization module, on the other hand, signals of a Logitech G29 device (including a steering wheel, throttle / brake / clutch pedal, and gear lever) are read through a Joystick Input plug-in, physical rotation angle of the steering wheel is mapped to a steering angle of TruckSim, physical travel of the pedal is converted into throttle / brake opening, gear signals are converted into gear codes, and based on steering torque calculated by TruckSim or road reaction force, force feedback signals are generated through a DirectX Force Feedback API, and the one-stop integration can solve the problem of non-compatible interface between G29 and multiple software and asynchronous feedback in the prior art, and does not need additional drivers.The virtual visualization and interaction module in the embodiment relies on the Unity 3D game engine, creates a virtual model matching the TruckSim vehicle geometry through a C# script, adds a Rigidbody, a Box Collider component to the vehicle body, adds a WheelCollider component to each wheel and associates the vehicle body, and constructs a UDP client through a C# script to receive data packets, parses and restores the vehicle state data in the same order and data type as the sending end, bypasses the built-in physics engine to directly drive the virtual vehicle Transform component to update the position, posture and wheel state, realizes synchronous visualization, and builds a virtual scene with multiple road types through a Terrain editor and a road plug-in, configures a weather system and a day and night cycle effect, which can solve the problems of large trajectory deviation and easy data packet parsing errors caused by relying on the built-in physics engine of Unity 3D in the prior art. The human-computer interaction device module selects a Logitech G29 force feedback steering wheel and pedal kit, captures the driver's steering, throttle and other operations through angle sensors and pressure sensors and transmits signals, and receives force feedback signals to provide tactile feedback. The force feedback is directly generated based on the dynamics data of TruckSim, avoiding intermediate filtering and solving the problem of disconnection between feedback and dynamics state. The simulation step synchronization module realizes synchronization by setting a fixed time step in Unity 3D and setting the same simulation step as TruckSim, and adding a millisecond-level timestamp to the UDP data packet, comparing the timestamp at the receiving end and supplementing the missing data through linear interpolation.

[0054] The coordination process between the modules in the platform is described below. The modules in the platform provided by the embodiment form a complete man-in-the-loop test closed loop through bidirectional data flow. The specific coordination logic is as follows: the driver performs steering, throttle, brake or gear shifting operations through the Logitech G29, the sensors of the device capture the operation signals and transmit them to MATLAB; MATLAB converts the operation signals into control instructions (such as steering angle, throttle opening) recognizable by TruckSim and sends them to the vehicle dynamics simulation module, TruckSim calculates the vehicle state data in real time according to the control instructions, and outputs the data back to MATLAB, which packages the vehicle state data into UDP data packets and sends them to Unity 3D, and generates force feedback signals based on the dynamics data of TruckSim and sends them to the Logitech G29; Unity 3D drives the virtual vehicle to update the position, posture and other states, which are observed by the driver through the display screen, and the Logitech G29 provides the driver with steering torque sensation or road vibration tactile feedback according to the force feedback signals; the simulation step synchronization module monitors the simulation steps and data timestamps of TruckSim and Unity 3D in real time, so that the whole process has low delay and ensures stable operation of the closed loop.

[0055] Based on the same idea, the embodiments of the present specification also provide a device corresponding to the above method, which can include a vehicle dynamics simulation module constructed by TruckSim software, used to calculate the kinematics and dynamics state data of the vehicle in real time according to the received control instructions, wherein the state data includes the global position, attitude information, vehicle body speed, vehicle body angular velocity, and the rotation speed and angle of each wheel of the vehicle; a data and control center module built in the MATLAB / Simulink environment, connected with the vehicle dynamics simulation module, the virtual visualization and interaction module, and the human-machine interaction device module, respectively, used to receive and analyze the state data output by the vehicle dynamics simulation module and to encapsulate, and also used to receive the control instructions output by the human-machine interaction device module and convert them into signals recognizable by the vehicle dynamics simulation module; a virtual visualization and interaction module constructed by Unity 3D engine, connected with the data and control center module, used to receive the vehicle state data processed by the data and control center module, and drive the vehicle model in the virtual scene to move based on the state data, wherein the virtual scene includes terrain, road, and environmental special effects; a human-machine interaction device module including a steering wheel, a pedal, and a gear, connected with the data and control center module, used to collect the operation instructions of the driver and transmit them to the data and control center module; wherein the data and control center module and the virtual visualization and interaction module perform data transmission through the UDP protocol.

[0056] The specific content of the high-fidelity human-machine interaction driving test platform based on digital twinning provided in the foregoing will be described in detail below, Figure 1 The architecture diagram of the high-fidelity human-machine interaction driving test scheme based on digital twinning proposed for the technical solution of the present application, the core of the platform lies in the architecture design based on the concept of digital twinning, which is composed of four closely coordinated modules: vehicle dynamics simulation, data and control center, virtual visualization and interaction, and human-machine interaction device. These modules can jointly build an efficient, accurate, and immersive real experience test environment.

[0057] The platform fully absorbs the core concept of digital twinning of virtual and real interaction in modern simulation testing, and successfully overcomes the limitations of traditional single simulation mode in accuracy and realism by skillfully integrating professional tools that are good at different fields. It realizes real-time data interaction and control feedback by building a digital mirror of the physical entity, thereby achieving seamless connection and collaborative work between the physical world and the digital world. This design not only improves the high fidelity of simulation, but also provides unprecedented realism and immersion for human-in-the-loop testing.

[0058] First, let's explain the vehicle dynamics simulation, which is mainly carried out by the TruckSim software. With its outstanding high-precision physical model and multi-body dynamics calculation capabilities, it accurately simulates the real kinematics and dynamics behavior of an autonomous vehicle under various complex working conditions. Whether it is the slight deformation of the suspension system or the complex mechanical interaction between the tire and the road, TruckSim can perform highly realistic reproduction. As the core driver of the system, it calculates the vehicle's attitude, speed, acceleration, and the rotation speed and angle of each wheel in real time based on the received control instructions. These calculations from the high-fidelity physical model lay a solid foundation for the authenticity and engineering accuracy of the entire platform, ensuring the scientific rigor of virtual vehicle motion.

[0059] Next, let's explain the data and control center, which is built in the MATLAB / Simulink environment. It undertakes multiple key responsibilities such as data relay, format conversion, control instruction distribution, and human-computer interaction device management. MATLAB uses its powerful data processing capabilities to analyze and package the original vehicle state data from TruckSim in real time, and transmits it to the virtual visualization module through efficient communication protocols. At the same time, it is also the "interpreter" between human-computer interaction devices and vehicle dynamics models, capable of recognizing real-time input signals from Logitech G29 steering wheel, pedals, and gears, and accurately converting them into control instructions recognizable by TruckSim. In addition, the center also reserves interfaces for future deployment of autonomous driving control algorithms, allowing it to receive simulated sensor data and generate vehicle control instructions, thereby achieving comprehensive algorithm-in-the-loop testing.

[0060] Next, let's explain virtual visualization and interaction, which is the core of the Unity 3D game engine. It fully utilizes the strong advantages of Unity 3D in three-dimensional graphics rendering and immersive scene construction to provide a realistic and detailed virtual driving environment for human-computer interaction driving tests. Unity 3D is not only responsible for rendering high-precision road models, traffic flow, and environmental elements in real time, but also accurately drives the vehicle model in the virtual scene based on the vehicle state data received from the data and control center, making the vehicle's motion trajectory, attitude, and even wheel rotation highly synchronized with the physical calculation results of TruckSim. Through this module, testers or drivers can directly observe the real-time behavior of the vehicle and interact with the virtual environment, thereby greatly improving the realism and user experience of the test.

[0061] The human-machine interaction device is described below. This module is an important part of the platform for implementing human-in-the-loop functions, represented by the Logitech G29 steering wheel and pedal kit. It serves as a physical bridge between the driver and the virtual world, providing users with a highly realistic driving experience. Through precise steering wheel angle sensors, throttle / brake pedal pressure sensors, and gear recognition, the device can capture the driver's intentions in real time. These real-time driver input data are transmitted to the data and control center, which in turn affects the vehicle dynamics simulation, forming a complete closed loop from driver operation to vehicle physical response and visual feedback. In addition, the force feedback function provided by the Logitech G29 can provide real physical touch to the driver according to the road reaction force or steering torque calculated by TruckSim, further enhancing the immersion of the simulation and the driver's perception of the vehicle state.

[0062] The contents of the virtual-real combined simulation are described below.

[0063] First, the content of scene generation is described. The realism of the virtual driving scene is the key to improving the immersion of the driver and the effectiveness of the test. Scene generation in this test platform is mainly completed in the Unity 3D environment, including terrain generation, road construction, and environmental effects.

[0064] Terrain generation mainly relies on the Terrain terrain editor in Unity 3D. This tool constructs realistic terrain through height maps, textures, and detail objects, and supports real-time editing and optimization.

[0065] There are various methods for road construction. Three-dimensional road models can be built using 3D modeling software such as 3D Max, Blender, and Solidworks, then exported in fbx format, which can be directly imported into the Asset folder of Unity 3D. After importing, it becomes a prefab, supporting free and full disassembly, which is convenient for subsequent scene editing and modification. High-precision map data (such as OpenStreetMap and high-precision point cloud data) can also be directly imported into Unity 3D. These data can be converted into three-dimensional models, including road grids, traffic signs, streetlights, and buildings. Through programmatic generation or manual refinement, a virtual scene similar to the real environment can be constructed. Here, the Road Architect plugin from Unity 3D Asset Store is used for custom road drawing, which can quickly draw complex roads such as S-bends, snake test roads, and city crossroads that meet the requirements of autonomous vehicle testing.

[0066] To further enhance the sense of immersion, various environmental effects can be added. For example, building a complex weather system can achieve real-time dynamic weather changes such as rain, snow, fog, wind, etc., and simulate their effects on visibility and road friction coefficient. It can also achieve real day and night cycle, simulate the movement of the sun and the moon, realize day and night alternation, affect the scene lighting. And realize the visual processing of the road surface characteristics, although the dynamics calculation is completed in TruckSim, but the road surface material in Unity 3D can visually distinguish asphalt, soil, wet road surface, etc., to provide visual cues for the driver.

[0067] The following describes the content of vehicle simulation control, which is the core of the virtual-real combination between TruckSim and Unity 3D. This platform adopts a master-slave cooperative control strategy, that is, TruckSim as the master unit for high-precision dynamics calculation, and Unity 3D as the slave unit responsible for visualization driving.

[0068] The following describes the content of TruckSim vehicle model configuration. In TruckSim, detailed vehicle models need to be established according to the actual parameters of the vehicle to be tested, such as mass, number of axles, wheelbase, center of mass position, wheel motor parameters, suspension characteristics, tire characteristics, steering system, transmission system, etc. The accuracy of these model parameters directly determines the physical realism of the simulation results.

[0069] Then configure the output variables of TruckSim to ensure that it can output all necessary vehicle state data, such as global position (X, Y, Z coordinates), attitude information (pitch, roll, yaw), vehicle speed (Vx, Vy), vehicle angular velocity (Wx, Wy, Wz), wheel speed and angle of each wheel. Configure the input variables of TruckSim to receive driver control commands such as steering angle, throttle, brake, clutch, gear from Simulink or automatic driving control commands.

[0070] The following describes the content of Unity 3D vehicle model driving. In Unity 3D, a vehicle model matching the geometry of the vehicle in TruckSim is created, and Wheel Collider, Rigidbody and BoxCollider plugins are added to the body and wheels for rigid body and collision configuration. This model can be a simplified rigid body, whose movement is no longer calculated independently by the physics engine of Unity 3D. By writing a C# script, the model mounted with the script in Unity 3D will receive and parse UDP data packets from Simulink in real time. The parsed data can be directly used to update the position, rotation, vehicle attitude angle, vehicle speed and steering state information of each wheel of the corresponding Transform component in Unity 3D.

[0071] To update the vehicle state information in Unity 3D correctly, the necessary vehicle control code is also needed in the script, which is based on the following calculation formula.

[0072] The calculation formula of the speed error in the PID controller is: ; Where represents the speed error, represents the target vehicle speed (m / s), represents the current vehicle speed (m / s).

[0073] The required motor torque is calculated based on the speed error, and the calculation formula is: ; Where, represents the motor torque applied to the wheel, represents the speed control gain, represents the rolling resistance torque.

[0074] And the rolling resistance torque is approximated here, simplified as a term related to mass and gravity, and the calculation formula is: ; Where, represents the direction symbol of the current speed, represents the vehicle mass, represents the acceleration of gravity, represents the empirical coefficient, and R is the wheel radius.

[0075] To achieve pose synchronization, the Euler angles or quaternions output by TruckSim need to be converted to Quaternion in Unity 3D and assigned to the rotation of the vehicle model.

[0076] TruckSim will output the actual angle and speed of each wheel, so to achieve wheel synchronization, a separate sub-object needs to be created for each wheel in Unity 3D, and their local rotation angle and rotation around the axis need to be adjusted in real time based on the received data, so as to achieve visually consistent wheel movement with the physical model. The separate configuration of the wheels is shown in Figure 2 .

[0077] In addition, it is also necessary to ensure that the simulation time steps of TruckSim and Unity 3D are synchronized as much as possible to avoid data delay or jump. The simulation step size can be directly modified in the software settings, or the timestamp in the UDP data packet can be checked and adjusted. The specific simulation step size settings are shown in Figure 3 .

[0078] The content of the UDP data communication is described below.

[0079] First, the content set by MATLAB is described. The UDP protocol is very suitable for data transmission in real-time simulation scenarios due to its connectionless, low delay, and real-time characteristics. The platform uses UDP data communication as the data communication method between TruckSim, MATLAB, and Unity 3D. The Simulink model established for UDP data communication is shown in Figure 4 To use the UDP protocol for data communication, the local IP address and the listening port number corresponding to the software process PID need to be checked. In this paper, the local IP address and the listening port number are queried using the computer's built-in program resource monitor. The S-Function module of Simulink can directly call the vehicle state parameters in TruckSim, so data transmission from TruckSim to MATLAB does not need to go through UDP communication. To achieve data transmission from MATLAB to Unity 3D, the Byte Pack and UDP Send modules in the Simulink model need to be used.

[0080] The UDP Send module receives the byte array from the Byte Pack module and sends it to Unity 3D. The correct IP address and port number need to be used, and they should be consistent with the IP address and port number set in the Unity 3D script. The related configuration of the UDP Send module is shown in Figure 5

[0081] The Byte Pack module is used to pack the vehicle state parameters in the S-Function module into bytes. The double type data is used in this technical solution. Double type data is stored in the form of floating point numbers in the computer, usually occupying 8 bytes. For example, when you set up eleven double type byte arrays, it means that the Byte Pack module will convert the eleven independent double values you input into a continuous byte stream according to a certain order, and then output a uint8 array with a length of 11x8=88 bytes. This array is the binary representation of all double data in memory. This method realizes efficient and real-time network transmission of high-precision floating point data, which is a key step in realizing virtual vehicle trajectory synchronization and human-computer interaction control. The related configuration of the Byte Pack module is shown in Figure 6

[0082] ​​​The following describes the settings of the Unity 3D end, and the data format of the unified UDP packet is the key. Usually, the structure or byte array is used to pack all the vehicle state variables (such as position X, Y, Z, Euler angle Roll, Pitch, Yaw, speed Vx, Vy, and wheel angle) in the pre-defined order. Usually, binary format is used for transmission to avoid string secondary parsing.

[0083] At the receiving end, in the C# script of Unity 3D, you need to parse this 88-byte array in the same order, data type, and byte sequence to correctly restore the original eleven double values.

[0084] In order to verify the human-in-the-loop simulation capability and user experience of the developed test platform, the virtual driving test based on Logitech G29 steering wheel is carried out. The following describes the test settings, Logitech G29 data acquisition and control mapping, high-fidelity test of the human-machine interaction driving test platform, and test results and analysis.

[0085] From the perspective of test settings, the driver hardware is selected as Logitech G29 force feedback steering wheel, accelerator / brake / clutch pedal, and H-shaped gear lever, aiming to restore the real driving operation experience; the virtual scene is constructed in Unity 3D, covering straight roads, curves, ups and downs, and various traffic signs. This paper constructs multiple complex scenes for testing, such as desert, snow, water road, grassland, urban road, and highway, as shown in Figure 7 .

[0086] And it realizes human-machine interaction driving test under different weather conditions, such as rain, snow, fog, and sunny weather. The key environmental parameters such as road friction coefficient are accurately configured in TruckSim to ensure the rigor of physical simulation; a multi-axle ten-wheel vehicle model is selected for testing, and its basic parameters are carefully calibrated in TruckSim to ensure the authenticity of its dynamics behavior.

[0087] From the perspective of data acquisition and control mapping of Logitech G29, the connection and data acquisition of Logitech G29 force feedback steering wheel is realized through the "Joystick Input" plug-in of MATLAB / Simulink, which realizes real-time data reading of Logitech G29 device. The specific Simulink model diagram is shown in Figure 8 .

[0088] Through the above model, the physical rotation angle of the Logitech G29 steering wheel is read and mapped to the steering input of TruckSim. The physical travel of the accelerator pedal is converted to 0-100% throttle opening and input to the throttle control port of TruckSim. The brake pedal is similar to the accelerator pedal, converted to 0-100% brake opening and input to the brake control port of TruckSim. If the TruckSim model requires a manual clutch, the clutch pedal signal will also be captured and transmitted. The H-shaped gear selector of the G29 can identify the current gear engaged, and this information is passed to TruckSim for gear shifting. Even the various function buttons on the steering wheel can be identified.

[0089] The Logitech G29 steering wheel supports force feedback, which greatly enhances the driving realism. Through Simulink, a force feedback signal can be generated in real time according to the steering torque calculated by TruckSim or the vibration caused by road unevenness and sent to the G29 steering wheel. For example, the steering wheel becomes heavy when turning sharply, and the steering wheel vibrates when driving over a bumpy road. This requires Simulink to communicate with G29 through a specific interface such as DirectX Force Feedback API.

[0090] From the perspective of high-fidelity testing of human-machine interactive driving test platforms, to verify the feasibility of high-fidelity human-machine interactive driving test platforms based on digital twinning, the following experiments were conducted: 30 drivers with different driving styles were selected, each of whom used the Logitech G29 steering wheel to control the car as the test system input, and conducted human-machine interactive driving tests in different scenarios and different weather conditions. After the control information was calculated by the TruckSim car dynamics simulation platform, the car state parameters were output to the Unity 3D end in real time through UDP communication, and the Unity 3D end updated the vehicle running picture in real time to feedback to the driver, allowing the driver to manipulate under this condition. Complex conditions such as S-bend, snake experiment, etc. The total simulation time is 20s, and the driver's control of the virtual vehicle in Unity 3D through the G29 steering wheel, system response delay, vehicle speed and angle error, and overall immersion are observed.

[0091] The test results and analysis are described below, from Figures 9-12The comparison can be clearly seen that the TruckSim and the Unity 3D exported vehicle speed and wheel angle change with time curve basically coincide, that is, the corresponding vehicle speed and wheel angle of TruckSim running simulation and the corresponding vehicle speed and wheel angle value in Unity 3D corresponding to the same time frame are basically perfect, the error is very small, so in the same coordinate system, the vehicle running track in TruckSim under the control of the driver and the virtual vehicle running track in the Unity 3D scene also maintain high level consistency. In terms of system response delay, the technical scheme of the application records the end-to-end time from the driver operating the Logitech G29 to the TruckSim response and finally reflected in the Unity 3D picture. The experimental data shows that the whole closed-loop delay usually keeps within 5 milliseconds, which is very important for human-in-the-loop simulation, ensuring that the driver can feel real-time and smooth feedback, and the low overhead characteristics of the UDP protocol play a decisive role in this process.

[0092] Finally, in terms of human-computer interaction experience, 30 drivers all think that the combination of Logitech G29 force feedback steering wheel and TruckSim high-precision physical model provides an immersive driving experience infinitely close to reality. The force feedback of the steering wheel, the linear response of the pedal and the accurate and real-time response of the vehicle to the driving instruction enable the driver to effectively perceive the vehicle dynamics and easily complete complex driving tasks such as accurate cornering and emergency braking.

[0093] To verify the technical effect of the technical scheme of the application, the applicant also compares the technical scheme of the application with other technical schemes in terms of test efficiency, scene fidelity and other dimensions. The comparison results are shown in Table 1 as follows: ; The virtual scene simulation in the above table can provide realistic visual and physical environment input, and such method is mainly suitable for perception algorithm development and decision planning test in controllable virtual environment. The traditional hardware-in-the-loop method realizes the function, reliability and robustness of the physical controller by connecting the real vehicle controller hardware (such as ECU) to the virtual simulation environment through real-time signal interface (CAN, LIN, Ethernet, etc.) under the real-time constraint of millisecond delay. Limited by the physical interface protocol and the complexity of the simulation model, the verification capability of the whole vehicle level, multi-ECU deep interaction or complex vehicle dynamics-environment coupling scene is limited. The technical scheme of the present application proposes a high-fidelity human-computer interaction driving test method based on digital twinning, which combines the advantages and disadvantages of virtual scene and traditional hardware-in-the-loop method well, not only guarantees high scene fidelity and short test period, but also reduces the economic cost to the minimum, that is, saves the high cost of using traditional high-performance real-time controller, and at the same time, can guarantee the test and verification of the whole vehicle level, multi-ECU deep interaction or complex vehicle dynamics-environment coupling scene under the real-time constraint of millisecond delay.

[0094] Meanwhile, as shown in Figure 13 , Figure 13For the technical solution of the present application in the actual scene of the multi-axle all-wheel drive vehicle, the figure shows the working process of the multi-axle all-wheel drive vehicle in the loop driving system based on digital twinning, taking the human-vehicle-environment closed loop as the core, and each module realizes the complete cycle of real vehicle operation-data transmission-dynamics calculation-virtual mapping-visual feedback through signal and control flow. Specifically, the driver operates the Logitech steering wheel, such as turning the steering wheel, simulating the throttle or brake operation, inputs the steering angle, power demand and other driving instructions, and initiates the driving action. The operation signal of the Logitech steering wheel is transmitted to the three-axle six-wheel vehicle with drive-by-wire function through drive-by-wire technology, directly controlling the motion state of the real vehicle, such as steering, acceleration, deceleration, etc., to complete the driving of the real vehicle according to the instructions. Then the sensors of the three-axle six-wheel vehicle, such as speed sensor, attitude sensor, wheel speed sensor, etc., collect its own dynamic signals, including vehicle speed, vehicle attitude, axle angle and speed, etc., and receive and transmit through the signal interaction medium responsible for data transmission and format conversion, so as to realize the extraction and transmission of real vehicle state data. Then the signal interaction medium transmits the real vehicle dynamic data to the dynamics simulation module, which is based on a high-fidelity vehicle model that needs to consider the complex dynamics characteristics such as tire-road interaction, multi-axle angle distribution, vehicle attitude change, etc. The dynamics simulation module can calculate the dynamics response of the vehicle in real time, such as tire force, vehicle attitude adjustment, multi-axle power distribution logic, etc., to complete the digital simulation of the dynamics of the real vehicle. The data calculated by the dynamics simulation is used to drive the simulation vehicle in the virtual scene, realizing digital twinning mapping, that is, making the motion state of the virtual vehicle, such as position, attitude, wheel system action, etc., real-time synchronized with the three-axle six-wheel vehicle in the physical world, and restoring the driving behavior of the real vehicle in the virtual environment. Finally, the virtual scene building module generates a virtual driving picture synchronized with the motion of the real vehicle, such as road, vehicle perspective, surrounding environment, etc., which is transmitted to the driver through visual feedback, so that the driver can obtain an immersive and consistent virtual driving experience, just like directly controlling the real vehicle on the real road. Ultimately, a closed loop of driver control-real vehicle motion-virtual feedback-driver re-control is formed. Through the whole link of hardware control-data flow-digital twinning-visual feedback, the system realizes the deep integration of human-vehicle-virtual environment, and provides a virtual-real interactive platform for the driving test of multi-axle all-wheel drive vehicles, such as dynamics verification and scene simulation.

[0095] The technical scheme of the application provides a high-fidelity human-computer interaction driving test scheme, which realizes efficient real-time transmission of data among TruckSim, MATLAB and Unity 3D through a UDP protocol, and effectively solves the technical challenge of integration of a traditional high-precision dynamic model and a high-performance graphics engine. Specifically, TruckSim is responsible for providing digital dynamic twin data of a vehicle, Unity 3D provides a high-resolution virtual driving scene and immersive visualization, and MATLAB serves as a core data relay and control hub, responsible for data format conversion, protocol communication and interface management of human-computer interaction devices (such as Logitech G29 steering wheel). Through this cooperative working mode, the scheme not only successfully replaces the traditional high-performance real-time controller, realizing accurate synchronization of virtual vehicle trajectory and physical model, but also successfully builds a virtual driving simulation environment supporting high-fidelity human-computer interaction, providing an economical, efficient, safe and flexible solution for development, testing and verification of autonomous vehicles.

[0096] Obviously, the above embodiments of the application are only examples for clearly illustrating the application, and are not intended to limit the implementation modes of the application. Those skilled in the art can make various obvious changes, readjustments and substitutions without departing from the protection scope of the application. It is unnecessary and impossible to enumerate all the implementation modes. Any modification, equivalent substitution and improvement within the spirit and principle of the application shall be included in the protection scope of the claims of the application.

Claims

1. A high-fidelity human-computer interactive driving test method based on digital twins, characterized in that, Includes the following steps: The driver's physical input is captured through force feedback steering wheel and pedal devices to generate operation signals; Perform the following operations using a data processing and control software environment: Receive the operation signal; The operation signal is converted into a control command, which includes at least one of steering angle, throttle opening, brake opening, clutch status, and gear position signal. The control command is sent to the vehicle dynamics simulation software; Receive dynamic state data corresponding to the vehicle under test from the vehicle dynamics simulation software; The vehicle dynamics state data is packaged into bytes to form a binary data packet; The binary data packet is sent to the 3D graphics engine via the UDP protocol; The vehicle dynamics simulation software receives the control command and performs real-time dynamic calculations on the digital twin model of the vehicle under test based on the control command, outputting the dynamic state data of the vehicle under test. The dynamic state data includes the global position coordinates, attitude angles, vehicle speed, vehicle angular velocity, and the rotational speed and angle of each wheel of the vehicle under test. Perform the following operations using a 3D graphics engine: The binary data packet is received and parsed according to the same order, data type, and byte order as the packaging process to reconstruct the vehicle dynamics state data; Based on the restored vehicle dynamics state data, the digital twin model corresponding to the vehicle under test in the three-dimensional virtual scene is directly driven to update its position, rotation attitude and wheel state, and then rendered and output. The driving process avoids the use of the physics engine built into the three-dimensional graphics engine for calculation. The vehicle dynamics simulation software, the data processing and control software environment, and the 3D graphics engine all run on the same or networked general-purpose computing device, and achieve real-time data interaction through the UDP protocol, together forming a human-machine-in-the-loop driving test closed-loop platform.

2. The high-fidelity human-computer interaction driving test method based on digital twins according to claim 1, characterized in that, The byte packing process specifically includes: Multiple vehicle dynamics state variables of type double-precision floating-point are converted into a continuous unsigned 8-bit integer array in a predefined order; wherein the total length of the binary data packet is N×8 bytes, and the symbol N represents the number of double-precision floating-point variables packaged.

3. The high-fidelity human-computer interaction driving test method based on digital twins according to claim 2, characterized in that, The process of parsing the binary data packet specifically includes: The received binary data packets are unpacked according to the same order, data type, and byte order as predefined in the packaging process to restore the original double-precision floating-point vehicle dynamics state data.

4. The high-fidelity human-computer interaction driving test method based on digital twins according to claim 1, characterized in that, The update of the rotational attitude specifically includes: The Euler angles or quaternion attitude data in the vehicle dynamics state data are converted into the quaternion format used by the 3D graphics engine and directly assigned to the rotation attribute of the digital twin model to drive its attitude change, thereby avoiding the use of the simple physics engine of the 3D graphics engine for calculation.

5. The high-fidelity human-computer interaction driving test method based on digital twins according to claim 4, characterized in that, The update of the wheel state also includes: based on the restored wheel angle and speed data, creating a separate sub-object for each wheel of the digital twin model in the three-dimensional virtual scene, and independently updating the local rotation and rotation around the axis of each wheel.

6. The high-fidelity human-computer interaction driving test method based on digital twins according to claim 1, characterized in that, The method further includes: synchronizing the simulation calculation step size of the vehicle dynamics simulation software and the three-dimensional graphics engine by synchronously setting the simulation calculation step size, and adding timestamps to UDP data packets and performing verification and adjustment to achieve time synchronization.

7. The high-fidelity human-computer interaction driving test method based on digital twins according to claim 1, characterized in that, The steps performed using the data processing and control software environment further include: generating a force feedback signal based on the steering torque or road vibration data calculated by the vehicle dynamics simulation software, and sending it to the force feedback steering wheel to provide physical tactile feedback.

8. A high-fidelity human-computer interactive driving test platform based on digital twins, characterized in that, include: The vehicle dynamics simulation module, built using TruckSim software, is used to calculate and output vehicle state data in real time, including the vehicle's global position coordinates, attitude information, vehicle speed, vehicle angular velocity, and the rotational speed and angle of each wheel, based on the input driver control commands or autonomous driving control commands. The Data and Control Center module, built using the MATLAB / Simulink environment, communicates with the vehicle dynamics simulation module. It directly calls and reads vehicle state data output by TruckSim via the S-Function module, performs data format conversion and protocol communication processing on the vehicle state data, and packages multiple double-type vehicle state variables into a byte array according to a predefined order using the Byte Pack module. This array is then sent via the UDP Send module to the same IP address and port as the virtual visualization and interaction module using the UDP protocol. Simultaneously, this module connects to and reads signals from the Logitech G29 steering wheel angle sensor, accelerator pedal pressure sensor, brake pedal pressure sensor, and gear selector via the Joystick Input plugin. It maps the physical rotation angle of the steering wheel to the steering input of TruckSim, converts the physical travel of the accelerator and brake pedals into accelerator and brake openings respectively, and transmits the gear signal to TruckSim for gear shifting. Furthermore, this module also uses DirectX Force Feedback to calculate the steering torque or road reaction force based on TruckSim. The API generates a force feedback signal and sends it to the Logitech G29 steering wheel; The virtual visualization and interaction module, built using the Unity 3D game engine, communicates with the data and control center module via UDP protocol. It is used to create vehicle models matching the vehicle geometry in TruckSim using a written C# script, and to configure Wheel Collider, Rigidbody, and Box Collider components for these models. The C# script receives byte array data packets from the UDP protocol in real time, parses the vehicle state data in the same order and data type as the sender, and uses the parsed data to directly update the position and rotation of the Transform component corresponding to the vehicle model in the Unity 3D scene, the vehicle's attitude angle, vehicle speed, and the steering state of each wheel, achieving synchronized visualization of the TruckSim physical model and the Unity 3D virtual vehicle's motion trajectory. The human-computer interaction device module adopts the Logitech G29 force feedback steering wheel and pedal kit, which is connected to the data and control center module. It is used to capture the driver's steering, accelerator, brake and gear shifting operations in real time through its angle sensor and pressure sensor, and receive the force feedback signal generated by the data and control center module to provide the driver with real steering torque feeling and road vibration tactile feedback. The vehicle dynamics simulation module, data and control center module, virtual visualization and interaction module, and human-machine interaction equipment module together constitute a human-machine-in-the-loop driving test closed-loop platform.

9. The high-fidelity human-computer interaction driving test platform based on digital twins according to claim 8, characterized in that, The platform also includes a simulation step size synchronization module, which is used to avoid data delays or jumps by uniformly setting the simulation step size of TruckSim and Unity 3D, and by verifying and adjusting the timestamps in UDP packets.

10. A high-fidelity human-computer interactive driving test device based on digital twins, characterized in that, include: The vehicle dynamics simulation module is configured to be implemented by high-precision vehicle dynamics simulation software, which is used to perform real-time dynamics calculations based on the received control commands and output the vehicle's dynamic state data. The data and control center module is configured to be implemented by a data processing and control software environment. It is used to receive operation signals from the human-machine interaction device and convert them into control commands that can be recognized by the vehicle dynamics simulation module. It also receives vehicle dynamics state data from the vehicle dynamics simulation module and sends the data through UDP protocol after byte packing. The virtual visualization and interaction module is configured to be implemented by a 3D graphics engine. It is used to receive and parse data packets via UDP protocol, restore vehicle dynamic state data, and directly drive the vehicle model in the virtual scene to perform motion updates and attitude synchronization based on the restored data, while performing rendering output. The human-machine interaction device is a force feedback steering wheel and pedal device, which is used to collect the driver's physical operations and convert them into electrical signals for transmission to the data and control center module; The vehicle dynamics simulation module, data and control center module, and virtual visualization and interaction module run on the same or networked general-purpose computing device and interact with each other via the UDP protocol, together forming a human-machine-in-the-loop driving test closed-loop platform.

Images