Vehicle driving training system and method, and storage medium

Abstract

The invention discloses a vehicle driving training system and method and a storage medium, and relates to the technical field of vehicle simulation driving, the system comprises a cockpit, an AR interaction device, a training device and a signal acquisition module, the AR interaction device is used for obtaining a real scene image of the cockpit and pose data of a driver; sending the real scene image and the pose data to training equipment; the signal acquisition module is used for acquiring a vehicle control signal input by a driver; sending the vehicle control signal to a training device; the training device is used for determining a virtual scene image in a preset virtual training scene according to the vehicle control signal and the pose data; fusing the virtual scene image with a preset virtual driving environment image and a real scene image to obtain a fused scene image; sending the fused scene image to AR interaction equipment; the AR interaction device is further configured to display the fused scene image to the driver. By applying the scheme provided by the embodiment of the invention, the training benefit can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle driving simulation, and in particular to a vehicle driving training system, method and storage medium. Background Art

[0002] Traditional real-life driving training is affected by many factors, such as high equipment costs, complex venues, possible equipment damage caused by misoperation, and difficulties in training support needs. At present, driving simulation training can be carried out based on simulators, using driving training simulators with semi-physical simulation components to present virtual training scenes through LCD displays. However, semi-physical driving simulators have the disadvantages of low training immersion and are easily disturbed by the external environment, resulting in low training efficiency. Summary of the invention

[0003] The present invention aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one object of the present invention is to provide a vehicle driving training system, method and storage medium to improve the training efficiency.

[0004] According to a first aspect of an embodiment of the present invention, a vehicle driving training system is provided, the system comprising: a driving cockpit, an AR interaction device, a training device, and a signal acquisition module, wherein: The AR interaction device is used to obtain a real-scene image of the cockpit and the driver's posture data; and send the real-scene image and the posture data to the training device; The signal acquisition module is used to obtain the vehicle control signal input by the driver; and send the vehicle control signal to the training device; The training device is used to determine a virtual scene image in a preset virtual training scene according to the vehicle control signal and the posture data; fuse the virtual scene image with a preset virtual driving environment image and the real scene image to obtain a fused scene image; and send the fused scene image to the AR interaction device; The AR interaction device is also used to display the fused scene image to the driver.

[0005] According to a second aspect of an embodiment of the present invention, a vehicle driving training method is provided, the method comprising: Obtain the real-life image of the cockpit and the driver’s posture data; Acquiring a vehicle control signal input by the driver; Determine a virtual scene image in a preset virtual training scene according to the vehicle control signal and the posture data; fuse the virtual scene image with a preset virtual driving environment image and the real scene image to obtain a fused scene image; The fused scene image is displayed to the driver.

[0006] According to a third aspect of an embodiment of the present invention, there is provided a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned vehicle driving training method is implemented.

[0007] In the solution provided by the embodiment of the present invention, the virtual scene image, the virtual driving environment image and the real scene image are superimposed and fused to achieve the combination of virtual and real, and a realistic scene situation can be created to provide a driving training environment for trainees, which can not only enhance the immersion of training, but also provide operation feedback close to the real vehicle, and the operation method and feeling are closer to the real vehicle driving situation. In this way, a more efficient, safer and more practical vehicle driving training method can be provided for training, shortening the real vehicle training time to improve the training efficiency.

[0008] Through this integration, a realistic driving training environment can be constructed, and through the input of vehicle control signals, the driver can interact with the scene, providing the driver with a highly realistic training experience to effectively improve driving professional skills and the ability to cope with different driving scenarios.

[0009] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a structural schematic diagram of a first vehicle driving training system provided by an embodiment of the present invention; Figure 2 is a structural schematic diagram of a second vehicle driving training system provided by an embodiment of the present invention; Figure 3 is a schematic diagram of a scene of vehicle driving training provided by an embodiment of the present invention; Figure 4 is a structural schematic diagram of a third vehicle driving training system provided by an embodiment of the present invention; Figure 5 is a flow chart of a video perspective method provided by an embodiment of the present invention; Figure 6 is a flow chart of a scene image fusion method provided by an embodiment of the present invention; Figure 7 is a flowchart of an image rendering method provided by an embodiment of the present invention; Figure 8 is a schematic diagram of an image field of view calculation method provided by an embodiment of the present invention; Fig. 9It is a flow chart of a vehicle driving training method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0012] The vehicle driving training system and method according to the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0013] In one embodiment of the present invention, see Figure 1 , provides a vehicle driving training system, including: a driving cockpit, an AR (Augmented Reality) interaction device, a training device, and a signal acquisition module, wherein: AR interactive equipment, used to obtain the real scene image of the cockpit and the driver's posture data; send the real scene image and posture data to the training equipment; A signal acquisition module is used to obtain the vehicle control signal input by the driver; and send the vehicle control signal to the training device; The training device is used to determine a virtual scene image in a preset virtual training scene according to the vehicle control signal and the posture data; fuse the virtual scene image with the preset virtual driving environment image and the real scene image to obtain a fused scene image; and send the fused scene image to the AR interaction device; The AR interactive device is also used to display fused scene images to the driver.

[0014] The real scene image is an image representing the interior environment of a real cockpit.

[0015] Figure 2 The schematic diagram of the structure of the cockpit is shown. The cockpit is a carrier that includes various vehicle components such as a head-mounted VR (Virtual Reality) device, a seat, and a driving control device. There are pulleys under the carrier and glass walls on the sides.

[0016] AR interactive devices include: image acquisition equipment, head-mounted VR display, and positioning tracker.

[0017] In this embodiment, the image acquisition device is Figure 2The binocular stereo camera in the system is mainly used to collect video stream information that simulates the binocular vision of human eyes. In addition, the binocular stereo camera can obtain the depth information of the image by comparing the pixel displacement of the left and right eye images. The depth information represents the distance of each pixel point perpendicular to the image plane. The head-mounted VR display can present the superimposed virtual training scene and real scene information. Specifically, the head-mounted VR display can be a pair of VR glasses. The positioning tracker can capture the posture data of the driver's head, so as to match the virtual camera and restore the relative posture and position of the driver in the virtual scene.

[0018] Pose data can include the driver's position in the actual scene and the steering angle of the head.

[0019] The position of the driver in the actual scene can be represented by a preset image coordinate system. The actual scene, the virtual scene image, and the virtual driving environment image can be represented by a three-dimensional image.

[0020] The actual scene and real scene image include the actual objects such as seats and steering wheels. Figure 1 The actual environment outside the cockpit can be removed in advance or masked with a green screen because it will be replaced by the virtual scene image during training. When the position of the seat fixed relative to the cockpit is used as the origin of the image coordinate system of the real scene image, the actual objects all have fixed coordinate values. Therefore, these coordinate values ​​can be recorded in advance, so that the unrecorded coordinate values ​​are determined as the actual environment that needs to be removed.

[0021] The virtual camera is used to simulate the driver's field of view from the first-person perspective in the virtual training scene.

[0022] Figure 2 The system also includes a signal acquisition module and a visual simulation computer. The visual simulation computer is a training device for obtaining a fused scene image.

[0023] The usage of the driving control device and the instrument device is described in the subsequent embodiments.

[0024] The virtual driving environment image is an image representing a virtual vehicle component. For example, a virtual rearview mirror, a reversing mirror, a door handle, etc. can be drawn in the image to form a virtual operating environment. Figure 2 The cockpit shown combined can fully present the structure of a vehicle.

[0025] Taking the rearview mirror as an example, the virtual driving environment image can record the pixel value and pixel position of the rearview mirror in the image, including the position and depth in the vertical and horizontal directions on the screen.

[0026] The virtual training scene can represent the external environment where the vehicle is located, for example, a wilderness with hills, trees, roads, etc. Figure 3 Among them, the frustum between the near clipping plane and the far clipping plane represents the field of view of the first perspective position.

[0027] The calculation method of the field of view size is as follows Figure 8 Embodiment of the invention.

[0028] The vehicle can have a preset initial position in the virtual training scenario.

[0029] The fusion process includes converting the image data in the virtual scene image, the virtual driving environment image and the real scene image into a single image in the same coordinate system, namely, the fused scene image. The specific fusion process can be found in the subsequent embodiments.

[0030] In the solution provided by the embodiment of the present invention, the virtual scene image, the virtual driving environment image and the real scene image are superimposed and fused to achieve the combination of virtual and real, and a realistic scene situation can be created to provide a driving training environment for trainees, which can not only enhance the immersion of training, but also provide operation feedback close to the real vehicle, and the operation method and feeling are closer to the real vehicle driving situation. In this way, a more efficient, safer and more practical vehicle driving training method can be provided for training, shortening the real vehicle training time to improve the training efficiency.

[0031] Through this integration, a realistic driving training environment can be constructed, and through the input of vehicle control signals, the driver can interact with the scene, providing the driver with a highly realistic training experience to effectively improve driving professional skills and the ability to cope with different driving scenarios.

[0032] The following describes a specific implementation method for determining a virtual scene image in a preset virtual training scene.

[0033] In one embodiment, the first perspective position of the driver in the virtual training scene may be determined based on the posture data; Displaying a first virtual scene image within a visible range of a first viewing angle position in a virtual training scene; The first viewing angle position is updated according to the vehicle control signal to obtain a second viewing angle position; A second virtual scene image located within a visible range of a second viewing angle position in the virtual training scene is displayed.

[0034] The positioning tracker obtains the driver's posture, including the position and rotation angle relative to the origin. According to the correspondence between the origin of the real scene image and the origin in the virtual training scene image, the first-person perspective position is obtained by transforming the position of the origin, and the rotation angle of the posture is also used to obtain the angle equivalent to the driver's observation in the virtual scene image.

[0035] The correspondence between the origin and the origin used in the above process can be preset. For example, in the virtual scene image, the position of the seat in the vehicle is also set as the origin to align with the real scene image.

[0036] Depend on Figure 3 In the embodiment shown, the image contained in the field of view of the first viewing angle position is the first virtual scene image. The second virtual scene image is similar to the first virtual scene image.

[0037] In this embodiment, by determining the virtual scene image, only the virtual training scene is updated to simulate the situation where the external environment of the vehicle changes during the movement of the vehicle, so that the driver can see different landscapes.

[0038] In contrast, the relative position of the environment inside the car relative to the driver, whether it is the real objects in the real scene image or the virtual objects in the virtual driving environment image, is fixed. After setting the position of the origin of the image coordinate system of the real scene image and the virtual driving environment image, by setting the objects in the two images to have a fixed relative position to the origin, these objects can be directly displayed at the preset position of the VR glasses.

[0039] Specifically, when the position of the seat relative to the cockpit remains unchanged, the position of the origin may be the position of the seat.

[0040] The following describes how to obtain a fused scene image.

[0041] Determine the conversion relationship between the image coordinate systems of the virtual scene image, the virtual driving environment image, and the real scene image; According to the determined conversion relationship, a first conversion image of the virtual scene image in the coordinate system where the real scene image is located is obtained, and a second conversion image of the virtual driving environment image in the coordinate system where the real scene image is located is obtained; A fused scene image is obtained based on the real scene image, the first converted image and the second converted image.

[0042] For real-life images, an image coordinate system can be pre-selected so that the selected image coordinate system is consistent with the Unity coordinate system where the virtual scene image is located, for example, both are left-hand coordinate systems; and the origin position is set to be consistent, for example, both are set to the seat position as the origin, and the conversion relationship can be determined based on the coordinates of the same object in different coordinate systems. The specific process of calculating the conversion relationship is shown in the subsequent embodiments.

[0043] It can be seen from the above embodiments that the image coordinate system of the virtual scene image can be the image coordinate system of the three-dimensional virtual training scene, and the origin of the image can use the origin preset in the virtual training scene, that is, the seat position.

[0044] The conversion relationship between the image coordinate systems of the real scene image and the virtual driving environment image can also be obtained in a similar manner.

[0045] The first conversion image is composed of pixel values ​​in the virtual scene image, and the pixel points corresponding to the partial pixel values ​​represent objects in the virtual scene image, such as trees and obstacles in the virtual training scene, that is, virtual objects. The positions of the pixel points corresponding to the partial pixel values ​​are positions obtained by conversion based on the positions in the virtual scene image and the conversion relationship between the obtained virtual scene image and the real scene image.

[0046] The second conversion image is similar to the first conversion image, and the difference lies only in the difference in the names and concepts such as the virtual scene image and the virtual driving environment image.

[0047] In the same coordinate system, the pixels of the real scene image, the first transformed image and the second transformed image form an image, which is a fused scene image.

[0048] The above implementation method converts the virtual scene image and the virtual driving environment image into the image coordinate system of the real scene image to obtain a fused scene image. In other embodiments, the virtual scene image and the real scene image may also be converted into the image coordinate system of the virtual driving environment image to obtain a fused scene image, similar to the above process. The embodiments of the present invention are not limited to this.

[0049] Since the virtual scene image, virtual driving environment image and real scene image are converted into the same coordinate system to form a composite image, the driver can simultaneously observe the complete vehicle structure composed of virtual and real components and the external training scene displayed by the virtual scene image through the fusion of the scene image, providing an immersive training experience.

[0050] In one embodiment, when obtaining the fused scene image, the occlusion relationship between objects in each image is also detected, as follows: Determine the depth information of the real scene image, the first conversion image, and the second conversion image to determine the virtual objects that are not blocked in the first conversion image and the second conversion image; and obtain a fused scene image based on the pixel data of the real scene objects and the pixel data of the virtual objects in the real scene image.

[0051] Occlusion is determined based on the depth information of each object. For example, when converted to the same image coordinate system, with the X and Y axes as the axes in the width and height directions of the image, and the Z axis as the axis perpendicular to the image. If the coordinate values of the X and Y axes of the positions of the pixels of the real object A and the positions of the pixels of the virtual object B are the same, and it is determined through the depth information that the coordinate value on the Z axis is A < B, it is considered that B is farther from the driver, thus detecting occlusion. Then, in the fused scene image, the pixels of the virtual object B are removed, and only the unoccluded real object and virtual object are displayed.

[0052] The unoccluded virtual objects include: completely unoccluded or partially unoccluded virtual objects. For partially unoccluded virtual objects, the occluded parts can also be measured according to the depth information, similar to the detection method of the above AB objects, so as to remove the partially occluded pixels.

[0053] In addition, for detected collision objects, such as when the coordinate values of the pixels of A and B on the X, Y, and Z axes are the same, the fused scene image can be adjusted by moving the position of the virtual object. For example, in the image, B is moved above the position represented by the original coordinates.

[0054] Specifically, the binocular stereo camera captures the depth and normal map of the real objects in the surrounding environment in real time and integrates them into the Unity rendering pipeline for collision detection with virtual objects.

[0055] In addition, when the Unity renders the fused scene image, shadows can also be projected for the virtual objects to create a more realistic scene effect.

[0056] In one embodiment, the vehicle control signal is input via at least one of the following devices: The driving control device, the gearshift control device, and the instrument device.

[0057] The driver can input control signals regarding the throttle, brake, and steering through the driving control device; the gearshift control device is used to input control signals indicating gear engagement or disengagement; the instrument device is used to input control signals for vehicle power control, vehicle startup, power supply for various devices, and turn signal control.

[0058] In one embodiment, the system further includes an auditory feedback device and a tactile feedback device. The auditory feedback device can be headphones for providing auditory information to the driver; the tactile feedback device can be a handle with a vibration function for providing tactile information to the driver, such as simulating object collision with vibration. Through the feedback of multiple senses, the driver can obtain a more immersive training experience and can effectively exercise the driver's ability to handle scenarios.

[0059] Figure 4A third structural schematic diagram of the vehicle driving training system provided by an embodiment of the present invention.

[0060] Among them, the control terminal includes a monitoring display and a control computer.

[0061] The instructor can use the guidance and control computer to issue training courses and specify the driver's driving tasks in the training courses. In addition, the instructor can use the guidance and control computer to control, monitor, guide and handle emergency situations of the training system, so as to achieve dynamic management of the entire training process. For example, voice commands can be sent to guide the driver or provide handling suggestions.

[0062] The guidance and control computer can receive the fused scene image and synchronize it to the preset monitoring display, that is, it can obtain the scene image seen by the driver in real time through the monitoring display, making it easier for the instructor to manage the training process.

[0063] On the driving side, the positioning tracker is used to obtain the driver's posture data; the VR glasses are used to display scene images to the driver, including the first scene image and the second scene image in the aforementioned embodiment.

[0064] Binocular stereo cameras are used to obtain real scene images.

[0065] The signal acquisition module obtains the vehicle control signal input by the driver from the driving control device, the speed control device, and the instrument device. As shown in the figure, the driving control device can provide throttle, brake, and steering signals, which are represented by analog AD, and the speed control device and the instrument device can transmit speed control signals through the CAN bus.

[0066] After the signal acquisition module obtains the vehicle control signal, it uses Ethernet to transmit the vehicle control signal to the driving training computer, that is, the training device in the aforementioned embodiment. The training device is installed with training software for obtaining fusion scene data. The binocular stereo camera and VR glasses can interact with the driving training computer via a USB cable. For example, the binocular stereo camera transmits real-scene image data to the driving training computer, and the VR glasses can transmit the posture data obtained by the positioning tracker installed on the VR glasses to the driving training computer via a USB cable. In addition, the driving training computer transmits the scene image to the VR glasses via a video cable.

[0067] The fused scene data can be obtained by superimposing it based on the video perspective, as shown in the schematic diagram. Figure 5 As shown, the binocular stereo camera captures the real scene of the driving operation environment to obtain real scene image data.

[0068] The driving training computer provides a virtual camera scene, including a virtual scene image and a virtual driving environment image, and performs virtual-real fusion to obtain fused scene data. The scene image is then determined based on the head posture data of the trainee.

[0069] When the head posture changes, the driving training computer can also adjust the virtual camera scene according to the position change of the head posture, re-fuse the virtual and the real, and obtain new fused scene data.

[0070] The specific fusion process includes: Figure 6 shown.

[0071] Among them, select the coordinate system and determine the coordinate origin, that is, select the image coordinate system of the real scene image and determine the coordinate origin of the image coordinate system. The binocular stereo camera uses a three-dimensional Cartesian coordinate system to specify the position and direction. The left-handed coordinate system is used in Unity. To keep the coordinate system one, set the binocular stereo camera to a left-handed coordinate system.

[0072] Camera position tracking is achieved by detecting camera movement through built-in sensors. That is, the position of the binocular stereo camera is tracked, and a virtual camera is set at the corresponding position in the virtual training scene to obtain the field of view. The binocular stereo camera has built-in sensors such as accelerometers, gyroscopes, and magnetometers, which can be used to detect the head position and movement direction of the trainee; the sensor can perform reliable high-frequency measurements of the acceleration and acceleration of the camera in motion; the binocular stereo camera uses computer vision and stereo SLAM technology to obtain the position and direction of the camera in space, providing complete 6DOF position tracking; in the driving training cabin, the body and head can be moved freely, and the camera can track the position.

[0073] The real scene information is collected through the binocular stereo camera to obtain the real scene image. The binocular stereo camera can capture high-definition 3D video with a wide field of view and output two synchronized left and right video streams in a side-by-side format on USB3.0; Unity reads the video stream information and loads each frame image into the memory. The frame image of the video stream is the real scene image.

[0074] The pixel displacement between the left and right images is compared to evaluate the depth and normal information. After evaluation, the depth information and normal information are integrated into the Unity rendering pipeline to determine the collision and occlusion of virtual objects with real objects, and to project Unity lighting effects.

[0075] Unity uses a binocular stereo camera to calculate the depth and normal information of each pixel in the image, loads the data into the video memory, sets the rendering state of the primitives, and sends DrawCall instructions to the GPU for image rendering; the Unity rendering pipeline process includes the application stage, the geometry stage, and the rasterization stage; the main task of the application stage is to provide the required geometric information for the GPU's rendering operation, that is, to output the rendering primitives; in the geometry stage, the GPU processes the rendering primitives, performs vertex-by-vertex and polygon-by-polygon operations, and transforms the vertex coordinates into the screen space; in the rasterization stage, the rasterization stage mainly converts the data transmitted in the previous stage, such as vertex position, depth information, normal direction, viewing direction, etc., into pixels on the screen and renders the final image.

[0076] The specific Unity rendering process is as follows Figure 7 shown.

[0077] Among them, the driving simulation training software is the software on the training device. In the application stage, the real-scene video is loaded, and the pixel depth and normal information of the real-scene video are calculated; and the model data is loaded. The model data describes the virtual driving environment image and the virtual scene image through three-dimensional coordinates, thereby preparing the scene data; in this case, the fused scene image can be obtained and the rendering state can be set, that is, the material, texture, shader and other information required for rendering are set, and the rendering primitives are output. The vertex data can be obtained from the rendering primitives for geometric stage processing.

[0078] The geometry stage is responsible for processing geometric information, determining the type and position of the primitives to be drawn, performing vertex-by-vertex and polygon-by-polygon operations on the primitives, transforming the vertex coordinates into the screen space and handing them over to the rasterizer for processing, and outputting the two-dimensional vertex coordinates of the screen space, the depth value corresponding to the vertex, shading and other information. The geometry stage sub-pipeline includes vertex shading, tessellation shading, geometry shading, clipping, screen mapping, etc. The tessellation shader and geometry shader are optional shaders.

[0079] When cropping, there are three kinds of relationships between a primitive and the camera: completely in the field of view, partially in the field of view, and completely out of the field of view. Primitives that are completely in the field of view will continue to be passed to the next pipeline stage, and those that are completely out of the field of view will not be passed down, and those that are partially in the field of view need to be cropped.

[0080] The rasterization stage includes triangle setup, triangle traversal, fragment shader, and fragment-by-fragment operation. In addition, it may also include template testing, depth testing, and merging and blending processes, and finally obtains a driving simulation training fused image, that is, the fused scene image in the aforementioned embodiment.

[0081] The following describes the implementation method of calculating the conversion relationship between the image coordinate systems in the aforementioned embodiment.

[0082] The conversion relationship is represented by the transformation matrix P obtained by the following composite transformation.

[0083] The transformation matrix P includes: (1) Translation matrix. Matrix multiplication is used to represent the translation of a point. The multiplication result increases the position offset of the point x, y, and z components respectively. The translation matrix is ​​as follows: =

[0084] Among them, x, y, z are coordinate components, , , is the translation distance on the three axes.

[0085] (2) Scaling matrix. Matrix multiplication can be used to scale a model along the x-axis, y-axis, and z-axis of space. The multiplication result multiplies the x, y, and z components by the scaling factor, respectively. k is the scaling multiple. The scaling matrix is: =

[0086] (3) Rotation matrix.

[0087] Rotation matrix around the x-axis. If the point is rotated θ degrees around the x-axis, the rotation matrix Rx(θ) is: Rx(θ)=

[0088] Rotation matrix around the y-axis. If the point is rotated θ degrees around the y-axis, the rotation matrix Ry(θ) is: Ry(θ)=

[0089] Rotation matrix around the z-axis. If the point is rotated θ degrees around the z-axis, the rotation matrix Rz(θ) is: Rz(θ)=

[0090] Composite transformation can combine translation matrix, rotation matrix and scaling matrix into a complex transformation process. The final transformation matrix P is: P=

[0091]

[0092] The following Figure 8 The illustrated embodiment illustrates how the viewing frustum between the near clipping plane and the far clipping plane represents the calculation of the field of view of the first viewing position.

[0093] Among them, y and z are the coordinate axes of the image coordinate system, the origin is the virtual camera position, Far is the distance from the origin to the far clipping plane, Near is the distance from the origin to the near clipping plane, and FOV is the angle of the field of view. NearClipPlaneHeight = 2 Near tan(FOV / 2) FarClipPlaneHeight = 2 Far tan(FOV / 2) NearClipPlaneHeight is the height of the near clipping plane, and FarClipPlaneHeight is the height of the far clipping plane.

[0094] The aspect ratio Aspect of the picture that the virtual camera can see can be calculated by the near clipping plane: Aspect = NearClipPlaneWidth / NearClipPlaneHeight Alternatively, it can be calculated using the far clipping plane: Aspect = FarClipPlaneWidth / FarClipPlaneHeight NearClipPlaneWidth is the width of the near clipping plane, and FarClipPlaneWidth is the width of the far clipping plane.

[0095] The projection matrix M (frustum) of perspective projection is as follows: M(frustum) =

[0096] The vertices contained in the vertex data can be transformed from the viewing space to the clipping space by multiplying the above transformation matrix: P(clip) = M(frustum)P(visw) =

[0097] The goal of the clipping space is to facilitate clipping of rendered primitives.

[0098] If a vertex is inside the viewing frustum, the coordinates of the vertex after transformation to the clipping space must satisfy: -w<=x<=w -w<=y<=w -w<=z<=w w is the width of the virtual camera's field of view.

[0099] After clipping is completed, the vertices are mapped to the screen and rasterized at the mapped position to display the fused scene image to the driver.

[0100] The process formula for screen mapping is: ScreenX =

[0101] ScreenY =

[0102] Among them, (ScreenX, ScreenY) is the coordinate mapped to the VR glasses. , , It is the x, y, and w components of the vertex in the four dimensions x, y, z, and w after homogeneous division. is the pixel width, is the pixel height.

[0103] In one embodiment of the present invention, see Fig. 9 , provides a vehicle driving training method, the method comprising: S901: Acquire a real-scene image of the cockpit and the position and posture data of the driver; S902: Acquire a vehicle control signal input by the driver; S903: determining a virtual scene image in a preset virtual training scene according to the vehicle control signal and the position and posture data; fusing the virtual scene image with a preset virtual driving environment image and the real scene image to obtain a fused scene image; S904: Display the fused scene image to the driver.

[0104] In one embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned vehicle driving training method is implemented.

[0105] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0106] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0107] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0108] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0109] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0110] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0111] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0112] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A vehicle driving training system, characterized in that: The system includes: a cockpit, an AR interaction device, a training device, and a signal acquisition module, wherein: The AR interaction device is used to obtain a real-scene image of the cockpit and the driver's posture data; and send the real-scene image and the posture data to the training device; The signal acquisition module is used to obtain the vehicle control signal input by the driver; and send the vehicle control signal to the training device; The training device is used to determine a virtual scene image in a preset virtual training scene according to the vehicle control signal and the posture data; fuse the virtual scene image with a preset virtual driving environment image and the real scene image to obtain a fused scene image; and send the fused scene image to the AR interaction device; The AR interaction device is also used to display the fused scene image to the driver.

2. The system according to claim 1, characterized in that The training device determines a virtual scene image in a preset virtual training scene according to the vehicle control signal and the posture data, including: Determining a first viewing angle position of the driver in the virtual training scene according to the posture data; Displaying a first virtual scene image in the virtual training scene within a visible range of the first viewing angle position; updating the first viewing angle position according to the vehicle control signal to obtain a second viewing angle position; A second virtual scene image located within the visible range of the second viewing angle position in the virtual training scene is displayed.

3. The system according to claim 2, characterized in that The training device fuses the virtual scene image with a preset virtual driving environment image and the real scene image to obtain a fused scene image, including: Determining a conversion relationship between the image coordinate systems of the virtual scene image, the virtual driving environment image, and the real scene image; According to the determined conversion relationship, a first conversion image of the virtual scene image in the coordinate system where the real scene image is located is obtained, and a second conversion image of the virtual driving environment image in the coordinate system where the real scene image is located is obtained; A fused scene image is obtained based on the real scene image, the first converted image and the second converted image.

4. The system according to claim 3, characterized in that The training device obtains a fused scene image based on the real scene image, the first converted image, and the second converted image, including: Determine depth information of the real scene image, the first conversion image, and the second conversion image to determine virtual objects that are not blocked in the first conversion image and the second conversion image; A fused scene image is obtained based on the pixel data of the real scene object and the pixel data of the virtual object in the real scene image.

5. The system according to claim 1, characterized in that The vehicle control signal is input via at least one of the following devices: Driving controls, speed controls, instrumentation.

6. The system according to claim 1, characterized in that The AR interactive device includes: an image acquisition device, a head-mounted VR display, and a positioning tracker.

7. The system according to claim 1, characterized in that The system also includes an auditory feedback device and a tactile feedback device.

8. The system according to claim 1, characterized in that The system also includes a control computer; the control computer can receive the fused scene image and synchronize it to a preset monitoring display.

9. A vehicle driving training method, characterized in that: The method comprises: Obtain the real-life image of the cockpit and the driver’s posture data; Acquiring a vehicle control signal input by the driver; Determine a virtual scene image in a preset virtual training scene according to the vehicle control signal and the position and posture data; fuse the virtual scene image with a preset virtual driving environment image and the real scene image to obtain a fused scene image; The fused scene image is displayed to the driver.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vehicle driving training method as claimed in claim 9 is implemented.

Images