A vehicle-mounted VR rear-seat entertainment method

Abstract

The present application discloses a vehicle-mounted VR rear-seat entertainment method, which relates to the field of VR, including: processing image data transmitted by a controller through a DCT plate negative reflection flat lens to generate aerial imaging data; using a plane reflector to reflect the virtual image corresponding to the aerial imaging data to generate real image data; performing voice recognition processing on the user's input voice to obtain voice control instruction data; collecting image data of the user's gesture through an infrared sensor, and performing image recognition to obtain gesture control instruction data; generating interactive control data according to the three-dimensional coordinates in the real image data, as well as voice control instructions and gesture control instructions; in view of the low imaging accuracy of the image imaging in the vehicle-mounted VR entertainment system in the prior art, the present application uses a DCT plate negative reflection flat lens and a plane reflector to generate high-quality aerial imaging, and adopts voice recognition and gesture recognition to realize multimodal interaction, etc., to provide realistic imaging effects.

Description

Technical Field

[0001] The present application relates to the field of VR, and in particular to a vehicle-mounted VR rear-seat entertainment method. Background Art

[0002] With the continuous development of social economy and the continuous progress of science and technology, cars have become an indispensable means of transportation in modern life. With the rapid development of automobile technology, people's requirements for in-vehicle entertainment systems are getting higher and higher. Among them, virtual reality (VR) technology, with its immersive visual experience and immersive interactive mode, is becoming a new trend and new direction in the development of in-vehicle entertainment systems.

[0003] At present, VR technology has been initially applied in in-vehicle entertainment systems, especially in rear-seat entertainment systems, where VR devices can provide passengers with independent private space and rich entertainment content. However, the existing in-vehicle VR entertainment systems still have some problems that need to be solved, which restricts their further promotion and application.

[0004] Among them, the poor image quality is one of the most prominent problems. Due to the small space inside the car, traditional VR display devices such as head-mounted displays have limited screen size and cannot provide a wide field of view and detailed images. Projection-based VR display systems, such as holographic projection screens, are limited by changes in lighting conditions and vibrations inside the car, resulting in unstable image quality, and clarity and stereoscopic perception cannot be guaranteed. Summary of the invention

[0005] In response to the problem of low image imaging accuracy in the in-vehicle VR entertainment system in the prior art, the present application provides an in-vehicle VR rear-seat entertainment method, which generates high-quality aerial imaging by using a DCT plate negative reflection flat lens and a plane reflector in a holographic projection screen system, and uses voice recognition and gesture recognition to achieve multimodal interaction. The three-dimensional space interaction model is updated according to voice control commands and gesture control commands, the interaction behavior between the user and the virtual object is judged, and interaction control data is generated, thereby controlling the in-vehicle entertainment system and providing a realistic aerial imaging effect.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present application provides a vehicle-mounted VR rear-seat entertainment method for controlling the vehicle-mounted entertainment system, including: through the controller of the vehicle-mounted entertainment system, respectively connecting the LCD display system, the holographic projection screen system, and the AR glasses system, and transmitting image data to each system; in the holographic projection screen system: processing the image data transmitted by the controller through the DCTplate negative reflection flat lens to generate aerial imaging data; using a plane reflector to reflect the virtual image corresponding to the aerial imaging data to generate real image data; performing voice recognition processing on the user's input voice to obtain voice control instruction data as the first control instruction; collecting image data of the user's gesture through an infrared sensor, and performing image recognition to obtain gesture control instruction data as the second control instruction; generating interactive control data according to the three-dimensional coordinates in the real image data, as well as the first control instruction and the second control instruction; transmitting the interactive control data to the controller to control the vehicle-mounted entertainment system.

[0008] Among them, the DCT (Directional Cylindrical Transmissive) plate negative reflection flat lens is a specially designed optical element used to generate aerial imaging. It consists of multiple cylindrical lenses, each with a different focal length and direction. When the image data passes through the DCT plate negative reflection flat lens, the light is split and redirected to form a virtual image in the air. The unique design of this lens allows light to be transmitted in a specific direction and reflected in other directions, thereby achieving an aerial imaging effect. A virtual image refers to an image formed on the extension line of the light after the light is reflected or refracted. A virtual image cannot be directly imaged on a screen or other physical medium, but is formed in the air by the convergence or divergence of light. In the holographic projection screen system, after the DCT plate negative reflection flat lens processes the image data, the aerial imaging data generated corresponds to the virtual image. The virtual image gives people a visual effect that the image is suspended in the air, but in fact there is no physical object at that location. Real image data refers to the real image data formed on the screen, projector or other physical medium after imaging through the optical system. Unlike a virtual image, a real image is formed by the direct convergence of light on a physical medium and can be directly observed on the medium. In the holographic projection screen system, after the plane mirror reflects the virtual image, real image data is generated. These real image data can be directly observed and interacted with by users, such as operating virtual objects in the real image through gestures or voice control. The real image data contains the three-dimensional coordinate information of the virtual object, which is used to generate interactive control data.

[0009] Furthermore, the image data transmitted by the controller is processed by the DCTplate negative reflection flat lens to generate aerial imaging data, including: preprocessing the image data, the preprocessing includes image denoising, image enhancement and image correction; diffracting and phase modulating the incident image data through the optical diffraction and phase modulation effect of the DCTplate negative reflection flat lens to generate three-dimensional aerial imaging light field data; based on the three-dimensional aerial imaging light field data, multi-viewing angle aerial imaging data is generated through multi-viewing angle acquisition and reconstruction algorithm.

[0010] Among them, the multi-view acquisition and reconstruction algorithm is a computer vision algorithm for generating multi-view aerial imaging data. Through multiple light field cameras or lens arrays distributed at different angles, the aerial imaging light field is collected to obtain two-dimensional light field image data from different perspectives. By comparing and analyzing the correspondence between light field images of different perspectives, the disparity between them, that is, the displacement of spatial points in images of different perspectives, is estimated. Common disparity estimation methods include local algorithms based on block matching and global algorithms based on global optimization. Due to factors such as occlusion and noise during the acquisition process, the collected light field data may be incomplete or have errors. Light field repair is to use the spatiotemporal consistency of light field data to repair missing or erroneous light field data to obtain a complete and accurate light field representation. Common repair methods include sparse representation repair based on dictionary learning and end-to-end repair based on deep learning. Based on the existing multi-view light field data, the light field data of other arbitrary perspectives are interpolated through the mapping relationship between perspectives, thereby obtaining a continuous multi-view light field representation. Common viewpoint interpolation methods include ray interpolation based on ray transport theory, viewpoint synthesis based on depth map estimation, etc. Using the restored and interpolated complete multi-view light field data, through ray tracing, volume rendering and other algorithms, the light field distribution in three-dimensional space is reconstructed, and the final multi-view aerial imaging image sequence is drawn and generated.

[0011] Furthermore, generating three-dimensional aerial imaging light field data includes: modulating the spatial frequency of the image data according to the parameters of the diffraction grating of the DCTplate negative reflection flat lens to generate frequency-domain modulated image data; wherein the parameters of the diffraction grating include a grating constant; performing phase modulation on the frequency-domain modulated image data according to the orientation state of liquid crystal molecules in the phase modulation liquid crystal layer of the DCTplate negative reflection flat lens to generate phase-modulated image data; and reconstructing the light field wavefront of the phase-modulated image data through the output end of the DCTplate negative reflection flat lens to generate three-dimensional aerial imaging light field data.

[0012] Among them, the orientation state (Orientation State), in the liquid crystal phase modulator, the arrangement mode and arrangement direction of the liquid crystal molecules will affect the phase of the incident light. The orientation state refers to the spatial arrangement state of the liquid crystal molecules, mainly including: Twisted Nematic (TN): The liquid crystal molecules are arranged in a spiral shape along the direction of the electric field, and the polarization direction of the incident light changes with the rotation of the liquid crystal molecules, thereby realizing phase modulation. Vertical Alignment (VA): The liquid crystal molecules change from vertical arrangement to tilted arrangement under the action of the electric field, and the incident light senses the variable birefringence effect, causing phase change. Parallel Alignment (PA): The liquid crystal molecules always maintain a horizontal arrangement, and the refractive index is adjusted by changing the molecular orientation angle, thereby continuously regulating the phase of the incident light. By controlling the size and distribution of the applied voltage, the orientation state of the liquid crystal molecules can be accurately adjusted, thereby realizing fine phase modulation of the image light field. Different orientation states have different modulation methods, and have different characteristics in terms of response speed, modulation accuracy, etc.

[0013] Light field wavefront reconstruction refers to the process of reconstructing the complete wavefront shape of light wave propagation in three-dimensional space using recorded light field information. Light field data records the intensity and directional distribution of light in space, but due to the aperture and resolution limitations of the imaging system, the recorded light field is discrete and limited in sampling. In order to reconstruct a continuous and complete three-dimensional light wave, the sampled light field needs to be interpolated, propagated, and processed. According to the angle-position representation of the light field (such as plane light field, spherical light field, etc.), a physical model of light propagation is established. Commonly used reconstruction algorithms include the angular spectrum method based on Fourier transform, the ASM method based on Fresnel diffraction, and the ray propagation method based on the ABCD matrix. These methods use the theory of optical propagation to coherently superimpose light field data and simulate the propagation behavior of light in three-dimensional space. Through light field reconstruction, the wavefront of light waves at any position and direction can be obtained. By matching the reconstructed three-dimensional light field with the spatial position, realistic aerial imaging is obtained.

[0014] Furthermore, multi-perspective aerial imaging data is generated, including: using multiple light field lenses to collect aerial imaging light field data at different perspectives to obtain multi-perspective light field data; constructing objective functions of disparity estimation and light field restoration; wherein the disparity estimation objective function is used to minimize the disparity error between light field data of different perspectives, and the light field restoration objective function is used to minimize the reconstruction error of the light field data; weightedly combining the disparity estimation objective function and the light field restoration objective function to obtain a reconstruction objective function; using an alternating direction multiplier method to solve the reconstruction objective function, by converting the reconstruction objective function into an augmented Lagrangian function and introducing a scale variable, the problem to be solved is decomposed into multiple sub-problems, and each sub-problem is solved alternately to obtain a disparity map and reconstructed light field data; according to the disparity map, by calculating the corresponding relationship between light field data of different perspectives, a mapping relationship between perspectives is established, and the reconstructed light field data is interpolated from a perspective using the mapping relationship to obtain multi-perspective aerial imaging data.

[0015] Among them, the light field lens (Light Field Lens) is a special lens that can simultaneously record the intensity and direction information of light. Unlike traditional 2D imaging lenses, light field lenses collect 4D light field data and additionally encode the spatial angle information of light. Common light field lens structures include: Microlens Array: A high-density microlens array is placed in front of the sensor, and each microlens corresponds to a macro pixel on the sensor. After passing through the microlens, the light will be imaged in the macro pixel, recording the angular distribution of the light. Coded Aperture: A mask with a special pattern is placed at the aperture. Through the diffraction and scattering of light on the pattern, a unique coding pattern is formed on the imaging plane, which contains the angle information of light. Multi-view Lens Array: An array composed of multiple imaging lenses is used, each lens images the scene from a different perspective, and light field data is obtained through parallax and correspondence. Light field cameras use light field lenses to collect light field data and record the distribution of light in three-dimensional space. The obtained 4D light field can be post-processed through digital refocusing, perspective synthesis and other methods to achieve effects such as adjustable focus and variable perspective, and has important applications in computational photography, three-dimensional reconstruction and other fields.

[0016] Alternating Direction Method of Multipliers (ADMM), ADMM is a numerical algorithm for solving large-scale optimization problems, especially for convex optimization problems with multiple variables and complex constraints. Its basic idea is to decompose the original problem into several simple sub-problems, and then solve each sub-problem alternately, continuously update the variables, and finally get the solution to the original problem. The original optimization problem is transformed into an equivalent constrained optimization problem by introducing auxiliary variables. The augmented Lagrangian function is introduced to move the constraint terms into the objective function. The augmented Lagrangian function is applied with alternating minimization to decompose the optimization problem into multiple sub-problems about each variable. Sub-problems generally have closed solutions or are easy to solve. Solve the sub-problems in turn, and update the original variable and dual variable according to the solution results. The sub-problem solution can be carried out in parallel to improve the computational efficiency. Repeat until the difference between the original variable and the auxiliary variable is less than the set threshold, or the maximum number of iterations is reached, and the algorithm converges.

[0017] Furthermore, the disparity estimation objective function expression is as follows:

[0018] Among them, P i,j P represents the reference view pixel block at position (i, j); m,n Represents the reference view pixel block P i,j The corresponding pixel blocks of other viewing angles are located at position (m, n); w i,j,m,n Represents the reference view pixel block P i,j With other perspective pixel blocks P m,n The weight coefficient between is used to balance the contribution between pixel blocks of different perspectives; V i,j,m,n represents the perspective prior offset between pixel blocks of different perspectives in a vehicle environment, which is estimated based on the prior knowledge of vehicle posture and road scene; ▽d i,j Represents the gradient of the disparity map, including the gradients in the horizontal and vertical directions; M i,j represents the parallax change caused by vehicle motion, which is estimated by the vehicle motion parameters; γ represents the balance factor, which is used to control the influence weight of the perspective prior term and the motion constraint term.

[0019] Furthermore, the objective function expression of light field restoration is as follows:

[0020] Where D represents the sparse dictionary matrix, which is a predefined matrix used to map light field data to the sparse representation space. Each column of the dictionary matrix represents a basic element, and the light field data can be represented by the linear combination of these basic elements; α represents the sparse representation sparse vector, which represents the projection coefficient of the light field data on the sparse dictionary D, that is, the light field data can be approximately represented by the linear combination of the basic elements in the dictionary matrix, and α is the coefficient of these linear combinations. L r represents the reconstructed light field data, which represents the light field data obtained after light field restoration; I(i,j) represents the illumination prior information in the vehicle environment, which represents the illumination information estimated according to the vehicle position and ambient lighting conditions in the vehicle environment, and I(i,j) represents the illumination intensity and color information at the (i,j)th position of the light field data; S(i,j) represents the scale factor of the illumination prior information, which is used to adjust the influence weight of the illumination prior information I(i,j) in the light field restoration objective function, and the value of S(i,j) can be set according to the lighting conditions and restoration requirements; V i,j,m,n V represents the parallax consistency constraint, which means that the light field restoration is constrained by the parallax estimation result to ensure the consistency between the light field restoration result and the parallax estimation result. i,j,m,n represents the parallax difference between the (i, j)th position and the (m, n)th position in the light field data; λ1 represents the balance factor of the total variation regularization term, which is used to control the total variation regularization term TV(L r ) is the weight in the light field restoration objective function to balance the spatial continuity and local smoothness of the light field data; λ2 represents the balance factor of the illumination prior term, which is used to control the illumination prior term The weight in the light field restoration objective function is used to balance the impact of the illumination prior information on the restoration result; λ3 represents the balance factor of the parallax consistency constraint term, which is used to control the parallax consistency constraint term The weight in the light field restoration objective function is used to balance the degree of constraint of the parallax estimation result on the restoration result;

[0021] Furthermore, real image data is generated, including: obtaining the emission angle data of the DCTplate negative reflection flat lens and the position parameters of the plane reflector, and determining the reflection angle parameters of the plane reflector through geometric optical calculation according to the emission angle data and the position parameters; inputting the aerial imaging light field data into the plane reflector, and using the reflection characteristics of the plane reflector to reflect the aerial imaging light field data irradiated on the reflection surface to obtain the aerial imaging light field data after reflection; wherein the aerial imaging light field data contains virtual image information corresponding to the aerial imaging data; according to the position parameters of the preset imaging plane, projecting the reflected aerial imaging light field data onto the preset imaging plane, and reflecting the virtual image corresponding to the aerial imaging data onto the preset imaging plane through the reflection effect of the plane reflector to form real image data.

[0022] Among them, position parameters refer to a set of parameters that describe the spatial position of an object. In a plane reflector imaging system, the following position parameters are mainly involved: Mirror position: used to determine the position of the plane reflector in three-dimensional space. Usually expressed by the coordinates (x, y, z) of the center point of the reflector. Imaging plane position: used to determine the position of the imaging screen that receives the real image in three-dimensional space. Also expressed by the coordinates (x, y, z) of the center point of the imaging plane. The position parameters determine the geometric relationship between the reflector and the imaging plane in space, and are the basis for ray tracing and image formation calculations. By adjusting the position parameters of the reflector and the imaging plane, the imaging position and size of the real image can be controlled.

[0023] The reflection angle parameter describes the angular relationship between the normal direction of the plane reflector surface and the direction of the incident light. According to the law of reflection, the incident angle is equal to the reflection angle, and the reflection angle parameter determines the spatial distribution of the reflected light. The main reflection angle parameters include: Reflector tilt angle: refers to the angle between the normal of the reflector surface and a reference direction (such as the horizontal direction). The tilt angle determines the overall deflection direction of the reflected light beam. Reflector rotation angle: refers to the rotation angle of the reflector surface around a certain axis (such as an axis perpendicular to the reflector surface). The rotation angle affects the deflection of the reflected light beam in the horizontal direction. By setting the tilt angle and rotation angle of the reflector, the spatial direction of the reflected light beam can be adjusted, thereby controlling the imaging position and direction of the real image.

[0024] Furthermore, voice control instruction data is obtained as a first control instruction, including: collecting the user's input voice through a microphone array in the vehicle entertainment system to obtain original voice data; preprocessing the original voice data, the preprocessing includes voice noise reduction, voice endpoint detection and voice feature extraction, to obtain preprocessed voice feature data; using a pre-trained convolutional neural network model, the preprocessed voice feature data is recognized to obtain voice recognition result data; according to a preset voice control instruction set, the voice recognition result data is converted into voice control instruction data that matches the voice control instruction set as the first control instruction.

[0025] Furthermore, gesture control instruction data is obtained as a second control instruction, including: collecting original image data of the user's gesture through an infrared sensor array in the vehicle entertainment system, and extracting infrared image data of the gesture area according to the position and depth information of the gesture area; using a pre-trained convolutional neural network model, performing feature extraction and classification on the infrared image data of the gesture area to obtain gesture recognition result data; according to a preset gesture control instruction set, converting the gesture recognition result data into gesture control instruction data matching the control instruction set as the second control instruction.

[0026] Among them, depth information refers to data describing the distance of an object from an observation point or reference plane in three-dimensional space. In gesture recognition systems, depth information is used to determine the spatial position and distance of the gesture area relative to the sensor. Common methods for obtaining depth information include: binocular vision: two infrared cameras are used to shoot the gesture area from different perspectives, and the depth information of the gesture is calculated by matching the left and right images based on the binocular parallax principle. Structured light: an infrared laser transmitter is used to project a specific pattern of structured light to the gesture area, and then an infrared camera is used to shoot the deformation of the pattern on the gesture surface, and the depth information of the gesture is calculated based on the degree of deformation. Time of flight (ToF): by emitting high-frequency modulated infrared light to the gesture area, and then measuring the time it takes for the infrared light to be reflected back to the sensor, the depth information of the gesture is calculated based on the speed of light and the time difference. Depth information can help the gesture recognition system determine the three-dimensional spatial position of the gesture, judge the distance between the gesture and the sensor, and improve the accuracy and robustness of gesture recognition. Through depth information, overlapping gestures can be distinguished, background interference can be eliminated, and more reliable gesture interaction control can be achieved.

[0027] The gesture control instruction set is a mapping relationship between a set of predefined gesture patterns and corresponding control commands. It associates the results of gesture recognition with specific control operations to achieve gesture-based human-computer interaction. The design of the gesture control instruction set needs to consider the following factors: Gesture types: Determine the supported gesture types based on the application scenario and interaction requirements, such as one-hand gestures, two-hand gestures, static gestures, dynamic gestures, etc. Each gesture corresponds to a unique identifier. Control commands: Define a set of control commands based on the system's functions and operating modes, such as move, rotate, zoom, select, confirm, cancel, etc. Each control command corresponds to a unique identifier. Mapping relationship: Establish a one-to-one or many-to-one mapping relationship between gesture types and control commands. Convert the recognized gesture identifiers into corresponding control command identifiers through table lookup or rule matching.

[0028] Furthermore, generating interactive control data includes: acquiring three-dimensional coordinate data in real image data, and constructing a three-dimensional space interaction model through a three-dimensional reconstruction algorithm according to the three-dimensional coordinate data, so as to represent the interactive relationship between the user and the virtual object; converting the voice control instruction data in the first control instruction into the motion parameters of the virtual object, and converting the gesture control instruction data in the second control instruction into the posture parameters of the virtual object; updating the spatial position and interactive state of the virtual object in the three-dimensional space interaction model according to the motion parameters and the posture parameters; calculating the spatial distance and relative posture between the user and the virtual object according to the spatial position and interactive state of the user and the virtual object in the updated three-dimensional space interaction model, and judging whether the preset interactive conditions are met, and if so, judging that there is an interactive behavior between the user and the virtual object, and triggering a corresponding interactive event; wherein the preset interactive conditions include a spatial distance threshold and a relative posture threshold, and the interactive behavior includes collision, selection and grabbing with the virtual object; selecting a corresponding interactive control instruction from a preset interactive control instruction set according to the type of the triggered interactive event, and generating interactive control instruction data; fusing the generated interactive control instruction data with the spatial position, posture and attribute information of the virtual object in the three-dimensional space interaction model, and generating interactive control data for controlling the in-vehicle entertainment system.

[0029] Among them, the 3D reconstruction algorithm refers to the computational method of recovering the 3D scene structure and object shape from 2D image or video data. Its purpose is to infer the 3D spatial coordinates and surface shape of objects in the scene based on 2D observation information from multiple perspectives. Common 3D reconstruction algorithms include: Binocular stereo vision: By matching the image features of the left and right perspectives, the parallax of the feature points is calculated to obtain the depth information of the feature points and reconstruct the 3D point cloud or surface. Structured light 3D reconstruction: By projecting a structured light pattern onto the surface of the object, analyzing the deformation and encoding of the pattern, and calculating the 3D coordinates of each pattern point, the 3D model of the object is reconstructed. Multi-view geometric reconstruction: By analyzing the correspondence and motion trajectory of the object feature points under multiple perspectives, the camera motion parameters and the 3D structure of the object are estimated to achieve dense 3D reconstruction. Deep learning 3D reconstruction: Using deep neural networks to directly predict 3D representations such as depth maps, normal vector maps, occupancy maps, etc. from monocular images or videos, and combining geometric optimization and mesh extraction algorithms to obtain 3D models. 3D reconstruction algorithms are key technologies for realizing the construction of 3D spatial interactive models. By reconstructing the features in the real image data in three dimensions, the three-dimensional spatial coordinates of the virtual objects and users can be obtained, providing a geometric basis for the analysis and control of interactive behaviors.

[0030] Motion parameters are a set of quantities that describe the motion state of a virtual object in three-dimensional space. Common motion parameters include: Translation: describes the position change of a virtual object in three-dimensional space, represented by a three-dimensional vector (x, y, z). Velocity: describes the rate of change of the position of a virtual object over time, represented by a three-dimensional vector (vx, vy, vz), and characterizes the speed and direction of the motion. Acceleration: describes the rate of change of the speed of a virtual object over time, represented by a three-dimensional vector (ax, ay, az), and reflects the changing trend of the motion state. Motion parameters can be converted through voice control commands. For example, the voice command "move forward" can be converted into a speed change in the positive direction of the z-axis of the virtual object. By setting appropriate motion parameters, the motion trajectory and behavior of virtual objects in three-dimensional space can be controlled.

[0031] The posture parameters describe the direction and rotation state of the virtual object in three-dimensional space. Common posture parameter representation methods include: Euler Angles: Use three angles (roll, pitch, yaw) to describe the rotation state of the object around three coordinate axes. Quaternion: Use a four-dimensional vector (x, y, z, w) to describe the rotation state of the object, avoiding the universal lock problem of Euler angles. Rotation Matrix: Use a 3x3 orthogonal matrix to describe the rotation transformation of the object, and coordinate transformation calculations can be performed directly. The posture parameters can be converted through gesture control instructions. For example, the gesture instruction "rotate left" can be converted into the rotation angle change of the virtual object around the vertical axis. By setting appropriate posture parameters, the direction and viewing angle of the virtual object in three-dimensional space can be controlled.

[0032] Interaction conditions are a set of criteria for triggering interaction events between virtual objects and users. Common interaction conditions include: Distance conditions: Determine whether the Euclidean distance between the virtual object and the user in three-dimensional space is less than the set threshold, which is used to trigger close-range interactions such as collision and selection. Line of sight conditions: Determine whether the angle between the user's line of sight and the direction of the virtual object is less than the set threshold, which is used to trigger visual interactions such as gaze and stare. Gesture conditions: Determine whether the user's gesture posture matches the preset gesture template, which is used to trigger gesture interactions such as grabbing and throwing. Voice conditions: Determine whether the user's voice command matches the preset command set, which is used to trigger voice-controlled interaction. The setting of interaction conditions needs to be determined according to specific interaction tasks and user habits. Through reasonable threshold selection and condition combination, natural and smooth three-dimensional interactive control can be achieved. At the same time, the judgment of interaction conditions also needs to consider real-time and robustness to avoid false triggering and missed detection.

[0033] Compared with the prior art, the advantages of this application are:

[0034] The spatial frequency of the image data is modulated by the diffraction grating of the DCT plate negative reflection flat lens, and then phase modulated by the phase modulation liquid crystal layer. Finally, the light field wavefront is reconstructed at the output end. The spatial frequency modulation enhances the spatial resolution of the image, the phase modulation compensates for the aberration, and the light field wavefront reconstruction restores the complete three-dimensional light field information, ultimately obtaining high-fidelity aerial imaging.

[0035] The reconstruction objective function including parallax estimation and light field restoration is constructed, the prior knowledge of the vehicle environment is introduced, and the alternating direction multiplier method is used to solve the objective function. The prior knowledge of the vehicle environment, such as the perspective prior and the illumination prior, provides additional constraint information and reduces the solution space of the optimization problem. Perspective interpolation makes up for the coherence between perspectives, making the imaging smoother and more natural, and multi-perspective aerial imaging data is obtained through perspective interpolation.

[0036] According to the emission angle of the DCT plate and the position of the plane reflector, the reflection angle parameters of the reflector are determined through geometric optical calculations. The aerial imaging light field data is irradiated onto the reflective surface of the reflector, and the reflected light field data is projected onto the preset imaging plane. The reflection law of geometric optics ensures the lossless mapping of imaging information during the conversion from virtual image to real image. The reasonable position of the preset imaging plane makes the real image clear and stable. The combination of virtual and real greatly improves the realism of imaging.

[0037] Based on the real image data and control instructions, a three-dimensional spatial interaction model is constructed to represent the interaction relationship between the user and the virtual object. The voice instructions are converted into the motion parameters of the virtual object, and the gesture instructions are converted into the posture parameters of the virtual object, and the spatial position and interaction status of the virtual object in the interaction model are updated in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0039] Figure 1 is an exemplary flow chart of a method for in-vehicle VR rear-seat entertainment according to some embodiments of the present application;

[0040] Figure 2 is an application schematic diagram shown according to some embodiments of the present application;

[0041] Figure 3 It is a schematic diagram of generating three-dimensional aerial imaging according to some embodiments of the present application;

[0042] Figure 4It is a schematic diagram of imaging of a DCTplate negative reflection flat lens according to some embodiments of the present application. DETAILED DESCRIPTION

[0043] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 This is an exemplary flow chart of an in-vehicle VR rear-seat entertainment method shown in some embodiments of the present application, including: through the controller of the in-vehicle entertainment system, respectively connecting the LCD display system, the holographic projection screen system, and the AR glasses system, and transmitting image data to each system; in the holographic projection screen system: processing the image data transmitted by the controller through the DCTplate negative reflection flat lens to generate aerial imaging data; using a plane mirror to reflect the virtual image corresponding to the aerial imaging data to generate real image data; performing voice recognition processing on the user's input voice to obtain voice control instruction data as the first control instruction; collecting image data of the user's gesture through an infrared sensor, and performing image recognition to obtain gesture control instruction data as the second control instruction; generating interactive control data according to the three-dimensional coordinates in the real image data, as well as the first control instruction and the second control instruction; transmitting the interactive control data to the controller to control the in-vehicle entertainment system.

[0045] Figure 2 This is an application diagram according to some embodiments of the present application. In this embodiment, the display part of the in-vehicle entertainment system includes a co-pilot screen LCD display, a rear holographic projection screen display, an AR glasses system and a VR glasses system, covering the display requirements of different areas and usage scenarios in the car. Among them, the co-pilot screen adopts LCD liquid crystal display technology, and is connected to the main control chip RK3588 through the GMSL solution of the MIPI interface, realizing stable and high-speed video transmission and display.

[0046] As the core display device of the whole system, the rear holographic projection screen is connected to RK3588 through the FPD-LINK solution of the MIPI interface, and uses holographic projection technology to generate high-quality aerial imaging to provide an immersive visual experience for rear passengers. The AR glasses system uses a Sony OLED micro display with a resolution of 1920*1080P and a field of view (FOV) of 45°. It is connected to the DP output of RK3588 through the USB-TypeC interface, and has excellent display effects and wearing comfort. The matching USB remote control can easily control multimedia playback.

[0047] The VR glasses system uses PICO4 as the terminal device, which is connected to RK3588 via WIFI to achieve the transmission of vehicle data and the coordination of VR applications. PICO 4 uses advanced Pancake optical lens technology, which greatly reduces the weight of the equipment. It is also equipped with two 2.56-inch Fast-LCD screens with a resolution of up to 4320×2160, a refresh rate of 90Hz, a pixel density of 1200PPI, and a pixel per degree (PPD) of 20.6, providing an ultra-high-definition and wide-viewing VR experience.

[0048] As the main control chip of the in-vehicle entertainment system, RK3588 has powerful computing power and rich interface resources. Through flexible interface configuration and transmission solutions, it realizes efficient connection and control of multiple display devices. Among them, the DPHY0 channel of the MIPI interface adopts the FPD-LINK solution to connect to the holographic projection screen system, and the DPHY1 channel adopts the GMSL solution to connect to the co-pilot entertainment screen system, making full use of the high bandwidth and low power consumption advantages of the MIPI interface to ensure the quality and stability of video transmission. The DP interface is used to connect to the AR glasses system, providing high-resolution video output for AR glasses, and realizing power supply and control of AR glasses through the USB interface. WIFI wireless connection is applied to the VR glasses system, and the whole vehicle data and VR application data are sent to the VR glasses through high-speed WIFI transmission, realizing seamless coordination between VR glasses and the vehicle system, and providing an immersive virtual reality experience.

[0049] The holographic projection system projects the image of the display screen into the air through an optical module, which can be directly observed by the naked eye. The system supports touch interaction of aerial projection and can realize the functions of the entertainment system. The core of the holographic projection screen system is the use of DCT plate negative reflection flat lens technology, which realizes medium-free aerial imaging. DCT plate is a special optical element with a fine diffraction grating structure on its surface. When the image light is irradiated on the DCT plate, diffraction occurs, and light from different directions will be separated and focused to different positions in the air to form a real image in the air. Compared with traditional transmission or reflection imaging methods, DCT plate negative reflection technology does not require additional imaging media, such as atomized media or projection screens, and can directly form high-quality suspended images in the air with higher clarity, larger viewing angles and better observation comfort.

[0050] Figure 3 is a schematic diagram of generating three-dimensional aerial imaging according to some embodiments of the present application, Figure 4The DCTplate negative reflection flat lens imaging schematic diagram shown in some embodiments of the present application, in the holographic projection screen system: before the image is projected, the holographic projection screen system will perform a series of preprocessing on the image data, including image denoising, image enhancement and image correction. The input image data is decomposed by wavelet, and the image is decomposed into low-frequency sub-bands and high-frequency sub-bands. The decomposition level is N (such as N = 3), and 1 low-frequency sub-band LL and 3N high-frequency sub-bands HL are obtained. i , LH i and HH i (i=1, 2, ..., N). For each high frequency subband HL i , LH i and HH i , calculate its noise variance σ i , and adaptively determine the wavelet threshold T according to the noise variance i : Among them, M i and N i is the size of the high-frequency subband under the i-th level decomposition. Perform soft threshold processing on the wavelet coefficients of each high-frequency subband: Among them, W ij is the wavelet coefficient of the jth high-frequency subband under the i-th level decomposition, W ij ' is the wavelet coefficient after threshold processing, sgn(*) is the sign function. The wavelet coefficient after threshold processing is reconstructed by wavelet to obtain the denoised image.

[0051] Image enhancement uses a multi-scale fusion enhancement algorithm based on Retinex theory to perform multi-scale Retinex decomposition on the input image data to obtain Retinex images R at N different scales. i (i=1, 2, ..., N): R i =log(I)-log(F i ×I), where I is the original image, F i is the Gaussian filter at the i-th scale, and * is the convolution operation. For each Retinex image R i , calculate its brightness mean m i and standard deviation s i , and adaptively enhance the Retinex image according to the mean and standard deviation: Among them, k is the enhancement coefficient, which can be adaptively adjusted according to the contrast of the image. i 'Perform multi-scale fusion to obtain the final enhanced image E: Among them, w i is the weight coefficient of the i-th scale, which can be adaptively adjusted according to the importance of the scale.

[0052] Image correction mainly aims at the geometric distortion and color distortion problems that may occur during the projection process. The optical parameters of the projection system are calibrated, including the intrinsic parameters (such as focal length, principal point coordinates, etc.) and extrinsic parameters (such as rotation matrix, translation vector, etc.) of the projector, as well as the geometric parameters of the projection screen (such as screen size, orientation, etc.). According to the calibrated parameters, a geometric distortion model of the projection system is established, such as a radial distortion model, a tangential distortion model, etc., to obtain the distortion function D(x, y). The input image data is subjected to distortion correction, and the distorted coordinates (x, y) are mapped to the corrected coordinates (u, v): (u, v) = D -1 (x,y), where D -1 is the inverse function of the distortion function, which can be obtained by table lookup or iterative solution. The corrected image data is color corrected, and the color space of the projector is mapped to the target color space (such as sRGB space) through the color mapping function to ensure the accuracy of color restoration.

[0053] Frequency domain modulation, according to the diffraction grating parameters of the DCT plate negative reflection plate lens, the input image data I(x,y) is modulated in spatial frequency. The image data I(x,y) is subjected to a two-dimensional discrete cosine transform (2D-DCT) to obtain a frequency domain representation I(u,v)=DCT[I(x,y)]. According to the grating constant d of the diffraction grating, the diffraction angle θ of the grating is calculated: Where λ is the wavelength of the incident light. According to the diffraction angle θ, the frequency domain data I(u,v) is modulated in the frequency domain to obtain the modulated frequency domain data I'(u,v):

[0054] Wherein, j is an imaginary unit. Perform two-dimensional inverse discrete cosine transform (2D-IDCT) on the modulated frequency domain data I'(u,v) to obtain the frequency domain modulated image data I'(x,y): I'(x,y)=IDCT[I'(u,v)].

[0055] Phase modulation, according to the orientation state of the liquid crystal molecules in the phase modulation liquid crystal layer of the DCT plate negative reflection flat lens, phase modulates the image data I'(x,y) after frequency domain modulation. According to the voltage distribution V(x,y) of the phase modulation liquid crystal layer, the orientation angle distribution φ(x,y) of the liquid crystal molecules is calculated: φ(x,y) = f[V(x,y)], where f[·] is the response function of the liquid crystal molecule orientation angle to voltage, which can be obtained through experimental measurement or theoretical model calculation. According to the orientation angle distribution φ(x,y) of the liquid crystal molecules, the phase delay distribution Φ(x,y) introduced by the phase modulation liquid crystal layer is calculated:

[0056] Wherein, Δn[φ(x,y)] is the birefringence response function of the liquid crystal material, and d is the thickness of the phase modulation liquid crystal layer. Phase modulation is performed on the frequency-domain modulated image data I'(x,y) to obtain phase-modulated image data I"(x,y): I"(x,y)=I'(x,y)×exp[jΦ(x,y)].

[0057] Light field wavefront reconstruction, through the output end of the DCT plate negative reflection flat lens, the phase modulated image data I" (x, y) is reconstructed to generate three-dimensional aerial imaging light field data. According to the output end parameters of the DCT plate negative reflection flat lens, such as the exit pupil diameter D and the focal length f, the transfer function H (u, v) of the light field propagation is calculated: Wherein, z is the distance from the output end to the imaging plane. Perform two-dimensional Fourier transform (2D-FFT) on the phase modulated image data I”(x,y) to obtain the frequency domain representation I”(u,v): I”(u,v)=FFT[I”(x,y)]; multiply the frequency domain data I”(u,v) with the transfer function H(u,v) to obtain the frequency domain data E(u,v) after light field propagation: E(u,v)=I”(u,v)×H(u,v); perform two-dimensional inverse Fourier transform (2D-IFFT) on the frequency domain data E(u,v) after light field propagation to obtain the three-dimensional aerial imaging light field data E(x,y,z): E(x,y,z)=IIFFT[E(u,v)].

[0058] Through N light field lenses, aerial imaging light field data at different viewing angles are collected to obtain multi-view light field data L n (i, j, u, v), where n represents the view number, (i, j) represents the spatial coordinates, and (u, v) represents the angle coordinates. Construct the disparity estimation objective function E d And the light field restoration objective function E r , the reconstruction objective function E is obtained through weighted combination.

[0059] Disparity estimation objective function E d It is used to minimize the parallax error between light field data from different viewing angles. The expression is as follows: Among them, P i,j P represents the reference view pixel block at position (i, j); m,n Represents the reference view pixel block P i,j The corresponding pixel blocks of other viewing angles are located at position (m, n); w i,j,m,n Represents the reference view pixel block P i,j With other perspective pixel blocks P m,n The weight coefficient between is used to balance the contribution between pixel blocks of different perspectives; V i,j,m,nrepresents the perspective prior offset between pixel blocks of different perspectives in a vehicle environment, which is estimated based on the prior knowledge of vehicle posture and road scene; ▽d i,j Represents the gradient of the disparity map, including the gradients in the horizontal and vertical directions; M i,j represents the parallax change caused by vehicle motion, which is obtained by estimating the vehicle motion parameters; γ represents the balance factor, which is used to control the influence weight of the perspective prior term and the motion constraint term. By using the prior knowledge of the vehicle environment, the pixel offset relationship between different perspectives is estimated in advance, providing a priori constraints for the parallax estimation, reducing the search space of the parallax estimation, and improving the estimation accuracy. At the same time, by introducing the motion constraint term M i,j , considering the impact of vehicle motion on disparity estimation, the disparity estimation result is made more consistent with the actual situation of vehicle motion and the robustness of disparity estimation is improved.

[0060] Light field restoration objective function E r It is used to minimize the reconstruction error of light field data. The expression is as follows: Where D represents the sparse dictionary matrix, which is a predefined matrix used to map light field data to the sparse representation space. Each column of the dictionary matrix represents a basic element, and the light field data can be represented by the linear combination of these basic elements; α represents the sparse representation sparse vector, which represents the projection coefficient of the light field data on the sparse dictionary D, that is, the light field data can be approximately represented by the linear combination of the basic elements in the dictionary matrix, and α is the coefficient of these linear combinations. L r represents the reconstructed light field data, which represents the light field data obtained after light field restoration; I(i,j) represents the illumination prior information in the vehicle environment, which represents the illumination information estimated according to the vehicle position and ambient lighting conditions in the vehicle environment, and I(i,j) represents the illumination intensity and color information at the (i,j)th position of the light field data; S(i,j) represents the scale factor of the illumination prior information, which is used to adjust the influence weight of the illumination prior information I(i,j) in the light field restoration objective function, and the value of S(i,j) can be set according to the lighting conditions and restoration requirements; V i,j,m,n V represents the parallax consistency constraint, which means that the light field restoration is constrained by the parallax estimation result to ensure the consistency between the light field restoration result and the parallax estimation result. i,j,m,n represents the parallax difference between the (i, j)th position and the (m, n)th position in the light field data; λ1 represents the balance factor of the total variation regularization term, which is used to control the total variation regularization term TV(L r ) is the weight in the light field restoration objective function to balance the spatial continuity and local smoothness of the light field data; λ2 represents the balance factor of the illumination prior term, which is used to control the illumination prior term The weight in the light field restoration objective function is used to balance the impact of the illumination prior information on the restoration result; λ3 represents the balance factor of the parallax consistency constraint term, which is used to control the parallax consistency constraint term The weight in the objective function of light field restoration is used to balance the degree of constraint of the parallax estimation result on the restoration result; by introducing the illumination prior I(i,j) and the scale factor S(i,j), the illumination information at different positions is estimated using the illumination prior knowledge in the vehicle environment, providing prior constraints for light field restoration, making the restoration result more consistent with the actual lighting conditions and improving the realism of light field restoration. At the same time, by introducing the parallax consistency constraint V i,j,m,n ,The disparity estimation results are used to constrain the light field restoration, ensuring the consistency of the restoration results with the disparity estimation results, and improving the accuracy and consistency of the light field restoration.

[0061] The disparity estimation objective function E d And the light field restoration objective function E r Weighted combination, we get the reconstruction objective function E: E = E d +ξ×E r , where ξ represents the weight coefficient. The reconstruction objective function E is solved by the alternating direction multiplier method to obtain the disparity map D n,n+1 (i, j) and reconstructed light field data L n (i,j,u,v). Convert the reconstruction objective function E into the augmented Lagrangian function L:

[0062]

[0063] Among them, λ i,j,n represents the Lagrange multiplier, Z n,n+1 (i,j) represents the scale variable, and β represents the penalty parameter. Decompose the augmented Lagrangian function L into multiple sub-problems and solve each sub-problem alternately: Fixed D n,n+1 (i,j), solve for Z n,n+1 (i,j) and S(i,j):

[0064]

[0065] Fixed Z n,n+1 (i,j) and S(i,j), solve for D n,n+1 (i,j):

[0066]

[0067] Update the Lagrange multiplier λ i,j,n :λ i,j,n '=λ i,j,n +β×(D n,n+1 (i,j)-Z n,n+1(i,j)). By iteratively solving the above sub-problems until convergence, the final disparity map D is obtained. n,n+1 (i, j) and reconstructed light field data L n (i,j,u,v).

[0068] According to the disparity map D n,n+1 (i, j), calculate the correspondence between the light field data of the nth viewing angle and the n+1th viewing angle. For the light field data L of the nth viewing angle n Each angle coordinate (u,v) in (i,j,u,v) is obtained through the disparity map D n,n+1 (i, j) calculates its corresponding spatial coordinates (i', j') at the n+1th viewing angle: i'=i+D n,n+1 (i,j)×u,j'=j+D n,n+1 (i, j) × v. Through the above calculation, the corresponding relationship between the angular coordinates (u, v) of the nth viewing angle and the spatial coordinates (i', j') of the n+1th viewing angle is established.

[0069] According to the correspondence between perspectives, a mapping relationship M from the nth perspective to the n+1th perspective is established n,n+1 For each angle coordinate (u, v) of the nth viewing angle, the corresponding light field data value L at the n+1th viewing angle is calculated by interpolation: n+1 (i',j',u,v): Among them, (i', j') is the corresponding spatial coordinates of the angular coordinates (u, v) of the nth perspective at the n+1th perspective, K is the interpolation window size, and w k,l is the interpolation weight coefficient. Through the above calculation, the mapping relationship M from the nth perspective to the n+1th perspective is established n,n+1 , to achieve the correspondence and conversion between light field data of different perspectives.

[0070] Using the mapping relationship between perspectives M n,n+1 , to reconstruct the light field data L n (i, j, u, v) is interpolated to obtain multi-view aerial imaging data. For any target view t, the light field data L of the target view is calculated by weighted combination of the mapping relationship between two adjacent views. t (i,j,u,v):

[0071] L t (i,j,u,v)=(1-a)×M n,t (L n (i,j,u,v))+α×M n+1,t (L n+1 (i,j,u,v)), where, M n,t and M n+1,tRespectively represent the mapping relationship between the nth and n+1th perspectives to the target perspective t, and α is the interpolation weight coefficient, which represents the relative position of the target perspective t between the nth and n+1th perspectives. Through the above calculations, the disparity map is used to establish the mapping relationship between perspectives, and the light field data under any target perspective is generated by weighted interpolation to obtain multi-perspective aerial imaging data.

[0072] According to the multi-view light field data L obtained by interpolation t (i, j, u, v), through the light field rendering algorithm, generate multi-view aerial imaging data I t (x, y), where t represents the target view number and (x, y) represents the imaging plane coordinates. The light field rendering algorithm generates aerial imaging from the target view by reprojecting and accumulating the light field data: t (x,y)=∑ u,v L t (x, y, u, v). Through the above steps, the disparity map is used to calculate the correspondence between perspectives, establish the mapping relationship between perspectives, and generate multi-perspective light field data through perspective interpolation, and finally generate multi-perspective aerial imaging data through the light field rendering algorithm.

[0073] Use a plane reflector to reflect the virtual image corresponding to the aerial imaging data to generate real image data; obtain the exit angle data of the DCTplate negative reflection flat lens and the position parameters of the plane reflector, and obtain the exit angle data θ0 of the DCTplate negative reflection flat lens, which represents the angle of the aerial imaging light field data when it exits from the DCTplate negative reflection flat lens. Obtain the position parameters of the plane reflector (x m ,y m ,z m ), which represents the position coordinates of the plane mirror in three-dimensional space.

[0074] Determine the reflection angle parameters of the plane reflector, according to the output angle data θ0 of the DCTplate negative reflection flat lens and the position parameters (x m ,y m ,z m ), through geometric optics calculation, determine the reflection angle parameter θ of the plane reflector r . Assume that the virtual image position of the aerial imaging data is (x v ,y v ,z v ), the reflection angle θ is calculated by the following geometric relationship r : Through the above calculation, the reflection angle parameter θ of the plane reflector is determined r , so that the reflected light can be accurately projected onto the preset imaging plane.

[0075] The aerial imaging light field data is reflected and processed. The aerial imaging light field data L(x, y, u, v) is input into the plane reflector. The reflection characteristics of the plane reflector are used to reflect the aerial imaging light field data irradiated on the reflection surface. According to the law of reflection, for the incident light (u, v), the direction of the reflected light (u', v') satisfies the following relationship: u'=ucos2θ r -vsin2θ r , v'=ucos2θ r +vsin2θ r .

[0076] For each light direction (u, v) in the aerial imaging light field data, the above reflection transformation is used to obtain the reflected light direction (u', v'), and generate the reflected aerial imaging light field data L'(x, y, u', v').

[0077] The reflected aerial imaging light field data is projected onto the imaging plane, and the position parameters (x s ,y s ,z s ), project the reflected aerial imaging light field data L'(x, y, u', v') onto the preset imaging plane. For each reflected light ray (x, y, u', v'), calculate its projection coordinates (x i ,y i ): Through the above projection transformation, the reflected aerial imaging light field data is projected onto the preset imaging plane to obtain the real image data I(x i ,y i ). Through the above steps, the virtual image corresponding to the aerial imaging data is reflected by a plane reflector, and the reflected light field data is projected onto a preset imaging plane to generate real image data.

[0078] The microphone array in the in-vehicle entertainment system is used to collect the user's input voice, and a microphone array M={M1, M2, ..., M N}, where N is the number of microphones. When the user inputs a voice, the microphone array collects the audio signal and obtains the original voice data X(t), where t is the time dimension. The collected original voice data X(t) is preprocessed, including voice noise reduction, voice endpoint detection, and voice feature extraction. The original voice data X(t) is denoised using noise reduction algorithms such as spectral subtraction or deep neural networks to remove environmental noise.

[0079] Noise and background interference are eliminated to obtain the noise-reduced speech data For the noise-reduced speech data Perform endpoint detection to determine the start and end positions of speech. Set the threshold θ based on features such as energy or zero-crossing rate to divide speech data into speech segments and non-speech segments. Perform feature extraction to extract the frequency domain and time domain features of speech. Commonly used speech features include Mel frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC), etc. After extracting speech features, the preprocessed speech feature data F = {f1, f2, ..., f L}, where L represents the feature dimension.

[0080] In a specific embodiment of the present application, the original speech data X(t) is subjected to a short-time Fourier transform (STFT) to obtain the spectrum X(f, t) of the speech signal. The spectrum N(f, t) of the background noise is estimated using a spectral subtraction algorithm, and the denoised speech spectrum is obtained by spectral subtraction. The spectrum of the speech after noise reduction Perform inverse short-time Fourier transform (ISTFT) to obtain the noise-reduced speech data For the noise-reduced speech data Perform frame processing to obtain a series of speech frames Where i represents the index of the frame. Calculate the energy feature E for each frame of speech data i Or zero-crossing rate characteristic Z i Energy characteristics Where N represents the frame length. Zero crossing rate feature Where sgn(*) represents the sign function. Set the energy threshold θ E Or zero-crossing rate threshold θ Z , divide the speech frame into speech segments and non-speech segments. If E i >θ E or Z i >θ Z , then mark the frame as a speech segment. Otherwise, mark the frame as a non-speech segment. Merge consecutive speech segments to get a complete speech segment.

[0081] Dialogue Perform pre-emphasis to enhance the high frequency part and obtain the pre-emphasized speech signal For pre-emphasized speech signals Perform framing and windowing to obtain a series of speech frames Perform fast Fourier transform (FFT) on each frame of speech data to obtain the spectrum of the speech frame The spectrum Filter through the Mel filter bank to get the Mel spectrum M s,i(m), where m represents the index of the Mel frequency. s,i (m) Perform discrete cosine transform (DCT) to obtain MFCC feature c s,i (l), where l represents the index of the MFCC coefficient. The MFCC features of all speech frames are concatenated to obtain a complete MFCC feature sequence F = {f1,f2,......,f L}, where L represents the feature dimension.

[0082] The pre-trained convolutional neural network model CNN is used to recognize the pre-processed speech feature data F to obtain the speech recognition result data. The input of the convolutional neural network model CNN is the speech feature data F, and the output is the probability distribution P of the speech recognition result = {p1, p2, ..., p K}, where K represents the number of speech recognition categories. The convolutional neural network model CNN extracts high-level representations of speech features through convolutional layers and pooling layers, and outputs the probability distribution of speech recognition results through fully connected layers and Softmax layers. By finding the category k* with the highest probability in the probability distribution P, the speech recognition result data R = C k* , where C = {c1, c2, ..., c K} represents a preset speech recognition category set.

[0083] According to the preset voice control instruction set I={I1,I2,......,I M}, converting the speech recognition result data R into speech control instruction data D matching the speech control instruction set as the first control instruction. Through a matching algorithm, such as keyword matching or semantic analysis, the speech recognition result data R is matched with the instructions in the speech control instruction set I to find the corresponding speech control instruction data D. If the match is successful, the speech control instruction data D is used as the first control instruction to control the operation of the in-vehicle entertainment system.

[0084] The infrared sensor is used to collect the image data of the user's gesture, and image recognition is performed to obtain the gesture control instruction data as the second control instruction; the infrared sensor array is used to collect the gesture image data and extract the gesture area. In the in-vehicle entertainment system, an array composed of N infrared sensors IR={IR1, IR2, ......, IR N}. The infrared sensor array can be laid out in the following ways: Linear layout: Arrange N infrared sensors at equal intervals in a straight line to form a one-dimensional infrared sensor array. Matrix layout: Arrange N infrared sensors into an m×n=N matrix, where m represents the number of rows and n represents the number of columns, satisfying m×n=N. Ring layout: Distribute N infrared sensors evenly on a ring to form a ring-shaped infrared sensor array. The following factors need to be considered when selecting infrared sensors: Wavelength range: Select an infrared wavelength range suitable for gesture recognition, usually in the near-infrared band (700nm-1000nm). Resolution: Select a sufficiently high spatial resolution to accurately capture the detailed information of the gesture. Frame rate: Select a higher frame rate to achieve real-time gesture recognition and interactive response. Field of view: Select an appropriate field of view to cover the spatial range of user gesture interaction.

[0085] The infrared sensor array continuously collects raw image data at a certain frame rate (such as 30 frames per second). Each infrared sensor unit works independently, receives the reflected infrared light, and converts the light intensity into an electrical signal. The electrical signals of all infrared sensor units constitute a frame of raw image data, which is represented as a two-dimensional matrix I (x, y, t), where x and y represent the coordinates of the sensor unit and t represents time.

[0086] Extract the gesture area location information, preprocess the original image data I(x, y, t), and obtain the preprocessed image data I'(x, y, t). Use a Gaussian filter to denoise the original image data, and the standard deviation of the filter is σ, and obtain the denoised image data I g (x, y, t). The denoised image data is smoothed using a mean filter, with a filter window size of w×w, to obtain the smoothed image data I s (x, y, t). Add the denoised and smoothed image data to obtain the preprocessed image data I'(x, y, t) = I g (x,y,t)+I s (x,y,t).

[0087] Through image segmentation algorithms such as threshold segmentation, the location information of the gesture area is extracted from the preprocessed image data I'(x,y,t). Set the threshold θ, binarize the preprocessed image data, and obtain the binary mask M(x,y,t) of the gesture area. If I'(x,y,t)>θ, then M(x,y,t)=1, indicating that the pixel belongs to the gesture area. Otherwise, M(x,y,t)=0, indicating that the pixel belongs to the background area. Perform morphological processing on the binary mask M(x,y,t) to remove noise and holes and obtain the outline of the gesture area. Use the dilation operator to dilate the binary mask to connect the disconnected gesture areas. The structural element of the dilation operator is sd The erosion operator is used to erode the dilated binary mask to remove small noise areas. The structural element of the erosion operator is s e . Perform contour extraction on the eroded binary mask to obtain the contour C(x,y,t) of the gesture area. Calculate the geometric center (x c ,y c ), as the position coordinates of the gesture area. Calculate the moment of the contour According to the definition of moment, the position coordinates of the gesture area are

[0088] The depth information of the gesture area is extracted based on the three-dimensional structure of the infrared sensor array and the reflection principle of infrared light. For each infrared sensor unit (i, j), the time difference Δt between the emission and reception of infrared light is calculated. i,j and the speed of light c, calculate the distance d from the gesture area to the sensor unit i,j The infrared sensor unit emits an infrared light pulse and records the emission time t e The infrared sensor unit receives the reflected infrared light pulse and records the receiving time t r . Calculate the time difference between transmission and reception Δt i,j =t r -t e According to the speed of light c and the time difference Δt i,j , calculate the distance from the gesture area to the sensor unit The distance value d for all sensor units i,j Perform average or median filtering to obtain the average depth of the gesture area Average filtering: Where N is the total number of sensor units. Median filter: Where median(*) represents the median operation. Through the above steps, the position coordinates (x c ,y c ) and average depth The extraction of gesture area position information is mainly achieved through image processing techniques such as image preprocessing, threshold segmentation, morphological processing and contour analysis, while the extraction of gesture area depth information utilizes the three-dimensional structure of the infrared sensor and the reflection principle of infrared light, and is obtained by calculating the time difference and the speed of light. c ,y c ) and average depth The infrared image data of the gesture area is extracted from the original image data I(x, y, t).

[0089] Define the region of interest (ROI) of the gesture area as (x c ,y c ) as the center and a square area with a side length of L, where L is set according to the size of the gesture area and the requirements of gesture recognition. Extract the infrared image data corresponding to ROI from the original image data g (x, y, t), as the infrared image data of the gesture area. g (x, y, t) is preprocessed by size normalization, brightness normalization, etc. to obtain the normalized gesture infrared image data

[0090] The pre-trained convolutional neural network model is used for gesture recognition. The infrared image data of the gesture area is trained using the pre-trained convolutional neural network model CNN. g (x, y, t) to extract and classify features and obtain gesture recognition result data. The input of the convolutional neural network model CNN is the infrared image data I of the gesture area. g (x, y, t), the output is the probability distribution of gesture recognition results P = {p1, p2, ..., p K}, where K represents the number of gesture recognition categories. The convolutional neural network model CNN extracts high-level features of gesture images through convolutional layers and pooling layers, and outputs the probability distribution of gesture recognition results through fully connected layers and Softmax layers. By finding the category k* with the highest probability in the probability distribution P, the gesture recognition result data P(G k* ), where G = {G1, G2, ..., G K} represents a set of preset gesture recognition categories.

[0091] The gesture recognition result data is converted into gesture control instruction data, according to the preset gesture control instruction set H = {H1, H2, ..., H M}, the gesture recognition result data R is converted into gesture control instruction data D matching the control instruction set as the second control instruction. The gesture recognition result data R is matched with the instructions in the gesture control instruction set H through a matching algorithm, such as gesture template matching or rule mapping, to find the corresponding gesture control instruction data D. If the match is successful, the gesture control instruction data D is used as the second control instruction to control the operation of the in-vehicle entertainment system.

[0092] Generate interactive control data according to the three-dimensional coordinates in the real image data, as well as the first control instruction and the second control instruction; first, it is necessary to obtain the three-dimensional coordinate data in the real image data. The real image data can be obtained in the following ways: Depth camera: Use a depth camera (such as Intel RealSense, Microsoft Kinect, etc.) to collect real-time depth image data. The depth camera obtains the depth value of each pixel in the scene through infrared structured light or time-of-flight technology to form a depth image D (x, y). Binocular camera: Use a binocular camera system to calculate the three-dimensional coordinate data in the scene through the parallax information of the left and right cameras. The binocular camera calculates the parallax value of the feature points by matching the feature points in the left and right images, and reconstructs the three-dimensional coordinate information according to the camera parameters and the parallax value. SLAM system: Use the SLAM (Simultaneous Localization and Mapping) system to collect real-time image and motion data through mobile devices (such as cameras, IMU, etc.), and reconstruct the three-dimensional structure of the scene and the motion trajectory of the camera in real time through the SLAM algorithm to obtain three-dimensional coordinate data.

[0093] After obtaining the real image data, the data needs to be preprocessed and the coordinates converted: filter and smooth the depth image to remove noise and outliers. Convert each pixel point (x, y, d) in the depth image into three-dimensional space coordinates (X, Y, Z), and calculate the corresponding three-dimensional coordinate value based on the camera's intrinsic parameters (such as focal length, principal point coordinates, etc.) and the depth value d. Adjust the scale and coordinate system of the three-dimensional coordinate data obtained by the binocular camera or SLAM system to make it consistent with the coordinate system of the virtual object. After the above steps, the three-dimensional coordinate data in the real image data can be obtained, which is represented as a set of three-dimensional point clouds P = {p1, p2, ..., p N}, where each point p i =(X i ,Y i ,Z i ) represents a three-dimensional space coordinate.

[0094] The 3D space interaction model is constructed by the 3D reconstruction algorithm, and the 3D space is divided into a regular voxel grid (VoxelGrid), where each voxel represents a cubic area. For each voxel, the number of 3D points inside it is counted, and whether the voxel is occupied is determined based on the number threshold. The surface of the occupied voxels is reconstructed to generate a continuous 3D surface model. Common surface reconstruction algorithms include MarchingCubes algorithm, Poisson surface reconstruction algorithm, etc. The surface reconstruction algorithm generates a 3D model surface represented by a triangle mesh (TriangleMesh). The position and shape of virtual objects, such as virtual buttons and virtual menus, are defined in the 3D space. The reconstructed 3D surface model is subjected to collision detection to determine whether the user's hand or interactive device collides with the virtual object. Common collision detection algorithms include collision detection based on hierarchical bounding box (Hierarchical Bounding Box) and collision detection based on distance field (Distance Field). According to the collision detection results, the interactive mapping relationship between the user and the virtual object is established. When the user's hand or interactive device collides with the virtual object, the corresponding interactive event is triggered, such as button click, menu selection, etc. Map interactive events to specific application functions or control instructions to enable interaction between users and virtual objects.

[0095] Speech recognition: Perform speech recognition on the user's voice commands and convert the voice signals into text-formatted commands. Common speech recognition algorithms include speech recognition based on hidden Markov models (HMM), speech recognition based on deep neural networks (DNN), etc. The result of speech recognition is a string of text, which represents the content of the user's voice commands. Semantic understanding: Perform semantic understanding on the recognized text commands and extract key information in the commands, such as action type, direction, speed, etc. Common semantic understanding methods include rule-based semantic parsing and machine learning-based intent recognition. The result of semantic understanding is structured command information, such as (action type: move, direction: forward, speed: medium). Parameter mapping: Based on the results of semantic understanding, the command information is mapped into motion parameters of the virtual object. Motion parameters include displacement vectors (indicates the moving direction and distance of the virtual object), speed v (indicates the moving speed of the virtual object), etc. Parameter mapping requires pre-defined mapping rules, such as "forward" corresponds to "Medium speed" corresponds to v=1.0, etc. Through the above steps, the voice control instruction data is converted into motion parameters of the virtual object, which are used to control the movement of the virtual object in the three-dimensional space.

[0096] Convert gesture control command data into posture parameters of virtual objects. Gesture recognition: Recognize user gestures and convert gesture data into discrete gesture categories or continuous gesture trajectories. Common gesture recognition algorithms include gesture recognition based on template matching and gesture classification based on machine learning. The result of gesture recognition is a gesture category (such as "rotation", "scaling", etc.) or a gesture trajectory (such as the movement path of a finger). Posture estimation: Estimate the posture parameters of virtual objects, such as rotation angle, scaling, etc., based on the results of gesture recognition. Posture estimation needs to consider the semantic information and geometric features of gestures, such as the direction, speed, and amplitude of gestures. Common posture estimation methods include posture estimation based on key points and posture estimation based on skeleton models. Parameter mapping: Map the posture parameters to the posture transformation matrix of the virtual object based on the results of posture estimation. The posture transformation matrix includes the rotation matrix R (indicating the rotation of the virtual object) and the scaling matrix S (indicating the scaling of the virtual object). Parameter mapping requires pre-defined mapping rules, such as "rotation" corresponds to the rotation matrix around a certain axis, and "scaling" corresponds to the scaling matrix along a certain direction. Through the above steps, the gesture control command data is converted into the posture parameters of the virtual object, which are used to control the posture transformation such as rotation and scaling of the virtual object in three-dimensional space.

[0097] According to the motion parameters and posture parameters, the spatial position and interaction state of the virtual object in the three-dimensional spatial interaction model are updated. Spatial position update: According to the displacement vector in the motion parameters and velocity v, calculate the new position of the virtual object in three-dimensional space. Assume that the current position of the virtual object is P0(x0,y0,z0), then after time Δt, the new position of the virtual object P1(x1,y1,z1) can be calculated by the following formula:

[0098] Update the position coordinates of the virtual object in the three-dimensional space interaction model to P1 (x1, y1, z1). Interaction state update: Update the posture and size of the virtual object according to the rotation matrix R and scaling matrix S in the posture parameters. Assuming that the current posture transformation matrix of the virtual object is M0, after the posture transformation, the new posture transformation matrix M1 of the virtual object can be calculated by the following formula: M1 = M1 × R × S; Update the posture transformation matrix of the virtual object in the three-dimensional space interaction model to M1. Collision detection and response: After updating the spatial position and posture of the virtual object, collision detection is required to determine whether the virtual object collides with other objects. If a collision occurs, it is necessary to calculate the collision response, such as bounce, stop motion, etc., according to the type and strength of the collision. Update the status of the virtual object in the three-dimensional space interaction model, such as velocity, acceleration, collision marker, etc.

[0099] Get the spatial position of the user and the virtual object: Get the user's position coordinates P from the updated three-dimensional spatial interaction model u (x u ,y u ,z u ) and the position coordinates P0(x0,y0,z0) of the virtual object. The user's position coordinates can be obtained through head tracking, hand tracking and other technologies, indicating the user's position in the three-dimensional space. The position coordinates of the virtual object have been calculated when updating the three-dimensional space interaction model, indicating the position of the virtual object in the three-dimensional space.

[0100] According to the position coordinates of the user and the virtual object, the Euclidean distance d between them is calculated. The spatial distance d can be calculated by the following formula: The spatial distance d represents the straight-line distance between the user and the virtual object, which is used to determine whether they are close.

[0101] Assume the user's posture transformation matrix is ​​M u , the posture transformation matrix of the virtual object is M0. Calculate the relative posture transformation matrix M of the user relative to the virtual object r , the formula is as follows: From the relative posture transformation matrix M r Extract the rotation angle θ from r (such as Euler angles, quaternions, etc.). The extracted rotation angle θ r and the preset relative posture threshold θ threshold Compare and determine whether the relative posture between the user and the virtual object meets the interaction conditions. Through the above steps, the spatial distance d between the user and the virtual object and the relative posture matrix M are calculated. rel , providing a basis for judging whether the preset interaction conditions are met.

[0102] According to the calculated spatial distance and relative posture, determine whether the preset interaction conditions are met. Determine whether the spatial distance meets the threshold condition: compare the calculated spatial distance d with the preset spatial distance threshold d threshold Compare. If d≤d threshold , then it is considered that the distance between the user and the virtual object meets the interaction condition. Spatial distance threshold d threshold It can be set according to specific application scenarios and interaction requirements, indicating the maximum interaction distance between the user and the virtual object.

[0103] Determine whether the relative posture meets the threshold condition: the calculated relative posture matrix M rel Compare with the preset relative attitude threshold. The relative attitude threshold can be calculated by the Euler angle threshold α threshold , β threshold , γ thresholdor quaternion threshold q threshold If the relative posture matrix M rel If the preset threshold condition is met, it is considered that the relative posture between the user and the virtual object meets the interaction condition. The relative posture threshold can be set according to the specific application scenario and interaction requirements, indicating the maximum posture deviation between the user and the virtual object.

[0104] Comprehensively determine whether the interaction condition is met: If both the spatial distance and the relative posture meet the preset threshold conditions, it is determined that the interaction condition is met between the user and the virtual object. If one of the spatial distance or the relative posture does not meet the preset threshold conditions, it is determined that the interaction condition is not met between the user and the virtual object. Through the above steps, it is determined whether the interaction condition is met between the user and the virtual object based on the preset spatial distance threshold and relative posture threshold.

[0105] If it is determined that the preset interaction conditions are met between the user and the virtual object, it is further determined that there is an interaction behavior between the user and the virtual object, and the corresponding interaction event is triggered. Determine the type of interaction behavior: Determine the type of interaction behavior between the user and the virtual object based on the user's gesture, position, speed and other information. Common types of interaction behaviors include: Collision: The user's hand or interactive device comes into contact with or intersects with the virtual object. Selection: The user selects the virtual object through gestures or cursor. Grab: The user grabs or drags the virtual object through gestures or buttons. The determination of the type of interaction behavior can be achieved through preset rules or machine learning algorithms, and classified and identified based on the user's interaction data.

[0106] Trigger corresponding interactive events: trigger corresponding interactive events based on the interactive behavior type determined. Interactive events may include: Collision event: when a user collides with a virtual object, a collision event is triggered, which can play collision sound effects, trigger collision animations, etc. Selection event: when a user selects a virtual object, a selection event is triggered, which can highlight the selected object, display object information, etc. Grab event: when a user grabs a virtual object, a grab event is triggered, which can attach the virtual object to the user's hand or interactive device and move it accordingly. The triggering of interactive events can be achieved by registering event listeners or callback functions. When a specific interactive behavior occurs, the corresponding event processing logic is executed.

[0107] Update the 3D space interaction model: Update the status and properties of virtual objects in the 3D space interaction model according to the triggered interaction events. For example, when a virtual object is selected, its material, color and other properties can be updated; when a virtual object is grabbed, its position, posture and other status can be updated. By updating the 3D space interaction model, the virtual object can respond to and feedback from user interactions, providing a real-time and dynamic interaction experience.

[0108] Predefinition of interactive control instruction sets: predefine a set of interactive control instruction sets, including control instructions corresponding to different types of interactive events. Each control instruction contains information such as instruction type, parameters, and execution conditions. For example, control instructions corresponding to collision events may include "stop motion", "bounce", etc.; control instructions corresponding to selection events may include "highlight display", "open menu", etc. Selection of interactive control instructions: select the corresponding interactive control instruction from the preset interactive control instruction set according to the type of interactive event triggered. Different control instructions can be selected according to the parameters of the interactive event (such as collision force, selection duration, etc.). The selected interactive control instruction contains the instruction type and related parameters.

[0109] Generation of interactive control instruction data: Generate corresponding interactive control instruction data based on the selected interactive control instruction. The interactive control instruction data contains information such as instruction type, parameter value, timestamp, etc. The instruction type indicates the specific operation to be performed, such as "stop motion", "highlight display", etc. The parameter value indicates the specific parameters required when performing the operation, such as collision force, highlight color, etc. The timestamp indicates the generation time of the instruction, which is used for synchronization and timing control. Transmission of interactive control instruction data: Send the generated interactive control instruction data to the three-dimensional space interactive model and related applications. The three-dimensional space interactive model updates the state and behavior of the virtual object according to the received interactive control instruction data. The application performs corresponding operations and functions according to the received interactive control instruction data, such as playing sound effects, updating the interface, etc.

[0110] Get the spatial position, posture and attribute information of the virtual object from the three-dimensional space interaction model. Get the position coordinates (x, y, z) of the virtual object in the three-dimensional space interaction model. The position coordinates represent the absolute position of the virtual object in the three-dimensional space. Posture information: Get the posture transformation matrix M of the virtual object in the three-dimensional space interaction model. The posture transformation matrix represents the rotation and scaling information of the virtual object relative to the three-dimensional space coordinate system. Extract the rotation angle (α, β, γ) (such as Euler angle) and scaling factor (s) from the posture transformation matrix. x ,s y ,s z ). Attribute information: Get the attribute information of the virtual object, such as object type, identifier, status, etc. Attribute information can be defined and extended according to specific application requirements. For example, for a virtual button object in an in-vehicle entertainment system, the attribute information may include button type (such as play, pause, etc.) and button status (such as pressed, released, etc.).

[0111] The generated interactive control instruction data is integrated with the spatial position, posture and attribute information of the virtual object in the three-dimensional spatial interaction model to generate interactive control data for controlling the in-vehicle entertainment system. The interactive control instruction data is aligned with the virtual object information according to the timestamp or serial number. The synchronization of the interactive control instruction data and the virtual object information in the time dimension is ensured. According to the instruction type and parameters in the interactive control instruction data, the attribute information of the virtual object is associated. For example, the "select" instruction is associated with the type and state of the virtual button object, and the "grab" instruction is associated with the angle information of the virtual knob object. According to the spatial position and posture information of the virtual object, the position and direction parameters in the interactive control instruction data are transformed.

[0112] The coordinate system of the interactive control command data is converted from the user coordinate system to the coordinate system of the in-vehicle entertainment system. For example, the user's gesture position is converted from the user coordinate system to the screen coordinate system of the in-vehicle entertainment system. A standardized interactive control data format is defined to encapsulate the fused interactive control command data and virtual object information. The interactive control data format can adopt common data serialization formats such as JSON, XML, and Protocol Buffers.

[0113] The interactive control data format should contain the following fields: Instruction type: indicates the type of interactive control instruction, such as "select", "grab", etc. Parameter value: stores specific parameter values ​​related to the instruction, such as the identifier of the virtual object, operation status, etc. Target object: specifies the target virtual object of the interactive control instruction. Timestamp: records the generation time of the interactive control data, which is used for timing synchronization and control. Encode and encapsulate the fused interactive control instruction data and virtual object information according to the defined data format.

[0114] The encapsulated interactive control data is sent to the in-vehicle entertainment system through the in-vehicle network. The in-vehicle network can use in-vehicle communication protocols and physical layer standards such as CAN, Ethernet, and MOST. Before the data is sent, the interactive control data is compressed, encrypted, and error-checked as necessary to ensure the security and integrity of the data. After receiving the interactive control data, the in-vehicle entertainment system parses the data format and extracts information such as the instruction type, parameter value, and target object. Based on the received interactive control data, the in-vehicle entertainment system performs corresponding operations and functions, such as switching interfaces, adjusting volume, and playing media. After executing the operation, the in-vehicle entertainment system can send feedback information to the interactive control system to confirm the execution result and status of the instruction.

Claims

1. A vehicle-mounted VR rear-seat entertainment method for controlling a vehicle-mounted entertainment system, characterized in that: include: Through the controller of the in-vehicle entertainment system, the LCD display system, the holographic projection screen system, and the AR glasses system are connected respectively, and the image data is transmitted to each system; In the holographic projection screen system: The image data transmitted by the controller is processed through the negative reflection flat lens of the DCT plate to generate aerial imaging data; A plane mirror is used to reflect the virtual image corresponding to the aerial imaging data to generate real image data; Performing voice recognition processing on the user's input voice to obtain voice control instruction data as the first control instruction; The infrared sensor is used to collect image data of the user's gesture, and image recognition is performed to obtain gesture control instruction data as the second control instruction; Generate interactive control data according to the three-dimensional coordinates in the real image data, and the first control instruction and the second control instruction; Transmit interactive control data to the controller to control the in-vehicle entertainment system; Generate multi-view aerial imaging data including: Through multiple light field lenses, aerial imaging light field data at different viewing angles are collected to obtain multi-view light field data; Construct objective functions for disparity estimation and light field restoration; the disparity estimation objective function is used to minimize the disparity error between light field data from different perspectives, and the light field restoration objective function is used to minimize the reconstruction error of light field data; The disparity estimation objective function and the light field restoration objective function are weightedly combined to obtain the reconstruction objective function; The reconstruction objective function is solved by using the alternating direction multiplier method. By converting the reconstruction objective function into an augmented Lagrangian function and introducing a scale variable, the problem is decomposed into multiple sub-problems. Each sub-problem is solved alternately to obtain the disparity map and reconstructed light field data. According to the disparity map, by calculating the correspondence between light field data of different perspectives, a mapping relationship between perspectives is established, and the reconstructed light field data is interpolated using the mapping relationship to obtain multi-perspective aerial imaging data; The disparity estimation objective function expression is as follows: Among them, P i,j P represents the reference view pixel block at position (i, j); m,n Represents the reference view pixel block P i,j The corresponding pixel blocks of other viewing angles are located at position (m, n); w i,j,m,n Represents the reference view pixel block P i,j With other perspective pixel blocks P m,n The weight coefficient between them is used to balance the contribution between pixel blocks of different viewing angles; V i,j,m,n Represents the perspective prior offset between pixel blocks of different perspectives in a vehicle environment; Represents the gradient of the disparity map, including the gradients in the horizontal and vertical directions; M i,j represents the parallax change caused by vehicle motion; γ represents the balance factor, which is used to control the influence weight of the perspective prior term and the motion constraint term.

2. The vehicle-mounted VR rear-seat entertainment method according to claim 1, characterized in that: Generate aerial imaging data including: Preprocess the image data, including image denoising, image enhancement and image correction; Through the optical diffraction and phase modulation of the DCTplate negative reflection flat lens, the incident image data is diffracted and phase modulated to generate three-dimensional aerial imaging light field data; Based on the three-dimensional aerial imaging light field data, multi-view aerial imaging data is generated through multi-view acquisition and reconstruction algorithms.

3. The vehicle-mounted VR rear-seat entertainment method according to claim 2, characterized in that: Generate 3D aerial imaging light field data, including: According to the parameters of the diffraction grating of the DCTplate negative reflection flat lens, the spatial frequency of the image data is modulated to generate frequency domain modulated image data; wherein the parameters of the diffraction grating include a grating constant; According to the orientation state of liquid crystal molecules in the phase modulation liquid crystal layer of the DCTplate negative reflection flat lens, phase modulation is performed on the image data after frequency domain modulation to generate phase modulated image data; The light field wavefront of the phase modulated image data is reconstructed through the output end of the DCTplate negative reflection flat lens to generate three-dimensional aerial imaging light field data.

4. The vehicle-mounted VR rear-seat entertainment method according to claim 1, characterized in that: The objective function expression of light field restoration is as follows: Where D represents the sparse dictionary matrix; a represents the sparse vector; L r represents the reconstructed light field data; I(i,j) represents the illumination prior information in the vehicle environment; S(i,j) represents the scale factor of the illumination prior information; V i,j,m,n represents the parallax consistency constraint; λ1 represents the balance factor of the total variation regularization term; λ2 represents the balance factor of the illumination prior term; λ3 represents the balance factor of the parallax consistency constraint term.

5. The vehicle-mounted VR rear-seat entertainment method according to any one of claims 2 to 4, characterized in that: Generate real image data, including: Obtain the output angle data of the DCT plate negative reflection flat lens and the position parameters of the plane reflector, and determine the reflection angle parameters of the plane reflector through geometric optical calculation according to the output angle data and the position parameters; The aerial imaging light field data is input into a plane reflector, and the aerial imaging light field data irradiated onto the reflective surface is reflected by using the reflective characteristics of the plane reflector to obtain the aerial imaging light field data after reflection; wherein the aerial imaging light field data contains virtual image information corresponding to the aerial imaging data; According to the position parameters of the preset imaging plane, the reflected aerial imaging light field data is projected onto the preset imaging plane. Through the reflection effect of the plane reflector, the virtual image corresponding to the aerial imaging data is reflected onto the preset imaging plane to form real image data.

6. The vehicle-mounted VR rear-seat entertainment method according to claim 5, characterized in that: The voice control instruction data is obtained as the first control instruction, including: The user's input voice is collected through the microphone array in the in-vehicle entertainment system to obtain original voice data; Preprocessing the original speech data includes speech noise reduction, speech endpoint detection and speech feature extraction to obtain preprocessed speech feature data; Using the pre-trained convolutional neural network model, the pre-processed speech feature data is recognized to obtain speech recognition result data; According to a preset voice control instruction set, the voice recognition result data is converted into voice control instruction data matching the voice control instruction set as the first control instruction.

7. The vehicle-mounted VR rear-seat entertainment method according to claim 6, characterized in that: Obtaining gesture control instruction data as a second control instruction includes: The original image data of the user's gesture is collected through the infrared sensor array in the in-vehicle entertainment system, and the infrared image data of the gesture area is extracted according to the position and depth information of the gesture area; Using the pre-trained convolutional neural network model, the infrared image data of the gesture area is subjected to feature extraction and classification to obtain gesture recognition result data; According to the preset gesture control instruction set, the gesture recognition result data is converted into gesture control instruction data matching the control instruction set as the second control instruction.

8. The vehicle-mounted VR rear-seat entertainment method according to claim 7, characterized in that: Generate interactive control data, including: Acquire three-dimensional coordinate data in the real image data, and construct a three-dimensional space interaction model based on the three-dimensional coordinate data through a three-dimensional reconstruction algorithm to represent the interaction relationship between the user and the virtual object; Converting the voice control instruction data in the first control instruction into motion parameters of the virtual object, and converting the gesture control instruction data in the second control instruction into posture parameters of the virtual object; updating the spatial position and interaction state of the virtual object in the three-dimensional space interaction model according to the motion parameters and posture parameters; According to the spatial position and interaction state of the user and the virtual object in the updated three-dimensional spatial interaction model, the spatial distance and relative posture between the user and the virtual object are calculated to determine whether the preset interaction conditions are met. If so, it is determined that there is an interaction behavior between the user and the virtual object, and a corresponding interaction event is triggered; wherein the preset interaction conditions include a spatial distance threshold and a relative posture threshold, and the interaction behavior includes collision, selection and grabbing with the virtual object; According to the type of the triggered interaction event, a corresponding interaction control instruction is selected from a preset interaction control instruction set to generate interaction control instruction data; The generated interactive control instruction data is integrated with the spatial position, posture and attribute information of the virtual object in the three-dimensional space interactive model to generate interactive control data for controlling the in-vehicle entertainment system.

Images