Immersive space virtual-real interaction method and system

Abstract

The invention discloses an immersive space virtual-real interaction method and system, and relates to the technical field of virtual reality interaction, and the method comprises the steps: collecting a full-view scene picture and illumination data through a camera, carrying out the preprocessing, simulating the real optical characteristics according to the scene picture and illumination data, and building a virtual scene; the virtual scene is evaluated and optimized; acquiring a user gesture image for analysis, identifying a user interaction command for virtual scene interaction, and displaying an interaction result in real time; and recording and storing the user interaction command and the interaction result. According to the method, the trueness and quality of the virtual scene are effectively improved, meanwhile, the user interaction command is recognized and analyzed by collecting the user gesture image and converting the user gesture image into the three-dimensional coordinate for virtual scene interaction, the gesture recognition accuracy is improved, and the experience effect of user virtual scene interaction is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality interaction technology, and in particular to an immersive space virtual-reality interaction method and system. Background Art

[0002] In recent years, with the rapid development of virtual reality (VR) and augmented reality (AR) technologies, immersive interaction has gradually become an important research direction in the field of human-computer interaction. Traditional virtual interaction methods are mainly based on image capture and scene presentation with limited viewing angles, which cannot fully simulate the optical properties and environmental details in the real world, resulting in user experience being limited to single-dimensional visual or tactile feedback. The combination of modern computer graphics and computer vision technology makes deep integration of virtual and real interactions possible. However, the existing technology is relatively rough in the collection and use of lighting data when reconstructing virtual scenes, and fails to fully reflect the complex light and shadow changes in the scene, resulting in a lack of realism in the virtual scene. Secondly, the user gesture recognition is poor in accuracy and robustness in dynamic interaction, making it difficult to achieve accurate and smooth user command operations. Summary of the invention

[0003] In view of the problems existing in the above-mentioned existing immersive space virtual-reality interaction methods and systems, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that the existing technology lacks full utilization of real environment lighting data and is difficult to accurately and smoothly identify user interaction instructions.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: an immersive space virtual-reality interaction method, which includes collecting full-view scene pictures and illumination data through a camera for preprocessing, and simulating real optical characteristics according to the scene pictures and illumination data to establish a virtual scene; the establishment of a virtual scene by simulating real optical characteristics according to the scene pictures and illumination data refers to mapping the light intensity in four polarization directions into Stokes vectors S, and calculating the degree of polarization DoP and the angle of polarization AoP according to the Stokes vector S. Define the azimuth angle as the polarization angle AoP , and calculate the zenith angle of the light by polarization ; Based on zenith angle Calculate the depth value with the full-view scene image : ; Where B is the camera baseline length, f is the camera focal length, and are the pixel displacements of the left and right views, respectively; Combined Depth Values Calculate the coordinates of the 3D point cloud: ; Where (u, v) is the pixel coordinate in the scene image, ( , ) is the optical center coordinate, (X, Y, Z) is the three-dimensional point cloud coordinate calculated based on the pixel points; The coordinates of the three-dimensional point clouds in the scene image are calculated sequentially to form three-dimensional point cloud data, and the three-dimensional point cloud data is densely completed based on the nearest neighbor interpolation; A geometric surface mesh model is constructed using 3D point cloud data, and the collected full-view scene image is mapped onto the surface of the geometric surface mesh model; The material reflectivity R is calculated using the Fresnel equation based on the lighting data: ; in and is the reflectivity and refractive index of the material, and are the incident angle and refraction angle of the light respectively; According to the zenith angle And material reflectivity adjust pixel intensity: ; in is the adjusted pixel intensity, is the basic pixel intensity, is the light regulation coefficient; Applying the adjusted pixel light intensity to the corresponding pixels on the geometric surface mesh model to render the geometric surface mesh model and generate a virtual scene; Evaluate and optimize virtual scenes; obtain user gesture images for analysis, identify user interaction commands for virtual scene interaction, and display interaction results in real time; record and store user interaction commands and interaction results.

[0006] As a preferred solution of the immersive space virtual-reality interaction method described in the present invention, the pre-processing of the full-view scene image and illumination data collected by the camera refers to using a circular multi-view camera array, defining the number and height of cameras according to the scene, calibrating the camera time, setting the camera intrinsic and extrinsic parameters to collect the full-view scene image, using a light sensor to collect illumination data, and using a Gaussian filter for noise reduction, and synchronously using a DoFP sensor to collect light intensity in four polarization directions of 0°, 45°, 90° and 135°.

[0007] As a preferred solution of the immersive space virtual-reality interaction method of the present invention, wherein: the evaluation and optimization of the virtual scene refers to randomly selecting scene pictures at different angles from the camera array, and obtaining virtual pictures at the same angle from the virtual scene, respectively calculating the average brightness and brightness variance of each scene picture and virtual picture, and taking the scene pictures and virtual pictures at the same angle as the control group; Calculate the structural similarity index SSIM of the scene images and virtual images in each control group; The average value of the structural similarity index SSIM of all control groups is calculated as the virtual scene evaluation index, and an evaluation threshold is set. If the virtual scene evaluation index is greater than or equal to the evaluation threshold, the virtual scene passes the evaluation and no optimization is required. If the virtual scene evaluation index is less than the evaluation threshold, full-view scene images and lighting data are collected again to establish a virtual scene for evaluation until the virtual scene evaluation index is greater than or equal to the evaluation threshold.

[0008] As a preferred solution of the immersive space virtual-reality interaction method of the present invention, the acquiring of user gesture images for analysis and identifying user interaction commands for virtual scene interaction includes acquiring user hand motion images through a smart wearable device, detecting and cropping user hand motion areas through YOLOv8, extracting multiple hand key point coordinates using MediaPipe Hands, and calculating the depth value of each hand key point coordinate respectively, and converting the hand key point coordinates into three-dimensional coordinates according to the depth value: ; in is the three-dimensional coordinate of the key points of the hand, is the depth value of the hand key point coordinates, are the coordinates of the hand key points, and is the image resolution, is the camera perspective, is the camera height; Construct a time window of length m frames, record the 3D coordinate trajectory of the key points of the hand, and use mean filtering and Kalman filtering to jointly optimize the trajectory, calculate the Euclidean distance between the 3D coordinates of the key points of the hand in each frame to construct the feature vector F; Using a head mounted display device to obtain user perspective data and convert it into Euler angles, construct a perspective rotation matrix based on the Euler angles, select multiple vertices in the virtual scene, and align the user perspective with the virtual scene based on the perspective rotation matrix; Extract the 3D coordinates of the palm center from the 3D coordinates of the key points of the hand in each frame of m frames, and calculate the Euclidean distance between the palm center coordinates and the 3D point cloud coordinates in the virtual scene. Calculate the average value of the Euclidean distances between all the 3D coordinates of the palm center in the m frames and the 3D point cloud coordinates in the virtual scene. Use the 3D point cloud coordinates of the virtual scene that minimizes the average value as the coordinates of the user interaction object. The motion trajectory of the three-dimensional coordinates of the hand key points in m frames is modeled through DMP, and the KNN classifier and action library are used to classify the motion trajectory to obtain the user's final interaction action, and virtual scene interaction is performed according to the user's interaction object and interaction action.

[0009] As a preferred solution of the immersive space virtual-reality interaction method described in the present invention, the real-time display of interaction results refers to updating the perspective rotation matrix by collecting the user's perspective data in real time, updating and aligning the virtual scene through the perspective rotation matrix, mapping the user's interaction actions into operation instructions of the interaction objects, and adjusting the interaction objects in the virtual scene according to the operation instructions, re-rendering the adjusted virtual scene for display, and reminding the user that the interaction is completed through vibration feedback and highlighted lines.

[0010] As a preferred solution of the immersive space virtual-reality interaction method described in the present invention, wherein: the recording and storage of user interaction commands and interaction results refers to recording the user's interaction commands and the virtual scenes after the interaction are completed in real time to form time series data and store them in a database. The database classifies the time series data according to each interaction command and marks it according to the timestamp, and regularly uploads the time series data to the cloud for backup.

[0011] Another object of the present invention is to provide an immersive space virtual-reality interaction system, which comprises: Data acquisition module, used to collect full-view scene images and lighting data for preprocessing; A virtual scene building module is used to simulate and generate real lighting data based on full-view scene images and lighting data to build virtual scenes and conduct evaluation and optimization; A gesture analysis module is used to obtain user gesture images and analyze and identify the user's interactive objects and interactive actions to generate virtual scene operation instructions for virtual scene interaction; The recording and storage module is used to record and store the user's interactive commands and virtual scene interaction results.

[0012] A computer device comprises: a memory and a processor; the memory stores a computer program, and the processor implements the steps of the above-mentioned immersive space virtual-reality interaction method when executing the computer program.

[0013] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned immersive space virtual-reality interaction method.

[0014] The beneficial effects of the present invention are as follows: the present invention collects full-view scene images and lighting data for processing, and simulates real lighting characteristics based on the lighting data to establish a virtual scene for evaluation and optimization, thereby effectively improving the realism and quality of the virtual scene. At the same time, by collecting user gesture images, converting them into three-dimensional coordinates, identifying and analyzing user interaction commands for virtual scene interaction, the accuracy of gesture recognition is improved, and the user's experience of virtual scene interaction is greatly enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0016] Figure 1 The figure is a flowchart of the virtual-reality interaction method in immersive space.

[0017] Figure 2 This is a structural diagram of the immersive space virtual-reality interaction method. DETAILED DESCRIPTION

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0021] Example 1 Reference Figure 1 , which is the first embodiment of the present invention, and which provides an immersive space virtual-reality interaction method, the immersive space virtual-reality interaction method comprising: S1, collecting full-view scene images and lighting data through the camera for preprocessing, and simulating real optical characteristics based on the scene images and lighting data to establish a virtual scene; Specifically, collecting full-view scene images and lighting data for preprocessing through cameras refers to using a circular multi-view camera array. By adopting a circular multi-view camera array, complete view information is provided, so that the geometric reconstruction of the virtual scene is more comprehensive and realistic, which is suitable for the panoramic presentation of immersive interactive scenes. The number and height of cameras are defined according to the scene. In this embodiment, the height of each camera is preferably set to 1.6 meters (the height of the human eye level) to ensure that the collection covers the main viewing angle range of the user. An industrial-grade camera that supports 4K resolution and 30 frames / second acquisition rate is selected to ensure high resolution and continuity of the collected images, and the camera time is calibrated. The camera's internal and external parameters are fixed to collect full-view scene images, a light sensor is used to collect illumination data, and a Gaussian filter is used for noise reduction processing. The DoFP sensor is used simultaneously to collect light intensity in four polarization directions of 0°, 45°, 90° and 135°. The DoFP sensor arranges polarization filters in different directions on each pixel, and can simultaneously obtain light intensity information in multiple polarization directions (for example, 0°, 45°, 90° and 135°). In the present invention, the DoFP sensor is used to collect polarization characteristic data of the scene, and these data are used to simulate real optical characteristics in the virtual scene, thereby enhancing the realism and visual effect of the virtual scene.

[0022] Before scene acquisition, the multi-view cameras are time-calibrated to ensure that the timing of the images captured by each camera is consistent. At the same time, the geometric characteristics of camera imaging are accurately described by setting internal parameters (focal length, principal point position, etc.) and external parameters (relative position and posture between cameras), ensuring the spatiotemporal consistency of multi-view data, providing an accurate geometric basis for subsequent multi-view image stitching and three-dimensional reconstruction, simulating the complex light and shadow changes in real scenes, and providing more realistic input data for dynamic lighting rendering of virtual scenes, thereby significantly improving the realism of virtual scenes.

[0023] Furthermore, simulating the real optical characteristics according to the scene image and illumination data to establish a virtual scene means mapping the light intensity in the four polarization directions into the Stokes vector S: ; in is the total intensity of light, is the linear polarization intensity of light in the horizontal and vertical directions, is the linear polarization intensity of light at 45°, is the circular polarization component, which is set to 0 in the DoFP sensor. , , as well as The light intensity in the four polarization directions of 0°, 45°, 90° and 135° respectively; The Stokes vector is a mathematical model that describes the polarization state of light. It fully expresses the intensity and polarization characteristics of light through four parameters. The introduction of the Stokes vector helps to accurately capture the optical characteristics of the scene and provides a basis for the subsequent calculation of the directionality and intensity distribution of light. Calculate the degree of polarization DoP and angle of polarization AoP according to the Stokes vector S: ; By calculating DoP and AoP, the directional information of light can be accurately extracted, providing reliable parameter support for the real light and shadow rendering of virtual scenes; Define the azimuth angle as the polarization angle AoP , and calculate the zenith angle of the light by polarization : ; Based on zenith angle Calculate the depth value with the full-view scene image : ; Where B is the camera baseline length. A camera is randomly selected from the camera array as the base camera to obtain the camera baseline length. f is the camera focal length. and are the pixel displacements of the left and right views, respectively; Combined Depth Values Calculate the coordinates of the 3D point cloud: ; Where (u, v) is the pixel coordinate in the scene image, ( , ) is the optical center coordinate, (X, Y, Z) is the three-dimensional point cloud coordinate calculated based on the pixel points; Based on the zenith angle and pixel displacement of the scene image, the depth value is calculated by the parallax method, and the 3D point cloud data is generated by combining the depth value. The point cloud is densified and completed to form a high-precision 3D scene, ensuring the integrity and detail expression of the virtual scene geometric model, and providing high-quality input for subsequent surface mesh construction and rendering; The coordinates of the three-dimensional point clouds in the scene image are calculated sequentially to form three-dimensional point cloud data, and the three-dimensional point cloud data is densely completed based on the nearest neighbor interpolation; The geometric surface mesh model is constructed using 3D point cloud data, and the collected full-view scene images are mapped to the surface of the geometric surface mesh model. Through the mapping of real scene textures, the appearance details of the virtual scene are closer to reality, enhancing the user's immersive experience; The material reflectivity R is calculated using the Fresnel equation based on the lighting data: ; in and are the reflection coefficient and refraction coefficient of the material, and are the incident angle and refraction angle of the light, respectively. The incident angle is calculated by the light source direction and the scene normal vector, and the refraction angle is calculated by Snell's law; According to the zenith angle And material reflectivity adjust pixel intensity: ; in is the adjusted pixel intensity, is the basic pixel light intensity, obtained through illumination data, The illumination adjustment coefficient is determined through experience or experiments. By adjusting the pixel light intensity, the dynamic matching of light and material is achieved, which improves the light and shadow effects and authenticity in the virtual scene. The adjusted pixel light intensity is applied to the corresponding pixels on the geometric surface mesh model to render the geometric surface mesh model and generate a virtual scene.

[0024] By innovatively introducing Stokes vector modeling, polarization information extraction and Fresnel equation calculation, complex optical properties are deeply integrated with virtual scene reconstruction to construct a virtual scene with high precision and high realism. In the process of realizing the densification of three-dimensional point cloud data and dynamic lighting adjustment, the detail expression and dynamic response ability of the virtual scene are further improved.

[0025] S2, evaluate and optimize the virtual scene; Specifically, evaluating and optimizing the virtual scene means randomly selecting scene images at different angles from the camera array, and obtaining virtual images at the same angle from the virtual scene, respectively calculating the average brightness and brightness variance of each scene image and virtual image, and using the scene images and virtual images at the same angle as the control group to ensure that the evaluation data has diversity and full coverage, reduce the quality evaluation error caused by the limitation of scene selection, and improve the reliability of the evaluation results; Calculate the structural similarity index SSIM of the scene images and virtual images in each control group: ; in and are the average brightness of image x and image y respectively, and is the brightness variance of image x and image y, is the covariance of image x and image y, and To stabilize constants, the similarity between real scene images and virtual scene images is measured to ensure the visual quality and realism of virtual scenes. By comparing brightness characteristics, the lighting inconsistency problem in virtual scenes can be quickly screened to provide intuitive data support for subsequent optimization. The average value of the structural similarity index SSIM of all control groups is calculated as the virtual scene evaluation index, and an evaluation threshold is set. If the virtual scene evaluation index is greater than or equal to the evaluation threshold, the virtual scene passes the evaluation and no optimization is required. If the virtual scene evaluation index is less than the evaluation threshold, full-view scene images and lighting data are collected again to establish a virtual scene for evaluation until the virtual scene evaluation index is greater than or equal to the evaluation threshold.

[0026] SSIM integrates brightness, contrast and structural information to more comprehensively reflect the quality of virtual scenes, providing an authoritative and reliable indicator for the evaluation of virtual scenes. The SSIM calculation average of all control groups is used to form a virtual scene evaluation index, which is compared with the set evaluation threshold to achieve quality standardization, which helps to realize the automated evaluation process.

[0027] S3, obtaining user gesture images for analysis, identifying user interaction commands for virtual scene interaction, and displaying the interaction results in real time; Specifically, obtaining user gesture images for analysis and identifying user interaction commands for virtual scene interaction includes obtaining user hand movement images through smart wearable devices, detecting and cropping user hand movement areas through YOLOv8, extracting multiple hand key point coordinates using MediaPipe Hands, and calculating the depth value of each hand key point coordinate respectively, and converting the hand key point coordinates into three-dimensional coordinates according to the depth value: ; in is the three-dimensional coordinate of the key points of the hand, is the depth value of the hand key point coordinates, are the coordinates of the hand key points, and is the image resolution, is the camera perspective, is the camera height; The generation of three-dimensional coordinates effectively solves the problem of missing depth information in two-dimensional gesture recognition and enhances the adaptability of gesture recognition in virtual scenes. High-precision three-dimensional coordinates provide stereoscopic data support for gesture trajectory analysis. MediaPipe Hands is an efficient framework specifically used for gesture detection and hand key point extraction. It can accurately capture the coordinate information of hand key points, including fingertips, joints, and palm centers. In the present invention, MediaPipeHands is responsible for extracting key point data in user hand movements, providing a basis for subsequent three-dimensional trajectory modeling and interactive operations, accurately obtaining the position of hand key points in three-dimensional space, and providing high-precision input data for virtual scene interaction. Construct a time window of length m frames, record the 3D coordinate trajectory of the key points of the hand, and use mean filtering and Kalman filtering to jointly optimize the trajectory, calculate the Euclidean distance between the 3D coordinates of the key points of the hand in each frame to construct the feature vector F; The time window captures the dynamic characteristics of hand movements and provides time dimension support for trajectory modeling; the trajectory optimization algorithm effectively smoothes the key point path, reduces data noise caused by hand shaking, and improves the continuity and reliability of the motion trajectory; Use a head-mounted display device to obtain user perspective data and convert it into Euler angles. According to the Euler angles, a perspective rotation matrix is ​​constructed to adjust the alignment between the virtual scene perspective and the user perspective to enhance the user's immersion. Multiple vertices in the virtual scene are selected, and the user perspective is aligned with the virtual scene based on the perspective rotation matrix. Extract the 3D coordinates of the palm center from the 3D coordinates of the key points of the hand in each frame of m frames, and calculate the Euclidean distance between the palm center coordinates and the 3D point cloud coordinates in the virtual scene. Calculate the average value of the Euclidean distances between all the 3D coordinates of the palm center in the m frames and the 3D point cloud coordinates in the virtual scene. Use the 3D point cloud coordinates of the virtual scene that minimizes the average value as the coordinates of the user interaction object to ensure the accuracy of the interaction target selection, avoid misoperation, and improve the stability of the interaction experience. The motion trajectory of the three-dimensional coordinates of the hand key points in m frames is modeled through DMP, and the KNN classifier and action library are used to classify the motion trajectory to obtain the user's final interaction action, and virtual scene interaction is performed according to the user's interaction object and interaction action.

[0028] DMP decomposes complex hand movements into easy-to-model motion primitives to achieve efficient modeling of gesture trajectories; the KNN classifier uses feature vectors to achieve high-precision motion recognition, enabling the virtual scene to accurately respond to user intentions. Combined with the user-selected interactive objects and classified actions, the virtual scene can provide real-time and accurate feedback on user operations, providing a highly immersive and interactive experience.

[0029] S4, recording and storing user interaction commands and interaction results; Specifically, real-time display of interaction results means updating the perspective rotation matrix by collecting the user's perspective data in real time, updating and aligning the virtual scene through the perspective rotation matrix, mapping the user's interaction actions into operation instructions of the interaction objects, and adjusting the interaction objects in the virtual scene according to the operation instructions, re-rendering the adjusted virtual scene for display, and reminding the user that the interaction is completed through vibration feedback and highlighted lines.

[0030] The user's perspective data is obtained in real time through the head-mounted display device, and the perspective rotation matrix is ​​updated based on this data to ensure that the display of the virtual scene is always synchronized with the user's perspective changes, enhancing the user's spatial immersion and interaction fluency. At the same time, it provides a basis for the accurate mapping of operation instructions in the virtual scene. Through precise perspective alignment and action mapping, the interaction results in the virtual scene can instantly reflect the user's operation intentions, achieving seamless connection from interaction input to virtual scene adjustment, and improving the intuitiveness of the operation. The dynamic rendering process displays the interaction effect in real time to ensure the user's instant feedback on the operation. At the same time, the rendering quality of the virtual scene can reflect the realism and accuracy of the operation. Vibration feedback provides users with tactile confirmation to enhance the perception of completed interaction; highlighted lines serve as visual cues to further clarify the operation object and interaction results. This multimodal feedback mechanism improves the user's cognitive accuracy and experience effect of the interaction completion status.

[0031] Furthermore, recording and storing user interaction commands and interaction results means recording the user's interaction commands and the virtual scene after the interaction is completed in real time to form time series data and store them in a database. The database classifies the time series data according to each interaction command and marks it according to the timestamp, and regularly uploads the time series data to the cloud for backup.

[0032] Example 2 Reference Figure 2 , which is a second embodiment of the present invention, which is different from the previous embodiment and provides an immersive space virtual-reality interaction system, which includes: Data acquisition module, used to collect full-view scene images and lighting data for preprocessing; A virtual scene building module is used to simulate and generate real lighting data based on full-view scene images and lighting data to build virtual scenes and conduct evaluation and optimization; A gesture analysis module is used to obtain user gesture images and analyze and identify the user's interactive objects and interactive actions to generate virtual scene operation instructions for virtual scene interaction; The recording and storage module is used to record and store the user's interactive commands and virtual scene interaction results.

[0033] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0034] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0035] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0036] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit with a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit with a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0037] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An immersive space virtual-reality interaction method, characterized by: include, The camera collects full-view scene images and lighting data for preprocessing, and simulates real optical properties based on the scene images and lighting data to create a virtual scene; The simulating real optical characteristics according to the scene image and illumination data to establish a virtual scene refers to mapping the light intensity in four polarization directions into Stokes vectors S; Calculate the degree of polarization DoP and angle of polarization AoP based on the Stokes vector S; Define the azimuth angle as the polarization angle AoP , and calculate the zenith angle of the light by polarization ; Based on zenith angle Calculate the depth value with the full-view scene image : Where B is the camera baseline length, f is the camera focal length, and are the pixel displacements of the left and right views, respectively; Combined Depth Values Calculate the coordinates of the 3D point cloud: Where (u, v) is the pixel coordinate in the scene image, ( , ) is the optical center coordinate, (X, Y, Z) is the three-dimensional point cloud coordinate calculated based on the pixel points; The coordinates of the three-dimensional point clouds in the scene image are calculated sequentially to form three-dimensional point cloud data, and the three-dimensional point cloud data is densely completed based on the nearest neighbor interpolation; A geometric surface mesh model is constructed using 3D point cloud data, and the collected full-view scene image is mapped onto the surface of the geometric surface mesh model; The material reflectivity R is calculated using the Fresnel equation based on the lighting data: in and are the reflection coefficient and refraction coefficient of the material, and are the incident angle and refraction angle of the light respectively; According to the zenith angle And material reflectivity adjust pixel intensity: in is the adjusted pixel intensity, is the basic pixel intensity, is the light regulation coefficient; Applying the adjusted pixel light intensity to the corresponding pixels on the geometric surface mesh model to render the geometric surface mesh model and generate a virtual scene; Evaluate and optimize virtual scenes; Acquire user gesture images for analysis, identify user interaction commands for virtual scene interaction, and display the interaction results in real time; Record and store user interaction commands and interaction results.

2. The immersive space virtual-reality interaction method according to claim 1, characterized in that: The preprocessing of the full-view scene image and illumination data collected by the camera refers to using a circular multi-view camera array, defining the number and height of cameras according to the scene, calibrating the camera time, setting the camera intrinsic and extrinsic parameters to collect the full-view scene image, using a light sensor to collect illumination data, and using a Gaussian filter for noise reduction, and synchronously using a DoFP sensor to collect light intensity in four polarization directions of 0°, 45°, 90° and 135°.

3. The immersive space virtual-reality interaction method according to claim 2, characterized in that: The evaluation and optimization of the virtual scene refers to randomly selecting scene images at different angles from the camera array, and obtaining virtual images at the same angle from the virtual scene, respectively calculating the average brightness and brightness variance of each scene image and virtual image, and using the scene images and virtual images at the same angle as a control group; Calculate the structural similarity index SSIM of the scene images and virtual images in each control group; The average value of the structural similarity index SSIM of all control groups is calculated as the virtual scene evaluation index, and an evaluation threshold is set. If the virtual scene evaluation index is greater than or equal to the evaluation threshold, the virtual scene passes the evaluation and no optimization is required. If the virtual scene evaluation index is less than the evaluation threshold, full-view scene images and lighting data are collected again to establish a virtual scene for evaluation until the virtual scene evaluation index is greater than or equal to the evaluation threshold.

4. The immersive space virtual-reality interaction method according to claim 3, characterized in that: The method of acquiring user gesture images for analysis and identifying user interaction commands for virtual scene interaction includes acquiring user hand motion images through a smart wearable device, detecting and cropping the user hand motion area through YOLOv8, extracting multiple hand key point coordinates using MediaPipe Hands, and calculating the depth value of each hand key point coordinate respectively, and converting the hand key point coordinates into three-dimensional coordinates according to the depth value: in is the three-dimensional coordinate of the key points of the hand, is the depth value of the hand key point coordinates, are the coordinates of the hand key points, and is the image resolution, is the camera perspective, is the camera height; Construct a time window of length m frames, record the 3D coordinate trajectory of the key points of the hand, and use mean filtering and Kalman filtering to jointly optimize the trajectory, calculate the Euclidean distance between the 3D coordinates of the key points of the hand in each frame to construct the feature vector F; Using a head mounted display device to obtain user perspective data and convert it into Euler angles, construct a perspective rotation matrix based on the Euler angles, select multiple vertices in the virtual scene, and align the user perspective with the virtual scene based on the perspective rotation matrix; Extract the 3D coordinates of the palm center from the 3D coordinates of the key points of the hand in each frame of m frames, and calculate the Euclidean distance between the palm center coordinates and the 3D point cloud coordinates in the virtual scene. Calculate the average value of the Euclidean distances between all the 3D coordinates of the palm center in the m frames and the 3D point cloud coordinates in the virtual scene. Use the 3D point cloud coordinates of the virtual scene that minimizes the average value as the coordinates of the user interaction object. The motion trajectory of the three-dimensional coordinates of the hand key points in m frames is modeled through DMP, and the KNN classifier and action library are used to classify the motion trajectory to obtain the user's final interaction action, and virtual scene interaction is performed according to the user's interaction object and interaction action.

5. The immersive space virtual-reality interaction method according to claim 4, characterized in that: The real-time display of interaction results refers to updating the perspective rotation matrix by collecting the user's perspective data in real time, updating and aligning the virtual scene through the perspective rotation matrix, mapping the user's interaction actions into operation instructions of the interaction objects, and adjusting the interaction objects in the virtual scene according to the operation instructions, re-rendering the adjusted virtual scene for display, and reminding the user that the interaction is completed through vibration feedback and highlighted lines.

6. The immersive space virtual-reality interaction method according to claim 5, characterized in that: The recording and storing of user interaction commands and interaction results refers to recording the user's interaction commands and the virtual scene after the interaction is completed in real time to form time series data and store it in a database. The database classifies the time series data according to each interaction command and marks it according to the timestamp, and regularly uploads the time series data to the cloud for backup.

7. An immersive space virtual-reality interaction system according to any one of claims 1 to 6, characterized in that: include, Data acquisition module, used to collect full-view scene images and lighting data for preprocessing; A virtual scene building module is used to simulate and generate real lighting data based on full-view scene images and lighting data to build virtual scenes and conduct evaluation and optimization; A gesture analysis module is used to obtain user gesture images and analyze and identify the user's interactive objects and interactive actions to generate virtual scene operation instructions for virtual scene interaction; The recording and storage module is used to record and store the user's interactive commands and virtual scene interaction results.

8. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the immersive space virtual-reality interaction method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the immersive space virtual-reality interaction method described in any one of claims 1 to 6 are implemented.

Images