An environment-adaptive metaverse scene perception system and method

Abstract

This invention discloses an environment-adaptive metaverse scene perception system and method, belonging to the field of data transmission technology. Specifically, it includes: real-time acquisition and preprocessing of multi-dimensional environmental and sensory data in a metaverse scene using a sensor array to construct a full-sensory environment dataset; precise acquisition of three-dimensional information of the physical environment using radar scanning and depth camera technology to establish a hyper-realistic three-dimensional model to simulate the complex ecosystems and environmental characteristics of the physical world; dynamic adjustment of metaverse scene elements using intelligent algorithms based on real-time full-sensory environment data, and introduction of user behavior prediction technology to intelligently adjust scene layout and interaction methods according to user history and current state to improve user experience; collection of user feedback through VR immersive interaction to evaluate the scene perception method, and iterative optimization based on the evaluation results to continuously improve the realism and interactivity of the metaverse scene; achieving efficient construction and intelligent management of the metaverse scene.

Description

Technical Field

[0001] This invention belongs to the field of data transmission technology, specifically an environment-adaptive metaverse scene perception system and method. Background Technology

[0002] With the development of virtual reality and augmented reality technologies, the concept of the metaverse is gradually becoming a reality. The metaverse refers to a shared virtual space supported by augmented reality, virtual reality, and internet technologies. It can be a digital environment simulating the real world or a completely fictional world. Scene perception is one of the key technologies for understanding a user's environment, involving multiple fields such as computer vision, deep learning, and sensor fusion. Its purpose is to enable devices or systems to recognize and understand their surrounding physical environment. Existing scene perception algorithms lack accuracy and robustness in complex environments, are slow to adjust to environmental changes, and have shortcomings in effectively perceiving the environment while protecting privacy.

[0003] For example, patent CN114816071A discloses a foot-sensing system and device for metaverse interaction, including: a stretchable sensor, a flexible array pressure sensor, a flexible array actuator, a controller, and a user terminal; the stretchable sensor is attached to the knee and the front of the ankle respectively to monitor the movement characteristics of the lower limbs; the flexible array pressure sensor is placed under the sole of the foot, in contact with the skin; the flexible array actuator is placed below the flexible array pressure sensor, on the inner surface of the shoe; the user terminal, through sensor information collection and analysis and sample training, establishes a virtual model of the object contacted by the sole of the foot, and establishes a mutual trust mechanism with the metaverse based on artificial intelligence algorithms; the controller, based on the magnitude of the force on the virtual model and combined with the feedback information from the sensors, controls the flexible array actuator to output a preset proportion of force to act on the foot. This technical solution simulates virtual stimuli from the metaverse through the flexible array actuator, enabling users to experience different scenarios when using shoes for metaverse interaction.

[0004] For example, patent CN116185206A discloses a method and system for synchronizing metaverse weather with real weather, including: constructing a coordinate map corresponding to a virtual city in the metaverse; mapping and binding the coordinate map of the virtual city in the metaverse with a designated real city map; synchronizing the real weather status information to the weather status information of the user's virtual city in the metaverse in real time; updating the scene model related to weather in the metaverse according to weather parameters to make corresponding visual changes; this technical solution obtains the real weather information of the user's current city in real time and synchronizes it with the weather of the virtual scene of the city in the metaverse, while updating the relevant scenes presented in the virtual scene of the city, which significantly improves the immersive experience of the city metaverse, and solves the problem that when users are roaming in the city in the existing virtual world, they cannot perceive the environmental changes of the current city. Such online worlds are too detached from reality, and users lack continuity and immersion when they first enter the metaverse from the real world.

[0005] The above-mentioned existing technologies all have the following problems: 1) The application scenarios and interaction methods are limited. For example, CN114816071A is mainly used to simulate the tactile experience of the feet in the metaverse, while ignoring the interactive experience of other senses in the metaverse; 2) The functions are singular. For example, CN116185206A only solves the problem of users perceiving real weather changes in the metaverse; 3) The system integration is low and the level of intelligence is limited. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes an environment-adaptive metaverse scene perception system and method. It collects and preprocesses multi-dimensional environmental and sensory data from the metaverse scene in real time using a sensor array to construct a full-sensory environment dataset. Utilizing radar scanning and depth camera technology, it accurately acquires three-dimensional information of the physical environment, establishing a hyper-realistic three-dimensional model to simulate the complex ecosystems and environmental characteristics of the physical world. Combining real-time full-sensory data, it employs intelligent algorithms to dynamically adjust metaverse scene elements and introduces user behavior prediction technology to intelligently adjust scene layout and interaction methods based on user history and current state, enhancing user experience. User feedback is collected through VR immersive interaction to evaluate the scene perception method, and iterative optimization is implemented based on the evaluation results to continuously improve the realism and interactivity of the metaverse scene. This achieves efficient construction and intelligent management of the metaverse scene.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An environment-adaptive metaverse scene perception method includes:

[0009] Step S1: In the metaverse scene, multi-dimensional environmental data and sensory data are collected in real time through a sensor array and preprocessed to obtain full sensory environmental data;

[0010] Step S2: Acquire three-dimensional information of the physical environment through radar scanning and depth cameras, construct a surreal three-dimensional model, and use the surreal three-dimensional model to simulate the environmental characteristics and complex ecosystems of the physical world;

[0011] Step S3: Based on the surreal 3D model and its simulation results, combined with real-time collected multi-sensory environmental data, intelligent algorithms are used to dynamically adjust the elements in the metaverse scene. At the same time, user behavior prediction methods are introduced to dynamically adjust the layout and interaction methods of the metaverse scene based on the user's historical behavior and current state.

[0012] Step S4: Collect user feedback on the metaverse scene using VR immersive interaction, evaluate the scene perception method based on user feedback and actual needs, and iterate and optimize it based on the evaluation results.

[0013] Specifically, step S2 includes the following steps:

[0014] S2.1: Use a multi-line LiDAR to scan the physical environment and acquire 3D point cloud data. Simultaneously, use a depth camera to capture color images and depth information of the environment, generating a depth image. Set the depth image I = {d1,…,d...} n}, where d n This represents the depth value of the nth pixel (u,v) in the depth image, where n represents the number of pixels in the depth image, u represents the x-coordinate of the pixel in the depth image, and v represents the y-coordinate of the pixel in the depth image.

[0015] S2.2: Use instant localization and mapping methods to perform data registration between 3D point cloud data and depth images;

[0016] S2.3: The registered 3D point cloud data is converted into a 3D mesh model using a point cloud reconstruction algorithm, and the depth image is applied to the surface of the 3D mesh model through adaptive texture mapping.

[0017] Specifically, step S2 further includes the following steps:

[0018] S2.4: Apply global illumination algorithm to simulate natural lighting and shadow effects, use physics engine to simulate the mechanical behavior between objects, and use acoustic simulation algorithm to generate the propagation and reflection effects of sound in three-dimensional space based on the sound source location, environmental material and geometry.

[0019] S2.5: Use meteorological simulation algorithms to simulate meteorological changes under different weather conditions, as well as the impact of day-night cycles and seasonal changes on the environment. At the same time, introduce AI-controlled organisms into the metaverse, use behavior tree algorithms to simulate the interaction behavior between organisms, and perform sentiment analysis on the interaction behavior between organisms.

[0020] S2.6: Based on the sentiment analysis results, combined with the simulation results in steps S2.4 and S2.5, dynamically adjust the lighting, shadows, and sound effects in the scene.

[0021] Specifically, the specific steps of S2.2 include:

[0022] S2.21: Extracting feature points P from 3D point cloud data L,n By calculating P C,n =T LC ×P L,n The 3D point cloud data is transformed into the camera coordinate system to obtain the initial camera pose P. C,n , among which, T LC This represents the transformation matrix from the lidar coordinate system to the camera coordinate system;

[0023] S2.22: Based on the numerical value d in the depth image n By back-projecting the pixel (u,v) into three-dimensional space, we obtain the corresponding three-dimensional point P. d,n =f(K,d n ), where K represents the camera's intrinsic parameter matrix, and f(·) represents the back projection function;

[0024] S2.23: Set the maximum number of iterations, based on the error e between the 3D point cloud and the depth image. n =|P d,n -P C,n The minimum error e is obtained through the iterative nearest point optimization algorithm. n,min This continues until the maximum number of iterations is reached.

[0025] S2.24: Outputs the registered 3D point cloud data, depth image, and optimized camera pose.

[0026] Specifically, step S3 includes the following steps:

[0027] S3.1: Acquire full-sensory environmental data X={x1,...,x m}, and load the surreal 3D model constructed based on radar scan and depth camera data. Simultaneously, acquire the simulation results from steps S2.4 and S2.5, where x m This represents the m-th full-sensory environmental data point, where m represents the number of full-sensory environmental data points.

[0028] S3.2: Analyze the simulation results and full-sensory environment data, and formulate adjustment strategies for virtual objects and scene attributes based on the analysis results. According to the adjustment strategies, dynamically adjust the position, size, color, and texture attributes of virtual objects, as well as the weather and time changes of the scene. At the same time, render the adjusted scene and objects in the virtual environment in real time.

[0029] S3.3: Record and analyze users' historical behavior data and current state information in the metaverse scene;

[0030] S3.4: Use behavior prediction strategies to predict user behavior, and dynamically adjust the element layout in the metaverse scene and optimize the interaction method based on the user behavior prediction results.

[0031] S3.5: Real-time feedback on adjustment effects and collection of user feedback data.

[0032] Specifically, the specific steps of the behavior prediction strategy in S3.4 include:

[0033] S3.41: Obtain the historical behavior data and current status information from step S3.3, integrate them into status data, and preprocess the status data;

[0034] S3.42: Construct a CNN-LSTM hybrid prediction model, introduce an incremental learning method, input the preprocessed state data into the CNN-LSTM hybrid prediction model for training, and set hyperparameters and training strategies.

[0035] S3.43: Use the trained CNN-LSTM hybrid prediction model to perform multimodal prediction of user behavior, combine the context information of the current metaverse scene to perform context-aware behavior prediction, and output the prediction results in multimodal form;

[0036] S3.44: Apply the prediction results to the metaverse scene in real time and make dynamic adjustments based on the difference between the user's actual behavior and the prediction results.

[0037] Specifically, the multidimensional environmental data in step S1 includes light intensity, sound level, user location, and movement; the sensory data includes temperature, humidity, odor, and touch.

[0038] An environment-adaptive metaverse scene perception system includes: a data processing module, a 3D modeling module, and an optimization module;

[0039] The data processing module is used to collect multi-dimensional environmental data and sensory data in real time, and to preprocess them to obtain full sensory environmental data.

[0040] The 3D modeling module uses radar scanning and depth cameras to capture 3D details of the physical environment, constructs a 3D model, and dynamically adjusts the elements of the metaverse scene through intelligent algorithms based on the 3D model and real-time multi-sensory environmental data, while predicting user behavior to optimize scene layout and interactive experience.

[0041] The optimization module uses VR immersive interaction to collect user feedback, evaluate the metaverse scene perception method, and iteratively optimize it based on feedback and needs.

[0042] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of an environment-adaptive metaverse scene perception method.

[0043] A computer-readable storage medium having computer instructions stored thereon, which, when executed, perform the steps of an environment-adaptive metaverse scene perception method.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. This invention proposes an environment-adaptive metaverse scene perception system and optimizes and improves its architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs and low production costs.

[0046] 2. This invention proposes an environment-adaptive metaverse scene perception method. By integrating advanced technologies such as sensor arrays, radar scanning, and depth cameras, it realizes real-time acquisition and processing of multi-dimensional environmental data and sensory data, constructs a highly realistic surreal 3D model, and effectively simulates the complex environmental characteristics and ecosystems of the physical world. This not only enhances the realism and immersion of the metaverse scene, but also provides users with a richer and more three-dimensional interactive experience.

[0047] 3. This invention proposes an environment-adaptive metaverse scene perception method, which introduces intelligent algorithms and user behavior prediction methods to achieve dynamic adjustment and layout optimization of metaverse scene elements, ensuring that the scene closely matches the user's actual needs and preferences; by collecting user feedback through VR immersive interaction and continuously iterating and optimizing accordingly, the accuracy of the scene perception method and the satisfaction of user experience are further improved, creating a more realistic, interactive, and personalized virtual world for users. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of an environment-adaptive metaverse scene perception method according to the present invention;

[0049] Figure 2This is a flowchart illustrating the principle of an environment-adaptive metaverse scene perception method according to the present invention.

[0050] Figure 3 This is a flowchart illustrating the surreal 3D model simulation of an environment-adaptive metaverse scene perception method according to the present invention.

[0051] Figure 4 This is a diagram of the architecture of an environment-adaptive metaverse scene perception system according to the present invention. Detailed Implementation

[0052] Example 1

[0053] Please see Figures 1-3 The present invention provides an embodiment of an environment-adaptive metaverse scene perception method, comprising the following steps:

[0054] Step S1: In the metaverse scene, multi-dimensional environmental data and sensory data are collected in real time through a sensor array and preprocessed to obtain full sensory environmental data;

[0055] The multidimensional environmental data in S1 includes light intensity, sound level, user location, and movement; the sensory data includes temperature, humidity, odor, and touch.

[0056] Step S2: Acquire three-dimensional information of the physical environment through radar scanning and depth cameras, construct a surreal three-dimensional model, and use the surreal three-dimensional model to simulate the environmental characteristics and complex ecosystems of the physical world;

[0057] Step S3: Based on the surreal 3D model and its simulation results, combined with real-time collected multi-sensory environmental data, intelligent algorithms are used to dynamically adjust the elements in the metaverse scene. At the same time, user behavior prediction methods are introduced to dynamically adjust the layout and interaction methods of the metaverse scene based on the user's historical behavior and current state.

[0058] Step S4: Collect user feedback on the metaverse scene using VR immersive interaction, evaluate the scene perception method based on user feedback and actual needs, and iterate and optimize it based on the evaluation results.

[0059] Furthermore, the specific steps of step S4 include:

[0060] S4.1: Select VR devices, such as VR headsets and controllers, and ensure that the devices have high resolution, low latency, and other characteristics to provide a high-quality immersive experience;

[0061] S4.2: Use 3D modeling software to construct a metaverse scene, including terrain, buildings, and characters, and import the constructed scene into the VR platform for initial debugging and optimization;

[0062] S4.3: Design the interaction methods between users and the metaverse scene, such as gesture recognition and voice control, to ensure that the interaction methods are natural, intuitive and in line with user habits;

[0063] S4.4: Conduct user testing, allowing users to freely explore the metaverse scene in the VR environment, experience various interactive functions, and encourage users to provide opinions and suggestions on the scene, interaction methods, etc.

[0064] S4.5: Collect user feedback through questionnaires and interviews, and evaluate scene perception methods using objective measurements such as eye tracking and EEG signals based on user feedback and actual needs, and record the evaluation results.

[0065] The specific steps of step S2 include:

[0066] S2.1: Use a multi-line LiDAR to scan the physical environment and acquire 3D point cloud data. Simultaneously, use a depth camera to capture color images and depth information of the environment, generating a depth image. Set the depth image I = {d1,...,d...} n}, where d n This represents the depth value of the nth pixel (u,v) in the depth image, where n represents the number of pixels in the depth image, u represents the x-coordinate of the pixel in the depth image, and v represents the y-coordinate of the pixel in the depth image.

[0067] S2.2: Use instant localization and mapping methods to perform data registration between 3D point cloud data and depth images;

[0068] S2.3: The registered 3D point cloud data is converted into a 3D mesh model using a point cloud reconstruction algorithm, and the depth image is applied to the surface of the 3D mesh model through adaptive texture mapping. In this invention, the point cloud reconstruction algorithm specifically selects the Poisson reconstruction method to convert the point cloud data into a 3D mesh model. The Poisson reconstruction method is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0069] Furthermore, the specific steps of texture mapping include:

[0070] (1) Depth image preprocessing: Preprocess the depth image, such as denoising and enhancing contrast;

[0071] (2) Texture mapping algorithm selection: In this invention, a projection-based texture mapping algorithm is selected;

[0072] (3) Texture mapping: The depth image is mapped onto the surface of the three-dimensional mesh model as a texture. By adjusting the texture coordinates and parameters, it is ensured that the texture can accurately fit each part of the mesh model. The specific process of texture mapping is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0073] S2.4: Apply global illumination algorithm to simulate natural lighting and shadow effects, use physics engine to simulate the mechanical behavior between objects, and use acoustic simulation algorithm to generate the propagation and reflection effects of sound in three-dimensional space based on the sound source location, environmental material and geometry.

[0074] Furthermore, the specific steps for simulating natural lighting and shadow effects using a global illumination algorithm include:

[0075] (1) Import the created 3D scene, including geometry, light source, and material, and set the position, color, and intensity attributes of the light source;

[0076] (2) Set up direct light source and indirect light source as needed to perform direct lighting and indirect lighting. Direct lighting is to calculate the lighting effect of the light source directly illuminating the surface of the object. Indirect lighting is to calculate the indirect lighting effect of reflection, refraction, scattering and other effects between objects through a global lighting algorithm. The calculation of the lighting effect of the light source directly illuminating the surface of the object and the global lighting algorithm are both existing technologies in this field and are not the inventive solutions of this application. They will not be described in detail here.

[0077] (3) The Shadow Mapping method is used to generate shadow effects between objects. Similarly, the specific process of generating shadows using the Shadow Mapping method is the existing technology in this field and is not the inventive solution of this application. It will not be described in detail here.

[0078] (4) Render the calculated lighting and shadow effects onto the image. Similarly, the process and steps of rendering the lighting and shadow effects are the existing technical content in this field and are not the inventive solution of this application. They will not be described in detail here.

[0079] Furthermore, the specific steps for simulating the mechanical behavior between objects using a physics engine include:

[0080] (1) Similarly, import the created physical model and set its properties;

[0081] (2) Apply Newton's laws of motion and conservation of momentum to calculate the interaction forces between objects, such as gravity, friction, and collision force. The solution process of Newton's laws of motion and the momentum conservation equation is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0082] (3) Collision detection algorithm is used to detect collisions between objects, and the position, velocity and acceleration of the objects are updated according to the collision detection results. The collision detection algorithm is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0083] (4) Render the motion state of the object onto the image to generate dynamic effects.

[0084] Furthermore, the specific steps of the acoustic simulation algorithm to generate the effects of sound propagation and reflection in three-dimensional space include:

[0085] (1) Define the location, sound pressure level, frequency and other attributes of the sound source;

[0086] (2) Construct a three-dimensional acoustic environment model, including walls, ground, and ceiling reflective surfaces;

[0087] The steps for constructing a 3D acoustic environment model include:

[0088] 1) Determine the modeling scope and set the target parameters. The modeling scope includes clearly defining the acoustic environment area to be simulated, such as indoor rooms and outdoor squares. The target parameters include reverberation time, sound pressure level, and frequency response.

[0089] 2) Use a sound level meter to measure the actual acoustic environment, record key acoustic parameters and sound signals, and at the same time use a 3D scanner to perform a 3D scan of the acoustic environment to obtain the geometry and size of the environment;

[0090] 3) Clean, denoise, and calibrate the collected data to ensure its accuracy and reliability;

[0091] 4) Based on the scan data, use SketchUp 3D modeling software to create a geometric model of the acoustic environment, including all objects that may affect sound propagation such as walls, ceilings, floors, and furniture. Assign acoustic material properties, such as sound absorption coefficient and reflection coefficient, to each object in the model. These properties will affect the propagation and reflection of sound in the environment.

[0092] 5) Use EASE acoustic simulation software to perform acoustic simulations, simulating the processes of sound propagation, reflection, and attenuation in three-dimensional space;

[0093] 6) Based on the modeling goals and requirements, set the acoustic parameters and boundary conditions for the simulation, including the location of the sound source, sound pressure level, frequency, etc., as well as the reverberation time and sound absorption coefficient of the environment;

[0094] 7) Run the simulation in the software, observe the propagation and reflection of sound in the three-dimensional sound environment, and adjust the model or simulation parameters as needed to obtain more accurate simulation results;

[0095] 8) Analyze the simulation results and evaluate whether the acoustic performance of the sound environment meets the requirements, including checking whether key acoustic parameters such as reverberation time and sound pressure level distribution meet the standards or design requirements.

[0096] 9) Based on the analysis results, optimize the acoustic environment model, including adjusting the position, shape or material properties of objects, and adding or reducing sound-absorbing materials.

[0097] (3) The wave equation is used to simulate the propagation of sound waves in three-dimensional space and to calculate the reflection and scattering effects of sound waves on the reflecting surface. The solution process of the wave equation is the prior art in this field and is not an inventive solution of this application. It will not be described in detail here.

[0098] (4) Use reverberation time parameters to simulate indoor sound field effects;

[0099] (5) Render the simulated sound effects onto the audio output device.

[0100] S2.5: Use meteorological simulation algorithms to simulate meteorological changes under different weather conditions, as well as the impact of day-night cycles and seasonal changes on the environment. At the same time, introduce AI-controlled organisms into the metaverse, use behavior tree algorithms to simulate the interaction behavior between organisms, and perform sentiment analysis on the interaction behavior between organisms.

[0101] Furthermore, the specific steps of the meteorological simulation algorithm include:

[0102] (1) Collect meteorological data through observation stations, satellites, and radars, and clean, denoise, calibrate, and format the collected meteorological data to meet simulation requirements;

[0103] (2) Based on meteorological principles and data characteristics, a numerical model is established to simulate changes in atmospheric state;

[0104] (3) Set the meteorological conditions at the start of the simulation, such as temperature, humidity, air pressure, and wind speed, and set the boundary conditions of the simulation area, such as the influence of topography, ocean, and land.

[0105] (4) Simulate the weather system using computer models, predict future weather conditions, and introduce time variables into the model to simulate the impact of day-night cycle and seasonal changes on meteorological changes;

[0106] (5) Present the simulation results in a visual manner, such as weather maps and meteorological cloud maps;

[0107] (6) Compare the actual observation data with the simulation results to verify the accuracy and reliability of the simulation, analyze the sources of error, and improve the simulation accuracy.

[0108] The implementation steps of AI-controlled biological and behavior tree algorithms include:

[0109] (1) Create a three-dimensional model of a creature in the metaverse and endow it with physical properties and behavioral characteristics;

[0110] (2) Integrate AI algorithms into biological models so that organisms can autonomously perceive the environment, make decisions and perform actions;

[0111] (3) Based on the behavioral characteristics of organisms, design a behavior tree structure, including root node, control flow node and execution node, and write implementation code for each node, defining its behavioral logic and return value;

[0112] (4) Run the behavior tree in the metaverse environment and select and execute the corresponding behavior according to the current environment and biological state;

[0113] (5) Set interaction rules between organisms, such as foraging, escaping, and competing, use behavior tree algorithm to simulate the interaction behavior between organisms, and select and execute corresponding actions according to the interaction rules. The calculation process of behavior tree algorithm is the prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0114] (6) Conduct emotional analysis on the interaction behavior between organisms to identify the main emotional tendencies, such as pleasure, sadness, and tension.

[0115] S2.6: Based on the sentiment analysis results, combined with the simulation results in steps S2.4 and S2.5, dynamically adjust the lighting, shadows, and sound effects in the scene.

[0116] For example, warm lighting and soft background music can be added to pleasant interactions; dim lighting and tense background sound effects can be added to tense situations.

[0117] The specific steps in S2.2 include:

[0118] S2.21: Extracting feature points P from 3D point cloud data L,n By calculating P C,n =T LC ×P L,n The 3D point cloud data is transformed into the camera coordinate system to obtain the initial camera pose P. C,n , among which, T LC This represents the transformation matrix from the lidar coordinate system to the camera coordinate system;

[0119] S2.22: Based on the numerical value d in the depth image n By back-projecting the pixel (u,v) into three-dimensional space, we obtain the corresponding three-dimensional point P. d,n =f(K,d n), where K represents the camera's intrinsic parameter matrix, and f(·) represents the back projection function;

[0120] S2.23: Set the maximum number of iterations, based on the error e between the 3D point cloud and the depth image. n =|P d,n -P C,n The minimum error e is obtained through the iterative nearest point optimization algorithm. n,min The iteration continues until the maximum number of iterations is reached. The goal of the iterative closest point optimization algorithm is to optimize the error function. The calculation formula of the iterative closest point optimization algorithm is existing technology in this field and is not an inventive solution of this application. It will not be elaborated here.

[0121] S2.24: Outputs the registered 3D point cloud data, depth image, and optimized camera pose.

[0122] The specific steps of step S3 include:

[0123] S3.1: Acquire full-sensory environmental data X={x1,...,x m}, and load the surreal 3D model constructed based on radar scan and depth camera data. Simultaneously, acquire the simulation results from steps S2.4 and S2.5, where x m This represents the m-th full-sensory environmental data point, where m represents the number of full-sensory environmental data points.

[0124] S3.2: Analyze the simulation results and full-sensory environment data, and formulate adjustment strategies for virtual objects and scene attributes based on the analysis results. According to the adjustment strategies, dynamically adjust the position, size, color, and texture attributes of virtual objects, as well as the weather and time changes of the scene. At the same time, render the adjusted scene and objects in the virtual environment in real time.

[0125] Furthermore, this invention employs a long short-term memory network to analyze the simulation results and full-sensory environmental data. Specific steps include:

[0126] S3.21: Convert the simulation results and full-sensory environment data into a format that can be processed by the recurrent neural network long short-term memory network model, and perform normalization and data partitioning;

[0127] S3.22: Load the pre-built Long Short-Term Memory (LSTM) network model, train the LSM network model using the partitioned training set, calculate the output through forward propagation, update the model parameters through backpropagation and Adam optimization algorithms, and evaluate the model performance using test set data. The construction process of the LSM network model, the forward propagation calculation formula, the backpropagation calculation formula, and the Adam optimization algorithm are all existing technologies in this field and are not inventive solutions of this application, and will not be described in detail here.

[0128] S3.23: Develop adjustment strategies for virtual object and scene attributes based on the analysis results of the Long Short-Term Memory Network Model;

[0129] S3.24: Dynamically adjust the position, size, color, texture attributes of virtual objects, as well as weather and time changes in the scene, according to the adjustment strategy;

[0130] S3.25: Render adjusted scenes and objects in real time within a virtual environment.

[0131] It should be noted that the main purpose of intelligent analysis of simulation results and full-sensory environment data in this invention is as follows: 1) By analyzing user behavior patterns and environmental data through intelligent algorithms, we can more accurately understand user needs and preferences, thereby dynamically adjusting the attributes of virtual environments and objects; 2) Real-time analysis of full-sensory environment data helps to simulate more realistic physical environments, making virtual scenes more lifelike. By analyzing this data and adjusting the attributes of virtual objects, including but not limited to color, texture, and shadow, we can further enhance the realism of the scene; 3) We can analyze the resource usage in the virtual environment and formulate optimization strategies based on the analysis results, such as dynamically adjusting the complexity and rendering quality of virtual objects to balance performance and visual effects and optimize resource allocation. After intelligent analysis of the simulation results and full-sensory environment data, we can obtain: 1) common user behavior patterns, such as browsing habits and interaction methods, providing a basis for formulating personalized adjustment strategies; 2) in-depth insights into the virtual environment state, such as light intensity, temperature distribution, and sound characteristics, through the analysis of full-sensory environment data; 3) specific adjustment strategies can be formulated, including adjustments to the position, size, color, and texture attributes of virtual objects, as well as the simulation of weather and time changes in the scene.

[0132] S3.3: Record and analyze users' historical behavior data and current state information in the metaverse scene;

[0133] S3.4: Use behavior prediction strategies to predict user behavior, and dynamically adjust the element layout in the metaverse scene and optimize the interaction method based on the user behavior prediction results.

[0134] S3.5: Real-time feedback on adjustment effects and collection of user feedback data.

[0135] The specific steps of the behavior prediction strategy in S3.4 include:

[0136] S3.41: Obtain the historical behavior data and current status information from step S3.3, integrate them into status data, and preprocess the status data. The integration method is to directly integrate them by splicing the beginning and end of the data.

[0137] S3.42: Construct a CNN-LSTM hybrid prediction model, introduce an incremental learning method, input the preprocessed state data into the CNN-LSTM hybrid prediction model for training, and set hyperparameters and training strategies.

[0138] Among them, the CNN-LSTM hybrid prediction model refers to a hybrid model composed of convolutional neural networks and long short-term memory networks. The convolutional neural network part is used to extract spatial features from state data, such as local patterns of user movement trajectories, while the long short-term memory network part is used to capture long-term dependencies in time series. At the same time, an attention mechanism is introduced to enhance the model's attention to important features.

[0139] The hyperparameters can be set, including learning rate, batch size, and number of iterations; the training strategies include learning rate decay and early stopping. The specific settings and operations of learning rate decay and early stopping are existing technologies in this field and are not inventive solutions of this application, so they will not be described in detail here. The purpose of setting incremental learning is to enable the model to continuously learn new data without forgetting too much old knowledge. This is mainly achieved by using regularization terms. The calculation process of regularization terms is existing technology in this field and is not inventive solutions of this application, so it will not be described in detail here.

[0140] Furthermore, the specific process of constructing a CNN-LSTM hybrid prediction model includes:

[0141] (1) Collect state data and perform preprocessing, including handling missing values, outliers, and standardization.

[0142] (2) Divide the state data into training set, validation set and test set, with the division ratio set to 70%:15%:15%;

[0143] (3) Convert the state data format into a three-dimensional tensor format of (seq_len, batch, input_size), where seq_len represents the sequence length, batch represents the batch size, and input_size represents the input size;

[0144] (4) Use convolutional neural networks (CNN) to extract spatial features of input state data, and design convolutional layers, activation layers, and pooling layers to construct CNN modules. The number of convolutional layers, kernel size, stride, and padding parameters need to be set according to the specific task and data characteristics.

[0145] (5) Use a Long Short-Term Memory (LSTM) network to capture the temporal dependencies of the state data, and design the LSTM network layers. Set the parameters of hidden layer size, number of layers, and whether to use bidirectional LSTM. Note that the input of the LSTM layer should be the feature sequence output by the CNN module.

[0146] (6) The output of the CNN module is transformed in dimension and then used as the input of the LSTM module to achieve feature fusion;

[0147] (7) Add a fully connected layer after the LSTM layer to map the output of the LSTM to the target prediction value and obtain the CNN-LSTM hybrid prediction model.

[0148] (8) Set hyperparameters, including learning rate, batch size, training epochs, and optimizer. The setting method of hyperparameters should be adjusted according to the experiment and the performance of the CNN-LSTM hybrid prediction model.

[0149] (9) The CNN-LSTM hybrid prediction model is trained using the training set, and the parameters of the CNN-LSTM hybrid prediction model are optimized by the backpropagation algorithm. The backpropagation algorithm is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0150] S3.43: Use the trained CNN-LSTM hybrid prediction model to perform multimodal prediction of user behavior, combine the context information of the current metaverse scene to perform context-aware behavior prediction, and output the prediction results in multimodal form;

[0151] S3.44: Apply the prediction results to the metaverse scene in real time and make dynamic adjustments based on the difference between the user's actual behavior and the prediction results.

[0152] Example 2

[0153] Please see Figure 4 Another embodiment of the present invention provides: an environment-adaptive metaverse scene perception system, comprising:

[0154] Data processing module, 3D modeling module, optimization module;

[0155] The data processing module is used to collect multi-dimensional environmental data and sensory data in real time, and to preprocess them to obtain full sensory environmental data.

[0156] The 3D modeling module uses radar scanning and depth cameras to capture 3D details of the physical environment and build a high-precision 3D model. Based on the 3D model, combined with real-time multi-sensory environmental data, it dynamically adjusts the elements of the metaverse scene through intelligent algorithms, while predicting user behavior to optimize scene layout and interactive experience.

[0157] The optimization module uses VR immersive interaction to collect user feedback, evaluates the metaverse scene perception method, and iteratively optimizes it based on feedback and needs.

[0158] The data processing module includes: a sensor array unit and a data preprocessing unit;

[0159] The sensor array unit includes a temperature sensor, a humidity sensor, a light sensor, and a sound sensor, which are used to collect physical quantities in the environment in real time, such as temperature, humidity, light intensity, and sound.

[0160] The data preprocessing unit is used to clean, denoise, filter, and standardize the raw data collected by the sensors to improve data quality and accuracy.

[0161] The 3D modeling module includes: a 3D data acquisition unit, a 3D modeling unit, an environment simulation unit, an intelligent algorithm unit, a user behavior prediction unit, and a dynamic adjustment unit;

[0162] The 3D data acquisition unit uses radar scanning and depth camera equipment to acquire 3D point cloud data and depth images of the physical environment;

[0163] The 3D modeling unit, based on the collected 3D point cloud data, uses point cloud reconstruction algorithm technology to construct an accurate 3D mesh model;

[0164] The environment simulation unit uses physics engines, lighting algorithms and other technologies to simulate the environmental characteristics of the physical world, such as lighting, shadows, physical collisions, and complex ecosystems, such as weather changes, day and night cycles, and seasonal changes.

[0165] The intelligent algorithm unit is used to analyze and process the data collected in real time, and dynamically adjust the elements in the scene, such as the position, size, color, texture and scene layout of virtual objects, based on the analysis results.

[0166] The user behavior prediction unit uses behavior prediction strategies to predict the user's future behavior based on the user's historical behavior and current state information, and adjusts the interaction method of the scene according to the prediction results.

[0167] The dynamic adjustment unit performs adjustments to scene elements and layouts based on the results of intelligent algorithms and user behavior predictions.

[0168] The optimization module includes: a VR interaction unit and an evaluation and optimization unit;

[0169] The VR interaction unit is used to provide an immersive VR interaction method, allowing users to experience the metaverse scene as if they were there, and to collect user feedback.

[0170] The evaluation and optimization unit is used to analyze and evaluate the collected user feedback, and to iteratively optimize the scene perception method based on the evaluation results and actual needs, so as to improve the user experience and the realism of the scene.

[0171] Example 3

[0172] An electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of an environment-adaptive meta-universe scene perception method. For details, please refer to the above method embodiments, which will not be repeated here.

[0173] A computer-readable storage medium storing computer instructions that, when executed, perform steps of an environment-adaptive metaverse scene perception method, wherein the storage medium may be a volatile or non-volatile computer-readable storage medium.

[0174] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

[0175] If the technical solution disclosed herein involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution disclosed herein involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An environment-adaptive metaverse scene perception method, characterized in that, include: Step S1: In the metaverse scene, multi-dimensional environmental data and sensory data are collected in real time through a sensor array and preprocessed to obtain full sensory environmental data; Step S2: Acquire three-dimensional information of the physical environment through radar scanning and depth cameras, construct a surreal three-dimensional model, and use the surreal three-dimensional model to simulate the environmental characteristics and complex ecosystems of the physical world; Step S3: Based on the surreal 3D model and its simulation results, combined with real-time collected multi-sensory environmental data, intelligent algorithms are used to dynamically adjust the elements in the metaverse scene. At the same time, user behavior prediction methods are introduced to dynamically adjust the layout and interaction methods of the metaverse scene based on the user's historical behavior and current state. Step S4: Collect user feedback on the metaverse scene using VR immersive interaction, evaluate the scene perception method based on user feedback and actual needs, and iterate and optimize it based on the evaluation results; The specific steps of step S2 include: S2.1: Use a multi-line LiDAR to scan the physical environment and acquire 3D point cloud data. Simultaneously, use a depth camera to capture color images and depth information of the environment, generating a depth image, and setting the depth image as... ,in, Represents the nth pixel in the depth image The depth value, where n represents the number of pixels in the depth image, u represents the x-coordinate of a pixel in the depth image, and v represents the y-coordinate of a pixel in the depth image; S2.2: Use instant localization and mapping methods to perform data registration between 3D point cloud data and depth images; S2.3: The registered 3D point cloud data is converted into a 3D mesh model using a point cloud reconstruction algorithm, and the depth image is applied to the surface of the 3D mesh model through adaptive texture mapping. The specific steps of step S2 also include: S2.4: Apply global illumination algorithm to simulate natural lighting and shadow effects, use physics engine to simulate the mechanical behavior between objects, and use acoustic simulation algorithm to generate the propagation and reflection effects of sound in three-dimensional space based on the sound source location, environmental material and geometry. S2.5: Use meteorological simulation algorithms to simulate meteorological changes under different weather conditions, as well as the impact of day-night cycles and seasonal changes on the environment. At the same time, introduce AI-controlled organisms into the metaverse, use behavior tree algorithms to simulate the interaction behavior between organisms, and perform sentiment analysis on the interaction behavior between organisms. S2.6: Based on the sentiment analysis results, combined with the simulation results in steps S2.4 and S2.5, dynamically adjust the lighting, shadows, and sound effects in the metaverse scene.

2. The environment-adaptive metaverse scene perception method as described in claim 1, characterized in that, The specific steps of S2.2 include: S2.21: Extracting feature points from 3D point cloud data Through calculation The 3D point cloud data is transformed into the camera coordinate system to obtain the initial camera pose. ,in, This represents the transformation matrix from the lidar coordinate system to the camera coordinate system; S2.22: Based on the numerical values ​​passed through the depth image , to pixels Back-projecting into three-dimensional space yields the corresponding three-dimensional points. Where K represents the camera's intrinsic parameter matrix, Represents the back projection function; S2.23: Set the maximum number of iterations, based on the error between the 3D point cloud and the depth image. The minimum error is obtained through the iterative nearest point optimization algorithm. This continues until the maximum number of iterations is reached. S2.24: Outputs the registered 3D point cloud data, depth image, and optimized camera pose.

3. The environment-adaptive metaverse scene perception method as described in claim 2, characterized in that, The specific steps of step S3 include: S3.1: Acquire full-sensory environmental data The simulation results from steps S2.4 and S2.5 are then loaded, and a surreal 3D model constructed based on radar scan and depth camera data is loaded. This represents the m-th full-sensory environmental data point, where m represents the number of full-sensory environmental data points. S3.2: Analyze the simulation results and full-sensory environment data, and formulate adjustment strategies for virtual objects and scene attributes based on the analysis results. According to the adjustment strategies, dynamically adjust the position, size, color, and texture attributes of virtual objects, as well as the weather and time changes of the scene. At the same time, render the adjusted scene and objects in the virtual environment in real time. S3.3: Record and analyze users' historical behavior data and current state information in the metaverse scene; S3.4: Use behavior prediction strategies to predict user behavior, and dynamically adjust the element layout in the metaverse scene and optimize the interaction method based on the user behavior prediction results. S3.5: Real-time feedback on adjustment effects and collection of user feedback data.

4. The environment-adaptive metaverse scene perception method as described in claim 3, characterized in that, The specific steps of the behavior prediction strategy in S3.4 include: S3.41: Obtain the historical behavior data and current status information from step S3.3, integrate them into status data, and preprocess the status data; S3.42: Construct a CNN-LSTM hybrid prediction model, introduce an incremental learning method, input the preprocessed state data into the CNN-LSTM hybrid prediction model for training, and set hyperparameters and training strategies. S3.43: Use the trained CNN-LSTM hybrid prediction model to perform multimodal prediction of user behavior, combine the context information of the current metaverse scene to perform context-aware behavior prediction, and output the prediction results in multimodal form; S3.44: Apply the prediction results to the metaverse scene in real time and make dynamic adjustments based on the difference between the user's actual behavior and the prediction results.

5. The environment-adaptive metaverse scene perception method as described in claim 4, characterized in that, The multidimensional environmental data in step S1 includes light intensity, sound level, user location, and movement; the sensory data includes temperature, humidity, odor, and touch.

6. An environment-adaptive metaverse scene perception system, used to implement the environment-adaptive metaverse scene perception method according to any one of claims 1-5, characterized in that, include: Data processing module, 3D modeling module, optimization module; The data processing module is used to collect multi-dimensional environmental data and sensory data in real time, and to preprocess them to obtain full sensory environmental data. The 3D modeling module is used to capture 3D details of the physical environment using radar scanning and depth cameras, construct a high-precision 3D model, and dynamically adjust the elements of the metaverse scene based on the 3D model and real-time multi-sensory data through intelligent algorithms, while predicting user behavior to optimize scene layout and interactive experience. The optimization module is used to collect user feedback through VR immersive interaction, evaluate the metaverse scene perception method, and iteratively optimize it based on feedback and needs.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the environmental adaptive metaverse scene perception method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores computer instructions, which, when executed, perform the steps of the environment adaptive metaverse scene perception method according to any one of claims 1-5.

Images