Environment self-adaptive meta-universe scene sensing system and method

Abstract

The invention discloses an environment adaptive meta-universe scene perception system and method, and belongs to the technical field of data transmission, and the method specifically comprises the steps: collecting and preprocessing multi-dimensional environment and sensory data in a meta-universe scene in real time through a sensor array, and constructing a full sensory environment data set; radar scanning and depth camera technologies are utilized to accurately obtain three-dimensional information of a physical environment, and a super-reality three-dimensional model is established to simulate a complex ecological system and environment characteristics of the physical world; in combination with real-time full-sensory environment data, an intelligent algorithm is adopted to dynamically adjust meta-universe scene elements, a user behavior prediction technology is introduced, scene layout and an interaction mode are intelligently adjusted according to user history and current states, and user experience is improved; user feedback is collected through VR immersive interaction, a scene perception method is evaluated, and iterative optimization is implemented according to an evaluation result so as to continuously improve the sense of reality and interactivity of the meta-universe scene; and efficient construction and intelligent management of the meta universe scene are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data transmission, and specifically relates to an environment-adaptive metaverse scene perception system and method. Background Art

[0002] With the development of virtual reality and augmented reality technologies, the concept of the metaverse has gradually become possible. The metaverse refers to a shared virtual space supported by augmented reality, virtual reality and Internet technologies. It can be a digital environment that simulates the real world or a completely fictional world. Scene perception is one of the key technologies for understanding the user's environment. It involves multiple fields such as computer vision, deep learning, and sensor fusion. The purpose is to enable devices or systems to recognize and understand the physical environment around them. Existing scene perception algorithms lack accuracy and robustness in complex environments. When the environment changes, the system adjusts slowly and has defects in effective environmental perception while protecting privacy.

[0003] For example, the patent with publication number CN114816071A discloses a foot sensing system and equipment for Metaverse interaction, including: stretchable sensors, flexible array pressure sensors, flexible array actuators, controllers and user terminals; stretchable sensors are attached to the knees and the front of the ankles to monitor the movement characteristics of the lower limbs; flexible array pressure sensors are placed under the soles of the feet and fit the sole skin; flexible array actuators are placed under the flexible array pressure sensors and on the inner surface of the shoes; the user terminal establishes a virtual model of the soles contacting objects through sensor information collection and analysis and sample training, and establishes a mutual trust mechanism with the Metaverse based on artificial intelligence algorithms; the controller controls the flexible array actuator to output a preset proportion of force on the foot according to the magnitude of the force on the virtual model and the feedback information from the sensor. This technical solution simulates the virtual stimulation of the Metaverse through the flexible array actuator, so that the shoes used for Metaverse interaction can allow users to experience different scenes.

[0004] For example, the patent with publication number CN116185206A discloses a method and system for synchronizing metaverse weather with real weather, including: constructing a coordinate map corresponding to a metaverse virtual city; mapping and binding the coordinate map of the metaverse virtual city with a designated real city map; synchronizing the real weather status information in real time to the weather status information of the user's metaverse virtual city; updating the scene model associated with the weather in the metaverse according to weather parameters to make corresponding visual changes; this technical solution obtains the real weather information of the user's current city in real time and synchronizes it with the weather of the metaverse city virtual scene, while updating the relevant scenes presented in the city virtual scene, significantly improving the immersive experience effect of the city metaverse, and solving the problem that when users in the existing virtual world are roaming in the city, they will not be able to perceive the environmental changes in the current city. Such an online world is too detached from reality, and users lack continuity and sense of immersion when they first enter the metaverse from the real world.

[0005] The above existing technologies all have the following problems: 1) There are limitations in application scenarios and interaction methods. For example, CN114816071A is mainly used to simulate the tactile experience of the feet in the metaverse, but ignores the interactive experience of other senses in the metaverse; 2) Single function. For example, CN116185206A only solves the problem of users perceiving real weather changes in the metaverse; 3) The system integration is low and the degree of intelligence is limited. Summary of the invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes an environment-adaptive metaverse scene perception system and method, which uses a sensor array to collect and preprocess the multi-dimensional environment and sensory data in the metaverse scene in real time to construct a full sensory environment data set; uses radar scanning and depth camera technology to accurately obtain three-dimensional information of the physical environment, and establishes a hyper-realistic three-dimensional model to simulate the complex ecosystem and environmental characteristics of the physical world; combines real-time full sensory data, uses intelligent algorithms to dynamically adjust metaverse scene elements, and introduces user behavior prediction technology to intelligently adjust scene layout and interaction methods according to user history and current status to improve user experience; collects user feedback through VR immersive interaction, evaluates scene perception methods, and implements iterative optimization based on the evaluation results to continuously improve the realism and interactivity of the metaverse scene; realizes efficient construction and intelligent management of metaverse scenes.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] An environment-adaptive metaverse scene perception method, comprising:

[0009] Step S1: In the metaverse scene, multi-dimensional environmental data and sensory data are collected in real time through a sensor array, and pre-processed to obtain full sensory environmental data;

[0010] Step S2: Acquire three-dimensional information of the physical environment through radar scanning and depth camera, build a hyper-realistic three-dimensional model, and use the hyper-realistic three-dimensional model to simulate environmental characteristics and complex ecosystems of the physical world;

[0011] Step S3: Based on the hyper-realistic 3D model and its simulation results, combined with the real-time collected full sensory environment data, an intelligent algorithm is used to dynamically adjust the elements in the metaverse scene. At the same time, a user behavior prediction method is introduced to dynamically adjust the metaverse scene layout and interaction mode according to the user's historical behavior and current status;

[0012] Step S4: Use VR immersive interaction to collect user feedback on the metaverse scene, evaluate the scene perception method based on user feedback and actual needs, and perform iterative optimization based on the evaluation results.

[0013] Specifically, the specific steps of step S2 include:

[0014] S2.1: Use a multi-line laser radar to scan the physical environment and obtain the three-dimensional point cloud data of the environment. At the same time, use a depth camera to capture the color image and depth information of the environment, generate a depth image, and set the depth image I = {d1,…,d n}, where d n Represents the depth value of the nth pixel (u, v) in the depth image, where n represents the number of pixels in the depth image, u represents the horizontal coordinate of the pixel in the depth image, and v represents the vertical coordinate of the pixel in the depth image;

[0015] S2.2: Use the simultaneous localization and mapping method to align the 3D point cloud data and the depth image;

[0016] S2.3: Use a point cloud reconstruction algorithm to convert the registered 3D point cloud data into a 3D mesh model, and fit the depth image to the surface of the 3D mesh model through adaptive texture mapping.

[0017] Specifically, the specific steps of step S2 also include:

[0018] S2.4: Apply global illumination algorithms to simulate natural lighting and shadow effects, use physics engines to simulate the mechanical behavior between objects, and use acoustic simulation algorithms to generate sound propagation and reflection effects in three-dimensional space based on the location of the sound source, environmental materials, and geometric shapes;

[0019] S2.5: Use meteorological simulation algorithms to simulate meteorological changes under different weather conditions, as well as the impact of day and night cycles and seasonal changes on the environment. At the same time, introduce AI-controlled creatures into the metaverse, use behavior tree algorithms to simulate the interactions between creatures, and perform sentiment analysis on the interactions between creatures;

[0020] S2.6: According to the emotion analysis results, combined with the simulation results in step S2.4 and step S2.5, dynamically adjust the lighting, shadow, and sound effects in the scene.

[0021] Specifically, the specific steps of S2.2 include:

[0022] S2.21: Extract feature points P from 3D point cloud data L,n , by calculating P C,n =T LC ×P L,n , transform the 3D point cloud data into the camera coordinate system to obtain the initial camera pose P C,n , where T LC Represents the transformation matrix from the lidar coordinate system to the camera coordinate system;

[0023] S2.22: According to the numerical value d in the depth image n , back-project the pixel point (u, v) into three-dimensional space to obtain the corresponding three-dimensional point P d,n =f(K,d n ), where K represents the intrinsic parameter matrix of the camera and f(·) represents the back-projection function;

[0024] S2.23: Set the maximum number of iterations, based on the error e between the 3D point cloud and the depth image. n =|P d,n -P C,n |, the minimum error e is obtained by iterative closest point optimization algorithm n,min , until the maximum number of iterations is reached;

[0025] S2.24: Output the registered 3D point cloud data, depth image and optimized camera pose.

[0026] Specifically, the specific steps of step S3 include:

[0027] S3.1: Obtain full sensory environment data X = {x1,...,x m}, and load the hyper-realistic 3D model built based on the radar scan and depth camera data, and at the same time, obtain the simulation results in steps S2.4 and S2.5, where x m represents the mth full sensory environment data, and m represents the number of full sensory environment data;

[0028] S3.2: Analyze the simulation results and full sensory environment data, and formulate adjustment strategies for virtual objects and scene attributes based on the analysis results. According to the adjustment strategies, dynamically adjust the position, size, color, and texture attributes of virtual objects, as well as the weather and time changes of the scene. At the same time, render the adjusted scenes and objects in real time in the virtual environment;

[0029] S3.3: Record and analyze the user’s historical behavior data and current status information in the Metaverse scenario;

[0030] S3.4: Use behavior prediction strategies to predict user behavior, and dynamically adjust the layout of elements in the metaverse scene based on the user behavior prediction results, and optimize the interaction method;

[0031] S3.5: Provide real-time feedback on adjustment effects and collect user feedback data.

[0032] Specifically, the specific steps of the behavior prediction strategy in S3.4 include:

[0033] S3.41: Obtain the historical behavior data and current state information in step S3.3, integrate them into state data, and pre-process the state data;

[0034] S3.42: Build a CNN-LSTM hybrid prediction model, introduce an incremental learning method, input the preprocessed state data into the CNN-LSTM hybrid prediction model for training, and set hyperparameters and training strategies;

[0035] S3.43: Use the trained CNN-LSTM hybrid prediction model to perform multimodal prediction of user behavior, combine the contextual information of the current metaverse scene, perform situation-aware behavior prediction, and output the prediction results in multimodal form;

[0036] S3.44: Apply the prediction results to the metaverse scenario in real time and make dynamic adjustments based on the difference between the user's actual behavior and the prediction results.

[0037] Specifically, the multi-dimensional environmental data in step S1 includes light intensity, sound level, user position and action; and the sensory data includes temperature, humidity, smell and touch.

[0038] An environment-adaptive metaverse scene perception system, comprising: a data processing module, a three-dimensional modeling module, and an optimization module;

[0039] The data processing module is used to collect multi-dimensional environmental data and sensory data in real time and perform pre-processing to obtain full sensory environmental data;

[0040] The 3D modeling module uses radar scanning and depth cameras to capture the 3D details of the physical environment and construct a 3D model. Based on the 3D model and combined with real-time full sensory environment data, it dynamically adjusts the metaverse scene elements through intelligent algorithms and predicts user behavior to optimize the scene layout and interactive experience.

[0041] The optimization module uses VR immersive interaction to collect user feedback, evaluate the metaverse scene perception method, and perform iterative optimization based on feedback and needs.

[0042] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of an environment-adaptive metaverse scene perception method when executing the computer program.

[0043] A computer-readable storage medium stores computer instructions, which, when executed, execute the steps of an environment-adaptive metaverse scene perception method.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention proposes an environment-adaptive metaverse scene perception system, and optimizes and improves the architecture, operating steps and processes. The system has the advantages of simple process, low investment and operating costs, and low production work costs.

[0046] 2. The present invention proposes an environment-adaptive metaverse scene perception method, which realizes the real-time collection and processing of multi-dimensional environmental data and sensory data by integrating advanced technologies such as sensor arrays, radar scanning, and depth cameras, and constructs a highly realistic hyper-realistic three-dimensional model. It effectively simulates the complex environmental characteristics and ecosystems of the physical world, which not only enhances the realism and immersion of the metaverse scene, but also provides users with a richer and more three-dimensional interactive experience.

[0047] 3. The present invention proposes an environment-adaptive metaverse scene perception method, introduces intelligent algorithms and user behavior prediction methods, realizes dynamic adjustment and layout optimization of metaverse scene elements, and ensures that the scene can closely fit the actual needs and preferences of users; collects user feedback through VR immersive interaction, and conducts continuous iterative optimization based on this, further improving the accuracy of the scene perception method and the satisfaction of the user experience, creating a more real, interactive and personalized virtual world for users. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of an environment-adaptive metaverse scene perception method of the present invention;

[0049] Figure 2This is a principle flow chart of an environment-adaptive metaverse scene perception method of the present invention;

[0050] Figure 3 This is a flow chart of a hyper-realistic three-dimensional model simulation of an environment-adaptive metaverse scene perception method of the present invention;

[0051] Figure 4 This is an architecture diagram of an environment-adaptive metaverse scene perception system of the present invention. DETAILED DESCRIPTION

[0052] Example 1

[0053] See also Figure 1-Figure 3 , an embodiment of the present invention provides: an environment-adaptive metaverse scene perception method, comprising the following steps:

[0054] Step S1: In the metaverse scene, multi-dimensional environmental data and sensory data are collected in real time through a sensor array, and pre-processed to obtain full sensory environmental data;

[0055] The multi-dimensional environmental data in S1 includes light intensity, sound level, user position and movement; the sensory data includes temperature, humidity, smell and touch.

[0056] Step S2: Acquire three-dimensional information of the physical environment through radar scanning and depth camera, build a hyper-realistic three-dimensional model, and use the hyper-realistic three-dimensional model to simulate environmental characteristics and complex ecosystems of the physical world;

[0057] Step S3: Based on the hyper-realistic 3D model and its simulation results, combined with the real-time collected full sensory environment data, an intelligent algorithm is used to dynamically adjust the elements in the metaverse scene. At the same time, a user behavior prediction method is introduced to dynamically adjust the metaverse scene layout and interaction mode according to the user's historical behavior and current status;

[0058] Step S4: Use VR immersive interaction to collect user feedback on the metaverse scene, evaluate the scene perception method based on user feedback and actual needs, and perform iterative optimization based on the evaluation results.

[0059] Furthermore, the specific steps of step S4 include:

[0060] S4.1: Select VR devices, such as VR helmets and controllers, and ensure that the devices have high resolution, low latency and other features to provide a high-quality immersive experience;

[0061] S4.2: Use 3D modeling software to build the metaverse scene, including terrain, buildings, and characters, and import the constructed scene into the VR platform for preliminary debugging and optimization;

[0062] S4.3: Design the interaction between users and Metaverse scenes, such as gesture recognition and voice control, to ensure that the interaction is natural, intuitive, and in line with user habits;

[0063] S4.4: Conduct user testing to allow users to freely explore the metaverse scenes in the VR environment and experience various interactive functions. Encourage users to put forward opinions and suggestions on scenes, interaction methods, etc.

[0064] S4.5: Collect user feedback through questionnaires and interviews, and use objective measurements such as eye tracking and EEG signals to evaluate scene perception methods based on user feedback and actual needs, and record the evaluation results.

[0065] The specific steps of step S2 include:

[0066] S2.1: Use a multi-line laser radar to scan the physical environment and obtain the three-dimensional point cloud data of the environment. At the same time, use a depth camera to capture the color image and depth information of the environment, generate a depth image, and set the depth image I = {d1,...,d n}, where d n Represents the depth value of the nth pixel (u, v) in the depth image, where n represents the number of pixels in the depth image, u represents the horizontal coordinate of the pixel in the depth image, and v represents the vertical coordinate of the pixel in the depth image;

[0067] S2.2: Use the simultaneous localization and mapping method to align the 3D point cloud data and the depth image;

[0068] S2.3: using a point cloud reconstruction algorithm to convert the registered three-dimensional point cloud data into a three-dimensional mesh model, and fitting the depth image to the surface of the three-dimensional mesh model through adaptive texture mapping. In the present invention, the point cloud reconstruction algorithm specifically selects a Poisson reconstruction method to convert the point cloud data into a three-dimensional mesh model, and the Poisson reconstruction method is a prior art content in the field, and is not an inventive solution of the present application, and is not described in detail here;

[0069] Furthermore, the specific steps of texture mapping include:

[0070] (1) Depth image preprocessing: preprocess the depth image, such as denoising and contrast enhancement;

[0071] (2) Texture mapping algorithm selection: In the present invention, a projection-based texture mapping algorithm is selected;

[0072] (3) Texture mapping: Map the depth image as a texture onto the surface of the three-dimensional mesh model, and adjust the texture coordinates and parameters to ensure that the texture can accurately fit each part of the mesh model. The specific process of texture mapping is the prior art content in this field and is not the inventive solution of this application, so it will not be elaborated here.

[0073] S2.4: Apply global illumination algorithms to simulate natural lighting and shadow effects, use physics engines to simulate the mechanical behavior between objects, and use acoustic simulation algorithms to generate sound propagation and reflection effects in three-dimensional space based on the location of the sound source, environmental materials, and geometric shapes;

[0074] Furthermore, the specific steps of applying the global illumination algorithm to simulate natural lighting and shadow effects include:

[0075] (1) Import the created 3D scene, including geometry, light source, and material, and set the position, color, and intensity properties of the light source;

[0076] (2) Direct light sources and indirect light sources are set as needed to perform direct illumination and indirect illumination. Direct illumination is to calculate the illumination effect of the light source directly irradiating the surface of the object. Indirect illumination is to calculate the indirect illumination effects such as reflection, refraction, and scattering between objects through a global illumination algorithm. The calculation of the illumination effect of the light source directly irradiating the surface of the object and the global illumination algorithm are both prior art contents in the field and are not the inventive solution of the present application, and are not elaborated here.

[0077] (3) Using the Shadow Mapping method to generate shadow effects between objects. Similarly, the specific process of using the Shadow Mapping method to generate shadows is the prior art content in this field, which is not the inventive solution of the present application and will not be described in detail here;

[0078] (4) Rendering the calculated lighting and shadow effects onto the image. Similarly, the process and steps of rendering the lighting and shadow effects are the prior art content in the field and are not the inventive solution of the present application, and will not be elaborated here.

[0079] Furthermore, the specific steps of using the physics engine to simulate the mechanical behavior between objects include:

[0080] (1) Similarly import the created physical model and set the properties;

[0081] (2) Applying Newton's laws of motion and momentum conservation to calculate the interaction forces between objects, such as gravity, friction, and collision force, wherein the process of solving Newton's laws of motion and momentum conservation equations is the prior art content in this field, and is not an inventive solution of the present application, and is not elaborated here;

[0082] (3) using a collision detection algorithm to detect collisions between objects, and updating the position, velocity, and acceleration of the objects according to the collision detection results, wherein the collision detection algorithm is a prior art in the art and is not an inventive solution of the present application, and is not described in detail herein;

[0083] (4) Render the motion state of the object onto the image to generate a dynamic effect.

[0084] Furthermore, the specific steps of the acoustic simulation algorithm to generate the propagation and reflection effects of sound in three-dimensional space include:

[0085] (1) Define the location, sound pressure level, frequency and other properties of the sound source;

[0086] (2) Construct a three-dimensional acoustic environment model, including walls, floors, and ceiling reflective surfaces;

[0087] The steps of constructing the three-dimensional acoustic environment model include:

[0088] 1) Determine the modeling scope and set the target parameters. The modeling scope includes the acoustic environment area that needs to be simulated, such as indoor rooms and outdoor squares. The target parameters include reverberation time, sound pressure level, and frequency response.

[0089] 2) Use a sound level meter to measure the actual sound environment, record key acoustic parameters and sound signals, and use a 3D scanner to perform a 3D scan of the sound environment to obtain the geometric shape and size of the environment;

[0090] 3) Clean, denoise and calibrate the collected data to ensure the accuracy and reliability of the data;

[0091] 4) Based on the scanned data, use SketchUp 3D modeling software to create a geometric model of the acoustic environment, including all objects that may affect sound propagation, such as walls, ceilings, floors, furniture, etc., and assign acoustic material properties to each object in the model, such as sound absorption coefficient, reflection coefficient, etc., among which these properties will affect the propagation and reflection of sound in the environment;

[0092] 5) Use EASE acoustic simulation software to simulate the propagation, reflection and attenuation of sound in three-dimensional space;

[0093] 6) According to the modeling objectives and requirements, set the simulated acoustic parameters and boundary conditions, including the location of the sound source, sound pressure level, frequency, etc., as well as the reverberation time and sound absorption coefficient of the environment;

[0094] 7) Run the simulation in the software to observe the propagation and reflection of sound in the 3D acoustic environment, and adjust the model or simulation parameters as needed to obtain more accurate simulation results;

[0095] 8) Analyze the simulation results to evaluate whether the acoustic performance of the acoustic environment meets the requirements, including checking whether key acoustic parameters such as reverberation time and sound pressure level distribution meet the standards or design requirements;

[0096] 9) Based on the analysis results, optimize the design of the acoustic environment model, including adjusting the position, shape or material properties of objects, and adding or reducing sound-absorbing materials.

[0097] (3) using a wave equation to simulate the propagation of sound waves in three-dimensional space and calculate the reflection and scattering effects of the sound waves on the reflecting surface, wherein the process of solving the wave equation is the prior art content in this field and is not the inventive solution of the present application, and is not described in detail here;

[0098] (4) Use reverberation time parameters to simulate indoor sound field effects;

[0099] (5) Render the simulated sound effect to the audio output device.

[0100] S2.5: Use meteorological simulation algorithms to simulate meteorological changes under different weather conditions, as well as the impact of day and night cycles and seasonal changes on the environment. At the same time, introduce AI-controlled creatures into the metaverse, use behavior tree algorithms to simulate the interactions between creatures, and perform sentiment analysis on the interactions between creatures;

[0101] Furthermore, the specific steps of the meteorological simulation algorithm include:

[0102] (1) Collect meteorological data through observation stations, satellites, and radars, and clean, denoise, calibrate, and format the collected meteorological data to meet simulation requirements;

[0103] (2) Based on meteorological principles and data characteristics, a numerical model is established to simulate changes in atmospheric conditions;

[0104] (3) Set the meteorological conditions at the beginning of the simulation, such as temperature, humidity, air pressure, and wind speed, and set the boundary conditions of the simulation area, such as the influence of terrain, ocean, and land;

[0105] (4) Simulate the weather system through computer models to predict future weather conditions, and introduce time variables into the model to simulate the impact of day and night cycles and seasonal changes on meteorological changes;

[0106] (5) Present the simulation results in a visual way, such as weather maps and meteorological cloud maps;

[0107] (6) Compare actual observation data with simulation results, verify the accuracy and reliability of the simulation, analyze the source of errors, and improve simulation accuracy.

[0108] The implementation steps of AI control organisms and behavior tree algorithms include:

[0109] (1) Create a three-dimensional model of organisms in the metaverse and assign them physical properties and behavioral characteristics;

[0110] (2) Integrate AI algorithms into biological models to enable organisms to autonomously perceive the environment, make decisions, and perform actions;

[0111] (3) Design a behavior tree structure based on the behavioral characteristics of the organism, including the root node, control flow node, and execution node, and write implementation code for each node to define its behavior logic and return value;

[0112] (4) Run the behavior tree in the metaverse environment, select and execute corresponding behaviors based on the current environment and biological state;

[0113] (5) setting interaction rules between organisms, such as foraging, escaping, and competing, using a behavior tree algorithm to simulate the interaction behaviors between organisms, and selecting and executing corresponding actions according to the interaction rules. The calculation process of the behavior tree algorithm is the prior art content in this field, and is not the inventive solution of the present application, and is not described in detail here;

[0114] (6) Perform sentiment analysis on the interactions between organisms and identify the main emotional tendencies, such as happiness, sadness, and tension.

[0115] S2.6: According to the emotion analysis results, combined with the simulation results in step S2.4 and step S2.5, dynamically adjust the lighting, shadow, and sound effects in the scene.

[0116] For example, add warm lighting and soft background music during pleasant interactions; add dim lighting and tense background sound effects during tense situations.

[0117] The specific steps of S2.2 include:

[0118] S2.21: Extract feature points P from 3D point cloud data L,n , by calculating P C,n =T LC ×P L,n , transform the 3D point cloud data into the camera coordinate system to obtain the initial camera pose P C,n , where T LC Represents the transformation matrix from the lidar coordinate system to the camera coordinate system;

[0119] S2.22: According to the numerical value d in the depth image n , back-project the pixel point (u, v) into three-dimensional space to obtain the corresponding three-dimensional point P d,n =f(K,d n), where K represents the intrinsic parameter matrix of the camera and f(·) represents the back-projection function;

[0120] S2.23: Set the maximum number of iterations, based on the error e between the 3D point cloud and the depth image. n =|P d,n -P C,n |, the minimum error e is obtained by iterative closest point optimization algorithm n,min , until the maximum number of iterations is reached, wherein the goal of the iterative closest point optimization algorithm is to optimize the error function, and the calculation formula of the iterative closest point optimization algorithm is the prior art content in this field, which is not the inventive solution of the present application and will not be described in detail here;

[0121] S2.24: Output the registered 3D point cloud data, depth image and optimized camera pose.

[0122] The specific steps of step S3 include:

[0123] S3.1: Obtain full sensory environment data X = {x1,...,x m}, and load the hyper-realistic 3D model built based on the radar scan and depth camera data, and at the same time, obtain the simulation results in steps S2.4 and S2.5, where x m represents the mth full sensory environment data, and m represents the number of full sensory environment data;

[0124] S3.2: Analyze the simulation results and full sensory environment data, and formulate adjustment strategies for virtual objects and scene attributes based on the analysis results. According to the adjustment strategies, dynamically adjust the position, size, color, and texture attributes of virtual objects, as well as the weather and time changes of the scene. At the same time, render the adjusted scenes and objects in real time in the virtual environment;

[0125] Furthermore, the present invention uses a long short-term memory network to analyze the simulation results and the full sensory environment data, and the specific steps include:

[0126] S3.21: Convert the simulation results and full sensory environment data into a format that can be processed by the recurrent neural network long short-term memory network model, and perform normalization and data partitioning;

[0127] S3.22: Load the pre-built long short-term memory network model, and use the divided training set to train the long short-term memory network model, calculate the output through forward propagation, update the model parameters through the back propagation algorithm and the Adam optimization algorithm, and use the test set data to evaluate the performance of the model, wherein the construction process of the long short-term memory network model, the forward propagation calculation formula, the back propagation calculation formula and the Adam optimization algorithm are all prior art contents in this field, and are not the inventive solution of the present application, and are not described in detail here;

[0128] S3.23: Formulate adjustment strategies for virtual object and scene attributes based on the analysis results of the long short-term memory network model;

[0129] S3.24: Dynamically adjust the position, size, color, texture attributes of virtual objects and the weather and time changes of the scene according to the adjustment strategy;

[0130] S3.25: Render the adjusted scenes and objects in real time in the virtual environment.

[0131] It should be noted that in the present invention, the main purpose of intelligent analysis of simulation results and full sensory environment data is: 1) By analyzing user behavior patterns and environmental data through intelligent algorithms, users' needs and preferences can be understood more accurately, thereby dynamically adjusting the properties of virtual environments and objects; 2) Real-time analysis of full sensory environment data helps to simulate a more realistic physical environment and make virtual scenes more realistic. By analyzing these data and adjusting the properties of virtual objects, including but not limited to color, texture, and shadow, the realism of the scene can be further enhanced; 3) Resource usage in the virtual environment can be analyzed, and optimization strategies can be formulated based on the analysis results, such as dynamically adjusting the complexity and rendering quality of virtual objects to balance performance and visual effects and optimize resource allocation. After intelligent analysis of the simulation results and full sensory environment data, we can obtain: 1) Identify common user behavior patterns, such as browsing habits, interaction methods, etc., to provide a basis for formulating personalized adjustment strategies; 2) Through the analysis of full sensory environment data, we can gain in-depth insights into the state of the virtual environment, such as light intensity, temperature distribution, and sound characteristics; 3) Specific adjustment strategies can be formulated, including adjustments to the position, size, color, and texture attributes of virtual objects, as well as simulation of weather and time changes in the scene.

[0132] S3.3: Record and analyze the user’s historical behavior data and current status information in the Metaverse scenario;

[0133] S3.4: Use behavior prediction strategies to predict user behavior, and dynamically adjust the layout of elements in the metaverse scene based on the user behavior prediction results, and optimize the interaction method;

[0134] S3.5: Provide real-time feedback on adjustment effects and collect user feedback data.

[0135] The specific steps of the behavior prediction strategy in S3.4 include:

[0136] S3.41: Obtain the historical behavior data and current state information in step S3.3, integrate them into state data, and pre-process the state data, wherein the integration method is to directly integrate by a head-to-tail splicing method;

[0137] S3.42: Build a CNN-LSTM hybrid prediction model, introduce an incremental learning method, input the preprocessed state data into the CNN-LSTM hybrid prediction model for training, and set hyperparameters and training strategies;

[0138] Among them, the CNN-LSTM hybrid prediction model refers to a hybrid model composed of a convolutional neural network and a long short-term memory network. The convolutional neural network part is used to extract spatial features in the state data, such as the local pattern of the user's movement trajectory, and the long short-term memory network part is used to capture the long-term dependencies in the time series. At the same time, the attention mechanism is introduced to enhance the model's attention to important features.

[0139] Hyperparameters can be set, including learning rate, batch size, and number of iterations; training strategies include learning rate decay and early stopping, among which the specific settings and operations of learning rate decay and early stopping are the prior art content in this field, and are not the inventive solution of this application, and are not elaborated here; and the purpose of setting incremental learning is to enable the model to continuously learn new data without forgetting too much old knowledge, which is mainly achieved by using the regularization term method. The calculation process of the regularization term method is the prior art content in this field, and is not the inventive solution of this application, and is not elaborated here.

[0140] Furthermore, the specific process of constructing the CNN-LSTM hybrid prediction model includes:

[0141] (1) Collect status data and perform preprocessing, including missing value and outlier processing and standardization;

[0142] (2) Divide the state data into a training set, a validation set, and a test set, where the division ratio is set to 70%:15%:15%;

[0143] (3) Convert the state data format into a three-dimensional tensor format of (seq_len, batch, input_size), where seq_len represents the sequence length, batch represents the batch size, and input_size represents the input size;

[0144] (4) Use convolutional neural network (CNN) to extract spatial features of input state data, and design convolutional layers, activation layers, and pooling layers to construct CNN modules. The number of convolutional layers, convolution kernel size, step size, and padding parameters need to be set according to specific tasks and data characteristics.

[0145] (5) Use the long short-term memory network (LSTM) to capture the temporal dependency of state data, design the long short-term memory network layer, set the parameters of the hidden layer size, number of layers, and whether to use bidirectional LSTM. It should be noted that the input of the LSTM layer should be the feature sequence output by the CNN module;

[0146] (6) The output of the CNN module is transformed into an input of the LSTM module to achieve feature fusion;

[0147] (7) Add a fully connected layer after the LSTM layer, map the output of the LSTM to the target prediction value, and obtain the CNN-LSTM hybrid prediction model.

[0148] (8) Setting hyperparameters, including learning rate, batch size, training rounds, and optimizer, and the hyperparameter setting method is adjusted according to the experiment and the performance of the CNN-LSTM hybrid prediction model;

[0149] (9) The CNN-LSTM hybrid prediction model is trained using the training set, and the parameters of the CNN-LSTM hybrid prediction model are optimized using the back propagation algorithm. The back propagation algorithm is a prior art in this field and is not an inventive solution of the present application, so it is not described in detail here.

[0150] S3.43: Use the trained CNN-LSTM hybrid prediction model to perform multimodal prediction of user behavior, combine the contextual information of the current metaverse scene, perform situation-aware behavior prediction, and output the prediction results in multimodal form;

[0151] S3.44: Apply the prediction results to the metaverse scenario in real time and make dynamic adjustments based on the difference between the user's actual behavior and the prediction results.

[0152] Example 2

[0153] See also Figure 4 Another embodiment provided by the present invention is an environment-adaptive metaverse scene perception system, comprising:

[0154] Data processing module, 3D modeling module, optimization module;

[0155] The data processing module is used to collect multi-dimensional environmental data and sensory data in real time and perform pre-processing to obtain full sensory environmental data;

[0156] The 3D modeling module uses radar scanning and depth cameras to capture the 3D details of the physical environment and build a high-precision 3D model. Based on the 3D model and combined with real-time full sensory environmental data, it dynamically adjusts the metaverse scene elements through intelligent algorithms and predicts user behavior to optimize the scene layout and interactive experience.

[0157] The optimization module uses VR immersive interaction to collect user feedback, evaluate the metaverse scene perception method, and perform iterative optimization based on feedback and needs.

[0158] The data processing module includes: a sensor array unit and a data preprocessing unit;

[0159] The sensor array unit includes a temperature sensor, a humidity sensor, a light sensor, and a sound sensor, which are used to collect physical quantities in the environment in real time, such as temperature, humidity, light intensity, and sound;

[0160] The data preprocessing unit is used to clean, denoise, filter and standardize the raw data collected by the sensor to improve the data quality and accuracy.

[0161] The 3D modeling module includes: 3D data acquisition unit, 3D modeling unit, environment simulation unit, intelligent algorithm unit, user behavior prediction unit, and dynamic adjustment unit;

[0162] A 3D data acquisition unit uses radar scanning and depth camera equipment to obtain 3D point cloud data and depth images of the physical environment;

[0163] The 3D modeling unit uses the point cloud reconstruction algorithm technology based on the collected 3D point cloud data to construct an accurate 3D mesh model;

[0164] The environmental simulation unit uses technologies such as physics engines and lighting algorithms to simulate environmental characteristics of the physical world, such as lighting, shadows, physical collisions, and complex ecosystems, such as weather changes, day and night cycles, and seasonal changes;

[0165] Intelligent algorithm unit, used to analyze and process the data collected in real time, and dynamically adjust the elements in the scene according to the analysis results, such as the position, size, color, texture and scene layout of virtual objects;

[0166] The user behavior prediction unit uses the behavior prediction strategy to predict the user's future behavior based on the user's historical behavior and current status information, and adjusts the scene's interaction method based on the prediction results;

[0167] The dynamic adjustment unit performs adjustment operations on scene elements and layouts based on the results of intelligent algorithms and user behavior predictions.

[0168] The optimization module includes: VR interaction unit, evaluation and optimization unit;

[0169] VR interaction unit, used to provide VR immersive interaction, allowing users to experience the metaverse scene in person, and collect user feedback;

[0170] The evaluation and optimization unit is used to analyze and evaluate the collected user feedback, and iteratively optimize the scene perception method based on the evaluation results and actual needs to improve the user experience and the realism of the scene.

[0171] Example 3

[0172] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of an environment-adaptive metaverse scene perception method are implemented. For details, please refer to the above method embodiment, which will not be repeated here.

[0173] A computer-readable storage medium stores computer instructions, which, when executed, execute the steps of an environment-adaptive metaverse scene perception method, wherein the storage medium may be a volatile or non-volatile computer-readable storage medium.

[0174] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention, and all of these are within the protection of the present invention.

[0175] If the disclosed technical solution involves personal information, the product using the disclosed technical solution has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the disclosed technical solution involves sensitive personal information, the product using the disclosed technical solution has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. An environment-adaptive metaverse scene perception method, characterized in that: include: Step S1: In the metaverse scene, multi-dimensional environmental data and sensory data are collected in real time through a sensor array, and pre-processed to obtain full sensory environmental data; Step S2: Acquire three-dimensional information of the physical environment through radar scanning and depth camera, build a hyper-realistic three-dimensional model, and use the hyper-realistic three-dimensional model to simulate environmental characteristics and complex ecosystems of the physical world; Step S3: Based on the hyper-realistic 3D model and its simulation results, combined with the real-time collected full sensory environment data, an intelligent algorithm is used to dynamically adjust the elements in the metaverse scene. At the same time, a user behavior prediction method is introduced to dynamically adjust the metaverse scene layout and interaction mode according to the user's historical behavior and current status; Step S4: Use VR immersive interaction to collect user feedback on the metaverse scene, evaluate the scene perception method based on user feedback and actual needs, and perform iterative optimization based on the evaluation results.

2. The method for environmentally adaptive metaverse scene perception according to claim 1, characterized in that: The specific steps of step S2 include: S2.1: Use a multi-line laser radar to scan the physical environment and obtain the three-dimensional point cloud data of the environment. At the same time, use a depth camera to capture the color image and depth information of the environment, generate a depth image, and set the depth image I = {d1,...,d n }, where d n Represents the depth value of the nth pixel (u, v) in the depth image, where n represents the number of pixels in the depth image, u represents the horizontal coordinate of the pixel in the depth image, and v represents the vertical coordinate of the pixel in the depth image; S2.2: Use the simultaneous localization and mapping method to align the 3D point cloud data and the depth image; S2.3: Use a point cloud reconstruction algorithm to convert the registered 3D point cloud data into a 3D mesh model, and fit the depth image to the surface of the 3D mesh model through adaptive texture mapping.

3. The method for environmentally adaptive metaverse scene perception according to claim 2, characterized in that: The specific steps of step S2 also include: S2.4: Apply global illumination algorithms to simulate natural lighting and shadow effects, use physics engines to simulate the mechanical behavior between objects, and use acoustic simulation algorithms to generate sound propagation and reflection effects in three-dimensional space based on the location of the sound source, environmental materials, and geometric shapes; S2.5: Use meteorological simulation algorithms to simulate meteorological changes under different weather conditions, as well as the impact of day and night cycles and seasonal changes on the environment. At the same time, introduce AI-controlled creatures into the metaverse, use behavior tree algorithms to simulate the interactions between creatures, and perform sentiment analysis on the interactions between creatures; S2.6: Based on the sentiment analysis results and combined with the simulation results in step S2.4 and step S2.5, dynamically adjust the lighting, shadow, and sound effects in the metaverse scene.

4. The method for environmentally adaptive metaverse scene perception according to claim 3, characterized in that: The specific steps of S2.2 include: S2.21: Extract feature points P from 3D point cloud data L,n , by calculating P C,n =T LC ×P L,n , transform the 3D point cloud data into the camera coordinate system to obtain the initial camera pose P C,n , where T LC Represents the transformation matrix from the lidar coordinate system to the camera coordinate system; S2.22: According to the numerical value d in the depth image n , back-project the pixel point (u, v) into three-dimensional space to obtain the corresponding three-dimensional point P d,n =f(K,d n ), where K represents the intrinsic parameter matrix of the camera and f(·) represents the back-projection function; S2.23: Set the maximum number of iterations, based on the error e between the 3D point cloud and the depth image. n =|P d,n -P C,n |, the minimum error e is obtained by iterative closest point optimization algorithm n,min , until the maximum number of iterations is reached; S2.24: Output the registered 3D point cloud data, depth image and optimized camera pose.

5. The method for environmentally adaptive metaverse scene perception according to claim 4, characterized in that: The specific steps of step S3 include: S3.1: Obtain full sensory environment data X = {x1,…,x m }, and load the hyper-realistic 3D model built based on the radar scan and depth camera data, and at the same time, obtain the simulation results in steps S2.4 and S2.5, where x m represents the mth full sensory environment data, and m represents the number of full sensory environment data; S3.2: Analyze the simulation results and full sensory environment data, and formulate adjustment strategies for virtual objects and scene attributes based on the analysis results. According to the adjustment strategies, dynamically adjust the position, size, color, and texture attributes of virtual objects, as well as the weather and time changes of the scene. At the same time, render the adjusted scenes and objects in real time in the virtual environment; S3.3: Record and analyze the user’s historical behavior data and current status information in the Metaverse scenario; S3.4: Use behavior prediction strategies to predict user behavior, and dynamically adjust the layout of elements in the metaverse scene based on the user behavior prediction results, and optimize the interaction method; S3.5: Provide real-time feedback on adjustment effects and collect user feedback data.

6. The method for environmentally adaptive metaverse scene perception according to claim 5, characterized in that: The specific steps of the behavior prediction strategy in S3.4 include: S3.41: Obtain the historical behavior data and current state information in step S3.3, integrate them into state data, and pre-process the state data; S3.42: Build a CNN-LSTM hybrid prediction model, introduce an incremental learning method, input the preprocessed state data into the CNN-LSTM hybrid prediction model for training, and set hyperparameters and training strategies; S3.43: Use the trained CNN-LSTM hybrid prediction model to perform multimodal prediction of user behavior, combine the contextual information of the current metaverse scene, perform situation-aware behavior prediction, and output the prediction results in multimodal form; S3.44: Apply the prediction results to the metaverse scenario in real time and make dynamic adjustments based on the difference between the user's actual behavior and the prediction results.

7. The method for environmentally adaptive metaverse scene perception according to claim 6, characterized in that: The multi-dimensional environmental data in step S1 includes light intensity, sound level, user position and movement; the sensory data includes temperature, humidity, smell and touch.

8. An environment-adaptive metaverse scene perception system, which is used to implement an environment-adaptive metaverse scene perception method according to any one of claims 1 to 7, characterized in that: include: Data processing module, 3D modeling module, optimization module; The data processing module is used to collect multi-dimensional environmental data and sensory data in real time and perform pre-processing to obtain full sensory environmental data; The 3D modeling module uses radar scanning and depth cameras to capture the 3D details of the physical environment and construct a 3D model. Based on the 3D model and combined with real-time full sensory environment data, it dynamically adjusts the metaverse scene elements through intelligent algorithms and predicts user behavior to optimize the scene layout and interactive experience. The optimization module uses VR immersive interaction to collect user feedback, evaluate the metaverse scene perception method, and perform iterative optimization based on feedback and needs.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of an environment-adaptive metaverse scene perception method as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the steps of an environment-adaptive metaverse scene perception method as described in any one of claims 1-7 are executed.

Images