Three-dimensional scene user interaction method and system based on digital twinning

Abstract

The invention relates to the technical field of digital twinning, and provides a digital twinning-based three-dimensional scene user interaction method and system, and the method comprises the following steps: obtaining physical scene data of a target scene, mapping the physical scene data into a virtual three-dimensional scene, and generating a digital twinning model; acquiring a pressing signal, and acquiring obstacle avoidance moving path data according to the pressing signal and the topographic data of the virtual three-dimensional scene; acquiring a rotation trigger signal, and acquiring scene real-time rotation angle data based on the rotation trigger signal; acquiring a zooming operation signal, and acquiring a target area focusing parameter according to the zooming operation signal; and collecting user operation habit data, predicting the user operation habit data based on the machine learning model to obtain a user behavior prediction result, and obtaining the scene state parameters according to the user behavior prediction result. According to the invention, the real-time updating of the physical scene state and the dynamic adjustment of the virtual scene operation on the physical scene are realized, and the real-time performance and intelligence of the three-dimensional scene interaction are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of digital twins, and particularly to a three-dimensional scene user interaction method and system based on digital twins. Background Art

[0002] With the rapid development of virtual reality and digital twin technologies, as a key technology for realizing data interaction between physical scenes and virtual environments, three-dimensional scene interaction technology has been widely applied in fields such as industrial design, urban planning, and virtual simulation. The three-dimensional scene interaction technology mainly realizes interactive operations such as moving, rotating, and scaling of the scene by acquiring physical scene data and mapping it into a virtual three-dimensional scene. However, in actual applications, there are still many deficiencies in the existing three-dimensional scene interaction technology in terms of data synchronization, interaction accuracy, and user experience.

[0003] The three-dimensional scene interaction methods in the prior art mainly rely on simple mouse and keyboard operations, combined with fixed parameter settings and one-way data transmission methods. In the operations of scene movement, rotation, and scaling, there is a lack of obstacle avoidance path planning, inertial rotation calculation, and depth perception, which easily lead to inaccurate operations, perspective jumps, or uneven scaling. Summary of the Invention

[0004] In view of this, the present invention proposes a three-dimensional scene user interaction method and system based on digital twins, which solves the problems in the prior art that in the operations of scene movement, rotation, and scaling, there is a lack of obstacle avoidance path planning, inertial rotation calculation, and depth perception, which easily lead to inaccurate operations, perspective jumps, or uneven scaling.

[0005] The technical solution of the present invention is implemented as follows: In the first aspect, the present invention provides a three-dimensional scene user interaction method based on digital twins, including the following steps:

[0006] Acquire the physical scene data of the target scene, map the physical scene data into a virtual three-dimensional scene, and generate a digital twin model;

[0007] Detect the left mouse button pressing event of the user, obtain the pressing signal, obtain the obstacle avoidance movement path data according to the pressing signal and the terrain data of the virtual three-dimensional scene, synchronize the obstacle avoidance movement path data to the digital twin model, and adjust the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0008] Detect the right mouse button pressing event of the user, obtain the rotation trigger signal, obtain the real-time rotation angle data of the scene based on the rotation trigger signal, synchronize the real-time rotation angle data of the scene to the digital twin model, and adjust the rotation state of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0009] Detect the user's mouse wheel operation, obtain the zoom operation signal, obtain the initial zoom ratio according to the zoom operation signal, collect the scene depth information, input the scene depth information and the initial zoom ratio into the depth perception algorithm to obtain the dynamic zoom ratio parameter, input the dynamic zoom ratio parameter into the viewpoint optimization algorithm to obtain the target area focusing parameter, synchronize the dynamic zoom ratio parameter to the digital twin model, and adjust the zoom state of the virtual scene according to the real-time feedback of the physical scene;

[0010] Collect the user operation habit data, predict the user operation habit data based on the machine learning model to obtain the user behavior prediction result, obtain the scene state parameter according to the user behavior prediction result, synchronize the scene state parameter to the digital twin model, and record the user operation habit data.

[0011] On the basis of the above technical solutions, preferably, the obtaining of the physical scene data of the target scene and mapping the physical scene data to the virtual three-dimensional scene to generate a digital twin model specifically includes:

[0012] Use the sensor network to collect data from the target scene to obtain geometric structure data, physical attribute data, and real-time status data. The sensor network includes lidar, cameras, and temperature and humidity sensors. During the data collection process, combine the real-time status data with the geometric structure data to generate the time-series physical state data of the target scene;

[0013] Generate a preliminary virtual three-dimensional scene model based on the geometric structure data and physical attribute data of the target scene, calibrate the preliminary virtual three-dimensional scene model to make the preliminary virtual three-dimensional scene model consistent with the geometric shape and physical state of the physical scene, obtain the virtual three-dimensional scene model, and update the parameter status in the virtual three-dimensional scene according to the collected real-time status data.

[0014] On the basis of the above technical solutions, preferably, the detecting of the user's left mouse button pressing event, obtaining the pressing signal, obtaining the obstacle avoidance movement path data according to the pressing signal and the terrain data of the virtual three-dimensional scene, synchronizing the obstacle avoidance movement path data to the digital twin model, and adjusting the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene specifically includes:

[0015] Detect the duration and pressing position of the user's left mouse button pressing event. When the duration exceeds the preset threshold, trigger the scene control interface to generate a pressing signal. In the virtual three-dimensional scene, generate four semi-transparent direction buttons of forward, backward, left, and right centered on the pressing position. The semi-transparent direction buttons are distributed in a cross shape, and the transparency of the semi-transparent direction buttons is dynamically adjusted according to the proximity of the mouse position;

[0016] Obtain the mobile control data of the user on the scene control interface, including the selection of the moving direction and the moving distance data. Input the mobile control data into the exponential smoothing algorithm for processing, obtain the scene moving speed parameter through adaptive weight calculation. Combine the scene moving speed parameter and the terrain data of the virtual 3D scene, and use the path planning algorithm to plan the obstacle avoidance path, generate the obstacle avoidance movement path data, synchronize the obstacle avoidance movement path data to the digital twin model in real time, and dynamically adjust the movement path based on the real-time state data of the physical scene.

[0017] Based on the above technical solutions, preferably, detect the user's right mouse button pressing event, obtain the rotation trigger signal, obtain the real-time rotation angle data of the scene based on the rotation trigger signal, synchronize the real-time rotation angle data of the scene to the digital twin model, and adjust the rotation state of the virtual 3D scene according to the real-time feedback of the physical scene. Specifically, it includes:

[0018] Use the camera and depth sensor to obtain the real-time position information of the user's mouse. The real-time position information of the mouse includes the two-dimensional coordinates of the mouse on the screen and the depth position of the mouse in the three-dimensional space. According to the real-time position information of the mouse, calculate the direction vector of the mouse ray emitted from the camera position, and use the collision detection algorithm of the mouse ray and the 3D model to calculate the intersection coordinates of the mouse ray and the 3D model, and obtain the collision point data;

[0019] The collision detection algorithm uses an improved ray tracing method, and the calculation formula is:

[0020]

[0021]

[0022] where t is the ray distance parameter, P0 is the starting point of the mouse ray, is the direction vector, is the normal vector of the patch of the 3D model, is a point on the patch of the 3D model, and P is the collision point coordinate;

[0023] Apply the density-based spatial clustering algorithm to the collision point data to determine the rotation center coordinates, and input the rotation center coordinates into the inertial rotation algorithm to calculate the real-time rotation angle data of the scene corresponding to the user's rotation operation.

[0024] Based on the above technical solutions, preferably, detecting the user's mouse wheel operation to obtain a zoom operation signal, obtaining an initial zoom ratio according to the zoom operation signal, collecting scene depth information, inputting the scene depth information and the initial zoom ratio into a depth perception algorithm to obtain a dynamic zoom ratio parameter, inputting the dynamic zoom ratio parameter into a viewpoint optimization algorithm to obtain a target area focusing parameter, synchronizing the dynamic zoom ratio parameter to the digital twin model, and adjusting the zoom state of the virtual scene according to the real-time feedback of the physical scene, specifically including:

[0025] Using a roller sensor to detect the roller operation direction and rolling distance of the user to obtain a zoom operation signal; calculating an initial zoom ratio according to the rolling distance, and collecting a scene depth map of the current view through a depth camera, and performing noise reduction processing and depth calibration on the scene depth map to generate scene depth information;

[0026] Inputting the scene depth information into a depth perception algorithm, calculating a dynamic zoom ratio parameter based on scene importance evaluation and depth hierarchy analysis, and using a viewpoint optimization algorithm to analyze the spatial distribution of key areas of the scene, and combining the user's perspective and zoom operation to generate a target area focusing parameter.

[0027] Based on the above technical solutions, preferably, the calculation formula of the depth perception algorithm is:

[0028]

[0029]

[0030] Among them, S(d) is the dynamic zoom ratio parameter, S0 is the initial zoom ratio, d is the actual depth distance from the current observation point to the scene object, d0 is the standard reference depth, λ is the depth attenuation coefficient, W(d) is the depth weight function, d focus is the depth value of the user's focus position, α is the weight attenuation rate, and β is the weight curve shape parameter;

[0031] The calculation formula of the viewpoint optimization algorithm is:

[0032]

[0033]

[0034]

[0035] Among them, F(p) is the target area focusing parameter, w i is the weight value of the i-th key point, G(·) is the spatial Gaussian distribution function, G(p,p i ) is the distance from position p to key point p iThe spatial influence degree, I(p i ) is the importance score of the i-th key point, n is the number of key points in the scene, v i is the visual saliency score of the i-th key point, v j is the visual saliency score of the j-th key point, C(d i ) is the depth compensation function, d i is the actual depth value of the i-th key point, d opt is the optimal depth value for observing the scene, and γ is the depth compensation coefficient.

[0036] Based on the above technical solutions, preferably, collecting user operation habit data, predicting the user operation habit data based on a machine learning model to obtain a user behavior prediction result, obtaining scene state parameters according to the user behavior prediction result, synchronizing the scene state parameters to the digital twin model, and recording user operation habit data specifically includes:

[0037] Collecting user operation habit data in real time, where the user operation habit data includes movement path, rotation angle, scaling ratio, and operation timing information, and inputting the user operation habit data into a machine learning model for training and prediction to obtain a user behavior prediction result;

[0038] Based on the user behavior prediction result, combining scene state parameters and historical interaction data, using a collaborative filtering algorithm to calculate user preference features, generating personalized interaction parameters for the current user, and applying the personalized interaction parameters to scene interaction control.

[0039] In a second aspect, the present invention also provides a three-dimensional scene user interaction system based on digital twin, and the system includes:

[0040] A digital twin construction module for obtaining physical scene data of a target scene, mapping the physical scene data into a virtual three-dimensional scene, and generating a digital twin model;

[0041] An obstacle avoidance movement path module for detecting a user left mouse button press event, obtaining a press signal, obtaining obstacle avoidance movement path data according to the press signal and the terrain data of the virtual three-dimensional scene, synchronizing the obstacle avoidance movement path data to the digital twin model, and adjusting the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0042] A scene rotation angle module for detecting a user right mouse button press event, obtaining a rotation trigger signal, obtaining real-time scene rotation angle data based on the rotation trigger signal, synchronizing the real-time scene rotation angle data to the digital twin model, and adjusting the rotation state of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0043] A dynamic zooming and focusing module is used to detect the operation of the user's mouse wheel, obtain a zooming operation signal, acquire an initial zoom ratio according to the zooming operation signal, collect scene depth information, input the scene depth information and the initial zoom ratio into a depth perception algorithm to obtain a dynamic zoom ratio parameter, input the dynamic zoom ratio parameter into a viewpoint optimization algorithm to obtain a target area focusing parameter, synchronize the dynamic zoom ratio parameter to a digital twin model, and adjust the zoom state of the virtual scene according to the real-time feedback of the physical scene;

[0044] A user operation habit module is used to collect user operation habit data, perform prediction on the user operation habit data based on a machine learning model to obtain a user behavior prediction result, acquire a scene state parameter according to the user behavior prediction result, synchronize the scene state parameter to the digital twin model, and record the user operation habit data.

[0045] In a third aspect, the present invention further provides an electronic device, including: at least one processor, at least one memory, a communication interface, and a bus;

[0046] Wherein, the processor, the memory, and the communication interface complete mutual communication through the bus, the memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the steps of a three-dimensional scene user interaction method based on digital twin.

[0047] In a fourth aspect, the present invention further provides a computer-readable storage medium, and the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the steps of a three-dimensional scene user interaction method based on digital twin.

[0048] The three-dimensional scene user interaction method and system based on digital twin of the present invention have the following beneficial effects compared with the prior art:

[0049] (1) By constructing a digital twin model of the physical scene and the virtual three-dimensional scene, establishing a two-way data synchronization mechanism, realizing the real-time update of the physical scene state and the dynamic adjustment of the physical scene by the virtual scene operation, combining the interactive operations of the left mouse button, the right mouse button, and the mouse wheel, respectively realizing the obstacle avoidance movement path planning, the rotation angle adjustment, and the dynamic zooming and focusing functions, and by collecting user operation habit data and using a machine learning model for behavior prediction, the real-time performance and intelligence of the three-dimensional scene interaction are improved;

[0050] (2) By combining a camera and a depth sensor, the two-dimensional coordinates and three-dimensional depth position information of the mouse are obtained. An improved ray tracing collision detection algorithm is used to calculate the intersection coordinates of the mouse ray and the three-dimensional model, and a density-based spatial clustering algorithm is utilized to determine the rotation center. An inertial rotation algorithm that takes into account the scene quality distribution and rotational inertia is combined to calculate the real-time rotation angle of the scene, thereby achieving precise control of the scene rotation operation and a smooth rotation effect;

[0051] (3) The dynamic scaling ratio parameter is calculated through a depth perception algorithm. By combining scene importance evaluation and depth hierarchy analysis, it is ensured that the scaling operation can be dynamically adjusted according to the actual depth distance of the observation point. At the same time, a view point optimization algorithm is used to analyze the spatial distribution of the key areas of the scene and generate the target area focusing parameter, thereby achieving the accuracy of scene scaling and the intelligent optimization of the user's perspective, and enhancing the experience effect of three-dimensional scene interaction. Brief Description of the Drawings

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following-described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0053] Figure 1 It is a flowchart of a three-dimensional scene user interaction method based on digital twin of the present invention;

[0054] Figure 2 It is a structural diagram of a three-dimensional scene user interaction system based on digital twin of the present invention. Detailed Embodiments

[0055] The following will describe clearly and completely the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0056] Please refer to Figure 1 , the present invention provides a three-dimensional scene user interaction method based on digital twin, including the following steps:

[0057] Obtain the physical scene data of the target scene, map the physical scene data into a virtual three-dimensional scene, generate a digital twin model corresponding to the physical scene, and establish a two-way data synchronization mechanism between the physical scene and the digital twin model for real-time updating of the state data of the virtual scene and adjusting the state of the physical scene according to the operation results of the virtual three-dimensional scene;

[0058] Detect the left mouse button press event of the user, obtain the press signal, and according to the press signal and the terrain data of the virtual three-dimensional scene, obtain the obstacle avoidance movement path data, synchronize the obstacle avoidance movement path data to the digital twin model, and adjust the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0059] Detect the right mouse button press event of the user, obtain the rotation trigger signal, obtain the real-time rotation angle data of the scene based on the rotation trigger signal, synchronize the real-time rotation angle data of the scene to the digital twin model, and adjust the rotation state of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0060] Detect the user's mouse wheel operation, obtain the zoom operation signal, obtain the initial zoom ratio according to the zoom operation signal, collect the scene depth information, input the scene depth information and the initial zoom ratio into the depth perception algorithm to obtain the dynamic zoom ratio parameter, input the dynamic zoom ratio parameter into the viewpoint optimization algorithm to obtain the target area focusing parameter, synchronize the dynamic zoom ratio parameter to the digital twin model, and adjust the zoom state of the virtual scene according to the real-time feedback of the physical scene;

[0061] Collect the user operation habit data, predict the user operation habit data based on the machine learning model to obtain the user behavior prediction result, obtain the scene state parameter according to the user behavior prediction result, synchronize the scene state parameter to the digital twin model, record the user operation habit data, and input the user operation habit data into the collaborative filtering algorithm to generate personalized interaction parameters.

[0062] Specifically, in this embodiment, by constructing a digital twin model of the physical scene and the virtual three-dimensional scene and establishing a two-way data synchronization mechanism, the real-time update of the physical scene state and the dynamic adjustment of the physical scene by the virtual scene operation are realized. Combining the interactive operations of the left mouse button, right mouse button and mouse wheel, the obstacle avoidance movement path planning, rotation angle adjustment and dynamic zoom focusing functions are respectively realized. By collecting the user operation habit data and using the machine learning model for behavior prediction, the real-time performance and intelligence of the three-dimensional scene interaction are improved.

[0063] The obtaining of the physical scene data of the target scene and mapping the physical scene data into the virtual three-dimensional scene to generate a digital twin model specifically includes:

[0064] Use a sensor network to collect data from the target scene, obtaining geometric structure data, physical property data, and real-time status data. The sensor network includes lidar, cameras, and temperature and humidity sensors. During data collection, combine the real-time status data with the geometric structure data to generate time-series physical state data of the target scene;

[0065] Generate a preliminary virtual 3D scene model based on the geometric structure data and physical property data of the target scene, and calibrate the preliminary virtual 3D scene model to make the preliminary virtual 3D scene model consistent with the geometric shape and physical state of the physical scene, obtaining a virtual 3D scene model. According to the collected real-time status data, update the parameter status in the virtual 3D scene in real time to ensure that the digital twin model can dynamically reflect the real-time changes of the physical scene, and achieve two-way synchronization between the digital twin model and the physical scene through a network or communication interface.

[0066] Specifically, in this embodiment, a sensor network deploying lidar, cameras, and temperature and humidity sensors is used to achieve comprehensive data collection of the geometric structure, physical properties, and real-time status of the target scene; by combining the real-time status data with the geometric structure data, generating time-series physical state data, and calibrating and updating the virtual 3D scene model in real time, it is ensured that the digital twin model can accurately reflect the dynamic changes of the physical scene, thereby achieving high-precision two-way synchronization between the physical scene and the digital twin model.

[0067] Detect the user's left mouse button press event, obtain the press signal, obtain obstacle avoidance movement path data according to the press signal and the terrain data of the virtual 3D scene, synchronize the obstacle avoidance movement path data to the digital twin model, and adjust the movement path of the virtual 3D scene according to the real-time feedback of the physical scene. Specifically include:

[0068] Detect the duration and press position of the user's left mouse button press event. When the duration exceeds the preset threshold, trigger the scene control interface to generate a press signal. In the virtual 3D scene, generate four semi-transparent direction buttons of forward, backward, left, and right centered on the press position. The semi-transparent direction buttons are distributed in a cross shape, and the transparency of the semi-transparent direction buttons is dynamically adjusted according to the proximity of the mouse position. When the mouse hovers over the semi-transparent direction button, reduce the transparency to highlight the currently selectable movement direction;

[0069] Obtain the mobile control data of the user on the scene control interface, including the selection of the moving direction and the moving distance data. Input the mobile control data into the exponential smoothing algorithm for processing, and obtain the smoothed scene moving speed parameter through adaptive weight calculation. Combine the scene moving speed parameter and the terrain data of the virtual three-dimensional scene, and use the path planning algorithm to perform obstacle avoidance path planning to generate obstacle avoidance moving path data. Synchronize the obstacle avoidance moving path data to the digital twin model in real time, and dynamically adjust the moving path based on the real-time state data of the physical scene.

[0070] Specifically, in this embodiment, by detecting the left mouse button press event, triggering the dynamic display of the semi-transparent cross-shaped direction button, and combining the intelligent adjustment of the button transparency and the exponential smoothing algorithm to process the mobile control data, the adaptive control of the scene moving speed is realized; at the same time, the path planning algorithm is used to perform obstacle avoidance path planning to ensure the smoothness and safety of the scene movement, thereby realizing the precise control and smooth experience of the three-dimensional scene movement operation.

[0071] Detect the user's right mouse button press event to obtain a rotation trigger signal. Based on the rotation trigger signal, obtain the real-time rotation angle data of the scene, synchronize the real-time rotation angle data of the scene to the digital twin model, and adjust the rotation state of the virtual three-dimensional scene according to the real-time feedback of the physical scene. Specifically, it includes:

[0072] Use the camera and depth sensor to obtain the real-time position information of the user's mouse. The real-time position information of the mouse includes the two-dimensional coordinates of the mouse on the screen and the depth position of the mouse in the three-dimensional space. According to the real-time position information of the mouse, calculate the direction vector of the mouse ray emitted from the camera position, and use the collision detection algorithm of the mouse ray and the three-dimensional model to calculate the intersection coordinates of the mouse ray and the three-dimensional model to obtain the collision point data;

[0073] The collision detection algorithm uses an improved ray tracing method, and the calculation formula is:

[0074]

[0075]

[0076] where t is the ray distance parameter, P0 is the starting point of the mouse ray, is the direction vector, is the normal vector of the patch of the three-dimensional model, is a point on the patch of the three-dimensional model, and P is the collision point coordinate;

[0077] Apply a density-based spatial clustering algorithm to the collision point data to determine the rotation center coordinates, and input the rotation center coordinates into the inertial rotation algorithm. The inertial rotation algorithm calculates the real-time rotation angle data of the scene corresponding to the user's rotation operation by considering the scene mass distribution and the moment of inertia.

[0078] Specifically, in this embodiment, the two-dimensional coordinates and three-dimensional depth position information of the mouse are obtained through a camera and a depth sensor. An improved ray tracing collision detection algorithm is used to accurately calculate the intersection coordinates of the mouse ray and the three-dimensional model, and a density-based spatial clustering algorithm is combined to determine the rotation center. Then, the real-time rotation angle of the scene is calculated through an inertial rotation algorithm that considers the scene mass distribution and the moment of inertia, thereby realizing the precise positioning of the scene rotation operation and the natural and smooth rotation effect.

[0079] Detect the user's mouse wheel operation, obtain a zoom operation signal, obtain an initial zoom ratio according to the zoom operation signal, collect the scene depth information, input the scene depth information and the initial zoom ratio into the depth perception algorithm to obtain a dynamic zoom ratio parameter, input the dynamic zoom ratio parameter into the viewpoint optimization algorithm to obtain a target area focusing parameter, synchronize the dynamic zoom ratio parameter to the digital twin model, and adjust the zoom state of the virtual scene according to the real-time feedback of the physical scene, specifically including:

[0080] Use a roller sensor to detect the user's roller operation direction and rolling distance to obtain a zoom operation signal; calculate the initial zoom ratio according to the rolling distance, and collect the scene depth map at the current perspective through a depth camera, and perform noise reduction processing and depth calibration on the scene depth map to generate standardized scene depth information;

[0081] Input the scene depth information into the depth perception algorithm, and calculate the dynamic zoom ratio parameter based on the scene importance evaluation and depth hierarchy analysis. Use the viewpoint optimization algorithm to analyze the spatial distribution of the key areas of the scene, and combine the user's perspective and zoom operation to generate a target area focusing parameter.

[0082] Specifically, in this embodiment, a roller sensor is used to detect the user's roller operation direction and rolling distance, combined with a depth camera to collect the scene depth map at the current perspective and perform noise reduction and calibration processing to generate standardized scene depth information; the dynamic zoom ratio is calculated through a depth perception algorithm based on the scene importance evaluation and depth hierarchy analysis, and the spatial distribution of the key areas of the scene is analyzed using the viewpoint optimization algorithm, thereby realizing the precise control of the scene zoom operation and the intelligent focusing of the target area.

[0083] The calculation formula of the depth perception algorithm is:

[0084]

[0085]

[0086] Among them, S(d) is the dynamic scaling ratio parameter, S0 is the initial scaling ratio, d is the actual depth distance from the current observation point to the scene object, d0 is the standard reference depth, λ is the depth attenuation coefficient, W(d) is the depth weight function, d focus is the depth value at the position of the user's focus of attention, α is the weight attenuation rate, and β is the weight curve shape parameter;

[0087] The calculation formula of the viewpoint optimization algorithm is:

[0088]

[0089]

[0090]

[0091] Among them, F(p) is the target area focusing parameter, w i is the weight value of the i-th key point, G(·) is the spatial Gaussian distribution function, G(p, p i ) is the spatial influence degree from position p to the key point p i , I(p i ) is the importance score of the i-th key point, n is the number of key points in the scene, v i is the visual saliency score of the i-th key point, v j is the visual saliency score of the j-th key point, C(d i ) is the depth compensation function, d i is the actual depth value of the i-th key point, d opt is the optimal depth value for observing the scene, and γ is the depth compensation coefficient.

[0092] Specifically, in this embodiment, through the depth perception algorithm, according to the actual depth distance from the user's current observation point to the scene object, the standard reference depth, and the depth attenuation coefficient, the scaling ratio parameter is dynamically adjusted to ensure that the scaling operation can adapt to the change of the scene depth, thereby realizing the accuracy and naturalness of the scaling operation.

[0093] By using the depth weight function and the depth value at the position of the user's focus of attention, combined with the weight attenuation rate and the weight curve shape parameter, the scaling ratio is dynamically adjusted, so that the user's focus of attention remains clear and prominent during the scaling process, improving the user's interaction experience.

[0094] Through the viewpoint optimization algorithm, based on the spatial distribution of the key regions in the scene, the visual saliency scores, and the depth compensation function, the focusing parameter of the target region is calculated, enabling the zoom operation to automatically focus on the important regions in the scene, enhancing the user's attention to the key content and the interaction efficiency.

[0095] The depth map of the scene is collected by a depth camera, and noise reduction and depth calibration are performed to generate standardized scene depth information, providing high-quality input data for the depth perception algorithm to ensure the accuracy of the zoom ratio calculation.

[0096] Combining the user's perspective, the zoom operation, and the scene depth information, the focusing parameter of the target region is dynamically adjusted, enabling the user to obtain the best viewing experience at different zoom levels and enhancing the user-friendliness of the 3D scene interaction.

[0097] Collecting the user operation habit data, predicting the user operation habit data based on a machine learning model to obtain the user behavior prediction result, obtaining the scene state parameter according to the user behavior prediction result, and synchronizing the scene state parameter to the digital twin model, recording the user operation habit data, specifically including:

[0098] Real-time collecting the user operation habit data of the user, where the user operation habit data includes the movement path, rotation angle, zoom ratio, and operation timing information, inputting the user operation habit data into the machine learning model for training and prediction to obtain the user behavior prediction result;

[0099] Based on the user behavior prediction result, combining the scene state parameter and the historical interaction data, using the collaborative filtering algorithm to calculate the user preference features, generating personalized interaction parameters for the current user, and applying the personalized interaction parameters to the scene interaction control.

[0100] Specifically, in this embodiment, the operation habit data of the user (including the movement path, rotation angle, zoom ratio, and operation timing information) is collected in real time, and these data are trained and predicted using a machine learning model to generate the user behavior prediction result. Combining the scene state parameter and the historical interaction data, the collaborative filtering algorithm is used to calculate the user preference features, thereby generating personalized interaction parameters and applying them to the scene interaction control.

[0101] Training and predicting user operation habit data through a machine learning model can accurately capture the user's operation patterns and behavior tendencies, thereby improving the accuracy of user behavior prediction. By using the collaborative filtering algorithm to combine the user behavior prediction results and historical interaction data, personalized interaction parameters for the current user are generated, enabling the system to dynamically adjust the interaction method according to the user's preferences, significantly enhancing the user's interaction experience. By predicting the user's operation behavior and adjusting the scene state parameters in advance, the repetition and complexity of user operations are reduced, and the efficiency and smoothness of three-dimensional scene interaction are improved. Real-time recording and updating of user operation habit data enable the system to continuously learn and optimize, gradually adapt to the user's operation habits, and form a more intelligent and user-friendly interaction mode.

[0102] Please refer to Figure 2 , the present invention also provides a three-dimensional scene user interaction system based on digital twin, and the system includes:

[0103] A digital twin construction module for obtaining the physical scene data of the target scene, mapping the physical scene data into a virtual three-dimensional scene, and generating a digital twin model;

[0104] An obstacle avoidance movement path module for detecting a user's left mouse button press event, obtaining a press signal, obtaining obstacle avoidance movement path data according to the press signal and the terrain data of the virtual three-dimensional scene, synchronizing the obstacle avoidance movement path data to the digital twin model, and adjusting the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0105] A scene rotation angle module for detecting a user's right mouse button press event, obtaining a rotation trigger signal, obtaining real-time scene rotation angle data based on the rotation trigger signal, synchronizing the real-time scene rotation angle data to the digital twin model, and adjusting the rotation state of the virtual three-dimensional scene according to the real-time feedback of the physical scene;

[0106] A dynamic zooming and focusing module for detecting a user's mouse wheel operation, obtaining a zooming operation signal, obtaining an initial zoom ratio according to the zooming operation signal, collecting scene depth information, inputting the scene depth information and the initial zoom ratio into a depth perception algorithm to obtain a dynamic zoom ratio parameter, inputting the dynamic zoom ratio parameter into a viewpoint optimization algorithm to obtain a target area focusing parameter, synchronizing the dynamic zoom ratio parameter to the digital twin model, and adjusting the zoom state of the virtual scene according to the real-time feedback of the physical scene;

[0107] A user operation habit module for collecting user operation habit data, predicting the user operation habit data based on a machine learning model to obtain a user behavior prediction result, obtaining scene state parameters according to the user behavior prediction result, synchronizing the scene state parameters to the digital twin model, and recording the user operation habit data.

[0108] Specifically, a three-dimensional scene user interaction system based on digital twin in this embodiment realizes the mapping and two-way synchronization from physical scene data to virtual three-dimensional scene through the system integration of a digital twin construction module, an obstacle avoidance movement path module, a scene rotation angle module, a dynamic zooming and focusing module, and a user operation habit module. Combining the collaborative work of functional modules such as obstacle avoidance movement path planning, scene rotation angle calculation, dynamic zoom ratio optimization, and user operation habit analysis, a complete three-dimensional scene interaction system architecture is constructed, enabling users to achieve precise movement, natural rotation, and intelligent zooming of the scene through the operations of the left mouse button, right mouse button, and scroll wheel. At the same time, the system can adaptively learn the user's operation habits and provide a personalized interaction experience.

[0109] The present invention also discloses an electronic device, including: at least one processor, at least one memory communication interface, and a bus. Among them, the processor, the memory, and the communication interface complete mutual communication through the bus. The memory stores program instructions executable by the processor, and the processor calls the program instructions to implement a three-dimensional scene user interaction method based on digital twin.

[0110] The present invention also discloses a computer-readable storage medium. The computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to implement all or part of the steps of the three-dimensional scene user interaction method described in the embodiments of the present invention. The storage medium includes various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory ROM, random access memory RAM, magnetic disks, or optical discs.

[0111] The above is only the preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A three-dimensional scene user interaction method based on digital twin, characterized in that, The method includes the following steps: Obtain the physical scene data of the target scene, map the physical scene data into a virtual three-dimensional scene, and generate a digital twin model; Detect the left mouse button pressing event of the user, obtain the pressing signal, obtain the obstacle avoidance movement path data according to the pressing signal and the terrain data of the virtual three-dimensional scene, synchronize the obstacle avoidance movement path data to the digital twin model, and adjust the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene; Detect the right mouse button pressing event of the user, obtain the rotation trigger signal, obtain the real-time rotation angle data of the scene based on the rotation trigger signal, synchronize the real-time rotation angle data of the scene to the digital twin model, and adjust the rotation state of the virtual three-dimensional scene according to the real-time feedback of the physical scene; Detect the user's mouse wheel operation, obtain the zoom operation signal, obtain the initial zoom ratio according to the zoom operation signal, collect the scene depth information, input the scene depth information and the initial zoom ratio into the depth perception algorithm to obtain the dynamic zoom ratio parameter, input the dynamic zoom ratio parameter into the viewpoint optimization algorithm to obtain the target area focusing parameter, synchronize the dynamic zoom ratio parameter to the digital twin model, and adjust the zoom state of the virtual scene according to the real-time feedback of the physical scene; Collect the user operation habit data, predict the user operation habit data based on the machine learning model to obtain the user behavior prediction result, obtain the scene state parameter according to the user behavior prediction result, synchronize the scene state parameter to the digital twin model, and record the user operation habit data.

2. The 3D scene user interaction method based on digital twin according to claim 1, wherein, The step of obtaining the physical scene data of the target scene, mapping the physical scene data into a virtual three-dimensional scene, and generating a digital twin model specifically includes: Use a sensor network to collect data on the target scene, obtain geometric structure data, physical property data, and real-time state data. The sensor network includes lidar, cameras, and temperature and humidity sensors. During the data collection process, combine the real-time state data with the geometric structure data to generate the time-series physical state data of the target scene; Generate a preliminary virtual three-dimensional scene model based on the geometric structure data and physical property data of the target scene, calibrate the preliminary virtual three-dimensional scene model to make the preliminary virtual three-dimensional scene model consistent with the geometric shape and physical state of the physical scene, obtain the virtual three-dimensional scene model, and update the parameter state in the virtual three-dimensional scene in real time according to the collected real-time state data.

3. The three-dimensional scene user interaction method based on digital twin according to claim 1, characterized in that The step of detecting the left mouse button pressing event of the user, obtaining the pressing signal, obtaining the obstacle avoidance movement path data according to the pressing signal and the terrain data of the virtual three-dimensional scene, synchronizing the obstacle avoidance movement path data to the digital twin model, and adjusting the movement path of the virtual three-dimensional scene according to the real-time feedback of the physical scene specifically includes: Detect the duration and pressing position of the user's left mouse button press event. When the duration exceeds a preset threshold, trigger the scene control interface to generate a pressing signal. In the virtual 3D scene, centered on the pressing position, generate four semi-transparent direction buttons for forward, backward, left, and right movement. The semi-transparent direction buttons are distributed in a cross shape, and the transparency of the semi-transparent direction buttons is dynamically adjusted according to the proximity of the mouse position. Obtain the user's movement control data on the scene control interface, including movement direction selection and movement distance data. Input the movement control data into the exponential smoothing algorithm for processing, obtain the scene movement speed parameter through adaptive weight calculation, combine the scene movement speed parameter and the terrain data of the virtual 3D scene, use the path planning algorithm for obstacle avoidance path planning, generate obstacle avoidance movement path data, synchronize the obstacle avoidance movement path data to the digital twin model in real time, and dynamically adjust the movement path based on the real-time state data of the physical scene.

4. A 3D scene user interaction method based on digital twin according to claim 1, characterized in that, Detect the user's right mouse button press event, obtain the rotation trigger signal, obtain the real-time scene rotation angle data based on the rotation trigger signal, synchronize the real-time scene rotation angle data to the digital twin model, and adjust the rotation state of the virtual 3D scene according to the real-time feedback of the physical scene. Specifically include: Use the camera and depth sensor to obtain the real-time mouse position information of the user. The real-time mouse position information includes the two-dimensional coordinates of the mouse on the screen and the depth position of the mouse in the three-dimensional space. According to the real-time mouse position information, calculate the direction vector of the mouse ray emitted from the camera position, and use the collision detection algorithm of the mouse ray and the 3D model to calculate the intersection coordinates of the mouse ray and the 3D model to obtain the collision point data. The collision detection algorithm uses an improved ray tracing method, and the calculation formula is: Among them, t is the ray distance parameter, and P0 is the starting point of the mouse ray. is the direction vector. is the normal vector of the patch of the 3D model. is a point on the patch of the 3D model, and P is the collision point coordinate. Apply the density-based spatial clustering algorithm to the collision point data to determine the rotation center coordinates, input the rotation center coordinates into the inertial rotation algorithm, and calculate the real-time scene rotation angle data corresponding to the user's rotation operation.

5. A three-dimensional scene user interaction method based on digital twin according to claim 1, characterized in that, Detect the user's mouse wheel operation, obtain the zoom operation signal, obtain the initial zoom ratio according to the zoom operation signal, collect the scene depth information, input the scene depth information and the initial zoom ratio into the depth perception algorithm to obtain the dynamic zoom ratio parameter, input the dynamic zoom ratio parameter into the view point optimization algorithm to obtain the target area focusing parameter, synchronize the dynamic zoom ratio parameter to the digital twin model, and adjust the zoom state of the virtual scene according to the real-time feedback of the physical scene. Specifically include: Use the roller sensor to detect the roller operation direction and rolling distance of the user to obtain the zoom operation signal; calculate the initial zoom ratio according to the rolling distance, and collect the scene depth map under the current view through the depth camera, perform noise reduction processing and depth calibration on the scene depth map to generate the scene depth information. Input the scene depth information into the depth perception algorithm. Based on the scene importance evaluation and depth level analysis, calculate the dynamic scaling ratio parameter. Use the viewpoint optimization algorithm to analyze the spatial distribution of the key areas of the scene. Combine the user's perspective and scaling operation to generate the target area focusing parameter.

6. The three-dimensional scene user interaction method based on digital twin according to claim 5, wherein, The calculation formula of the depth perception algorithm is: Among them, S(d) is the dynamic scaling ratio parameter, S0 is the initial scaling ratio, d is the actual depth distance from the current observation point to the scene object, d0 is the standard reference depth, λ is the depth attenuation coefficient, W(d) is the depth weight function, and d focus is the depth value at the position of the user's focus of attention, α is the weight attenuation rate, and β is the weight curve shape parameter; The calculation formula of the viewpoint optimization algorithm is: Among them, F(p) is the focusing parameter of the target area, w i is the weight value of the i-th key point, G(·) is the spatial Gaussian distribution function, G(p, p i ) is the spatial influence degree from position p to the key point p i , I(p i ) is the importance score of the i-th key point, n is the number of key points in the scene, v i is the visual saliency score of the i-th key point, v j is the visual saliency score of the j-th key point, C(d i ) is the depth compensation function, d i is the actual depth value of the i-th key point, d opt is the optimal depth value for observing the scene, and γ is the depth compensation coefficient.

7. The 3D scene user interaction method based on digital twin according to claim 1, wherein Collect the user operation habit data, predict the user operation habit data based on the machine learning model to obtain the user behavior prediction result. Obtain the scene state parameter according to the user behavior prediction result, synchronize the scene state parameter to the digital twin model, and record the user operation habit data. Specifically include: Collect the user operation habit data of the user in real time. The user operation habit data includes the movement path, rotation angle, scaling ratio, and operation timing information. Input the user operation habit data into the machine learning model for training and prediction to obtain the user behavior prediction result; Based on the user behavior prediction result, combine the scene state parameter and historical interaction data, use the collaborative filtering algorithm to calculate the user preference feature, generate the personalized interaction parameter for the current user, and apply the personalized interaction parameter to the scene interaction control.

8. A three-dimensional scene user interaction system based on digital twin, which is used to execute a three-dimensional scene user interaction method based on digital twin as described in any one of claims 1-7, characterized in that, The system includes: The digital twin construction module is used to obtain the physical scene data of the target scene, map the physical scene data to the virtual 3D scene, and generate the digital twin model; The obstacle avoidance movement path module is used to detect the user's left mouse button press event, obtain the press signal, obtain the obstacle avoidance movement path data according to the press signal and the terrain data of the virtual 3D scene, synchronize the obstacle avoidance movement path data to the digital twin model, and adjust the movement path of the virtual 3D scene according to the real-time feedback of the physical scene; The scene rotation angle module is used to detect the user's right mouse button press event, obtain the rotation trigger signal, obtain the real-time scene rotation angle data based on the rotation trigger signal, synchronize the real-time scene rotation angle data to the digital twin model, and adjust the rotation state of the virtual 3D scene according to the real-time feedback of the physical scene; The dynamic scaling and focusing module is used to detect the user's mouse wheel operation, obtain the scaling operation signal, obtain the initial scaling ratio according to the scaling operation signal, collect the scene depth information, input the scene depth information and the initial scaling ratio into the depth perception algorithm to obtain the dynamic scaling ratio parameter, input the dynamic scaling ratio parameter into the viewpoint optimization algorithm to obtain the target area focusing parameter, synchronize the dynamic scaling ratio parameter to the digital twin model, and adjust the scaling state of the virtual scene according to the real-time feedback of the physical scene; The user operation habit module is used to collect the user operation habit data, predict the user operation habit data based on the machine learning model to obtain the user behavior prediction result, obtain the scene state parameter according to the user behavior prediction result, synchronize the scene state parameter to the digital twin model, and record the user operation habit data.

9. An electronic device, characterized in that, Include: At least one processor, at least one memory, a communication interface, and a bus; Among them, the processor, the memory, and the communication interface complete communication with each other through the bus. The memory stores program instructions executable by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, and the computer instructions cause the computer to implement the method according to any one of claims 1 to 7.

Images