Virtual reality fusion method based on unmanned aerial vehicle video

Abstract

The invention discloses a virtual reality fusion method based on an unmanned aerial vehicle video, and relates to the technical field of virtual reality, and the method comprises the steps: predicting the current and next concern points of a user through eye movement tracking and Kalman filtering, preferentially distributing rendering resources to the areas, and employing a dynamic resource distribution mechanism to achieve the fusion of the current and next concern points of the user; according to the method, resources are allocated to a region about to be concerned by a user in advance, meanwhile, the processing priority of a secondary region is reduced, waste of computing resources is avoided, meanwhile, the moving track of a fixation point of the user is predicted through a Kalman filter, the next concern point of the user is rendered in advance, rendering of the concern region is completed when the visual angle is rapidly switched, and the user experience is improved. Compared with the prior art, the method has the advantages that image lagging and blurring felt by a user are reduced, rendering efficiency is greatly improved, user experience is smoother and more natural, a multi-level cache framework is adopted, the problem of loading delay in a traditional method is avoided, and the response speed of a system is effectively increased.

Description

Technical Field

[0001] The invention relates to the technical field of virtual reality, and in particular to a virtual reality fusion method based on unmanned aerial vehicle video. Background Art

[0002] In a virtual reality environment, different scene areas have different effects on user perception, and virtual reality mainly focuses on the user's immersive experience. For the user's immersive experience, the details in front of the eyes have a greater impact on the immersion, such as target objects, foreground elements, etc., while the distant view or secondary areas that are not of concern have less impact on the immersion. Therefore, how to effectively allocate computing resources to ensure high-quality rendering of key detail areas is an important direction for virtual reality software optimization.

[0003] In traditional virtual reality fusion methods, all areas of the scene are generally rendered with the same quality. Although this type of rendering method is simple, it will cause a waste of computing resources for highly complex scenes, such as drone videos of cities or natural environments that contain a large number of dynamic elements. The rendering of secondary areas takes up a lot of resources, and the rendering of key area details cannot be guaranteed. At the same time, the details cannot be adjusted dynamically according to the user's attention. In addition, the user's sense of immersion also requires a high degree of synchronization of tactile feedback and sound. When interacting with virtual objects, if the tactile feedback lags or does not match the vision, the user will obviously feel the illusion of "being out of reality", which greatly weakens the sense of immersion.

[0004] In response to the problem of scene rendering, some traditional methods introduce the concept of regional priority, which gives priority to rendering the area where the user is looking and reduces the details of other areas. However, when the user's perspective shifts quickly or interacts dynamically, this method cannot capture new focus points in time and quickly adjust the rendering details, resulting in a broken experience, a decrease in the overall consistency of the scene, and a loss of immersion. In terms of visual-tactile synchronization, traditional visual-tactile synchronization methods rely on a fixed-delay fusion strategy, that is, setting a fixed synchronization delay window between the visual and tactile devices to keep the feedback of the two consistent. Such methods perform well in static or low-speed interactions, but perform poorly in high-dynamic environments, such as fast-moving scenes in drone videos. When the drone suddenly accelerates or turns, the scene changes faster, and the fixed synchronization strategy cannot be adjusted in real time, resulting in tactile feedback being out of sync with the visual performance, with obvious lag or advance. Although some traditional solutions try to reduce lag by accelerating the response time of the tactile device or increasing the data transmission rate, this is still not enough to solve the fundamental problem. Therefore, a virtual reality fusion method based on drone videos is urgently needed to solve such problems. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a virtual reality fusion method based on drone video to solve the problem that in drone video scenes, when processing local scene details, traditional virtual reality fusion methods cannot adjust rendering details in time according to dynamic changes of the scene, resulting in waste of computing resources or missing details, and it is difficult to maintain consistency in the synchronization of vision and touch, which destroys the user's immersion and interactive experience.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The present invention provides a virtual reality fusion method based on drone video, which comprises:

[0009] Step S1, scene data mining,

[0010] Use convolutional neural networks to analyze drone videos, mark important areas and target objects in the videos, and pre-render them.

[0011] Eye tracking is used to locate the user's current focus point, that is, the current gaze area, and the next focus point is predicted based on the movement trajectory of the current focus point;

[0012] Here, important areas and target objects refer to the key content in the video, such as the subject captured by the drone (such as the tracked cat) and the environment in which it interacts (such as buildings, trees, etc.);

[0013] Step S2, dynamic resource allocation,

[0014] High-quality rendering is performed on the user's focus point, while reducing the processing priority of other areas. The focus points include the current focus point and the next focus point. Resources are allocated in advance based on the predicted next focus point.

[0015] Through dynamic resource allocation, efficient rendering of the focus point can be maintained in the drone video scene, and allocation is made based on the user's current visual focus and predicted visual focus. That is, resources are allocated in advance for the next view movement. Even if the user moves the view quickly, the details of the focus area can still be rendered in advance, avoiding delays or blurred images felt by users.

[0016] Step S3, synchronous adjustment of vision and touch,

[0017] The tactile feedback of objects in the focus area is differentiated, and the object materials are divided into hard, soft, smooth, and rough. When the user interacts with certain objects in the video, the tactile feedback intensity and mode of the interaction point are dynamically adjusted according to the user's interaction point and the object material.

[0018] Furthermore, the important areas and target objects in the video are marked in step S1 as follows:

[0019] Assume that the t-th frame in the video is represented by a matrix F(t), whose size is W×H, where W and H represent the width and height respectively, and the pixel values ​​of the three channels are represented by F rgb (t), scale normalization and color standardization:

[0020] in, is the normalized data matrix of the t-th frame, μ represents the mean of the pixel values ​​of the frame, σ represents the standard deviation of the pixel values ​​of the frame, both of which are the mean and standard deviation under the RGB three channels;

[0021] Use convolutional neural network to extract important features in each frame of image. Important features refer to important areas and target objects. Let F(t) be the input frame, then:

[0022] X l =f(W l *X l-1 +b l ), where X l is the output feature map of the lth layer, f is the Sigmoid activation function, W l is the convolution kernel weight matrix of the lth layer, * represents the convolution operation, X l-1 It is the input feature map of the previous layer, initially it is a video frame b l is the bias term of the lth layer;

[0023] The YOLO target detection model is used to detect the extracted important features and generate multiple bounding boxes to represent the location and area of ​​the target object. Assuming that there are N target objects in each frame F(t), the detection result is the bounding box B of each target. i and its confidence score S i :

[0024] B i =(x i ,y i ,w i ,h i ),

[0025] S i =P(c i |||F(t)), where B i represents the bounding box of the i-th target object, x i ,y i is the coordinate of the upper left corner of the bounding box, w i ,h i is the width and height, S i Represents the confidence score of the target object, defined as the target belonging to a certain category c i The probability P(c i |||F(t));

[0026] This step detects and localizes important features in each frame, and bounding boxes and confidence scores are used to determine the location and importance of objects, respectively.

[0027] Furthermore, in step S1, the important areas and target objects are pre-rendered:

[0028] Confidence score S based on target detection i , assign a priority to each target area and plan pre-rendering, priority P i Calculate based on the confidence of the target and the focus point G(t) of the user's line of sight:

[0029] P i =αS i +β·d(B i ,G(t)), where P i is the rendering priority of the i-th target area, α and β are weight coefficients, balancing the target confidence and distance weight, d(B i ,G(t)) represents the target bounding box B i The distance from the user's current focus point G(t) is calculated as:

[0030] Among them, (x g ,y g ) is the coordinate of the user's focus point G(t), (x i ,y i ) is the target bounding box B i The center coordinates of

[0031] This step integrates the object's confidence score and the user's gaze distance, so that the target objects near the focus point have a higher rendering priority;

[0032] Based on the priority P i , pre-render important areas and target objects, assuming that the rendering resources in the video frame are R and the total resource budget is R total , each target object is allocated rendering resources according to priority:

[0033] Among them, R i is the rendering resource allocated to the i-th target object, P i is the priority of the i-th target object, R total is the total available rendering resources, N is the total number of target objects detected in the current frame, and P j is the priority of the jth target object;

[0034] This allows more rendering resources to be allocated to important target objects and areas.

[0035] Furthermore, the next focus point is predicted in step S1 as follows:

[0036] Assume that the user's focus point at time t is G(t), and its position is represented by a two-dimensional coordinate (x t ,y t ), record the position of the focus point in the past k time steps, and form a time series G(tk), G(t-k+1),…, G(t), G(t) = (x t ,y t ), where G(t) is the user’s focus point at time t, x t and t are the horizontal and vertical coordinates of the focus point in the current frame respectively;

[0037] Calculate the sight speed and acceleration based on the change in the position of the focus point in adjacent time steps:

[0038] Where v(t) is the user's line of sight speed at time t, G(t)-G(t-1)=(x t -x t-1 ,y t -y t-1 ), represents the difference between the positions of two adjacent focus points, and Δt is the time interval between adjacent time steps;

[0039] Where a(t) is the user's line of sight acceleration at time t, and v(t-1) is the line of sight velocity at time t-1.

[0040] Furthermore, the method of predicting the next focus point in step S1 also includes:

[0041] Based on the current position, velocity and acceleration of the point of interest, the Kalman filter is used to predict the next point of interest, and the position, velocity and acceleration of the point of interest are constructed as a state vector X(t):

[0042] Among them, x t ,y t is the coordinate of the current focus point, v x (t),v y (t) are the horizontal and vertical line of sight speeds, respectively, a x (t),a y (t) are the lateral and longitudinal line of sight accelerations, respectively;

[0043] Predict the location of the focus point at the next moment t+1:

[0044] X(t+1)=FX(t)+w(t), where X(t+1) is the state vector at time t+1, including the predicted position, velocity, and acceleration of the point of interest, and F is the state transfer matrix, which represents the state change of the system from time t to t+1: w(t) is the process noise vector;

[0045] Assume that the actual observed point of interest is Z(t+1), and use the Kalman gain K(t+1) to update the predicted value:

[0046] X(t+1)=X(t+1) pred +K(t+1)(Z(t+1)-HX(t+1) pred ),

[0047] Among them, X(t+1) pred is the predicted value of the Kalman filter, Z(t+1) is the actual observation value, i.e. the position of the user’s focus point at time t+1, and H is the observation matrix;

[0048] The Kalman gain K(t+1) is calculated as:

[0049] K(t+1)=P(t+1)H T (HP(t+1)H T +R) -1 ,

[0050] Where P(t+1) is the predicted state covariance matrix, R is the observed noise covariance matrix, and T represents the transposition operation;

[0051] Through the prediction and update process of the Kalman filter, we can get the next point of concern.

[0052] Furthermore, a multi-level cache architecture is used in the dynamic resource allocation in step S2:

[0053] The data of the focus area is placed in the fast response cache, and the data of other areas (such as background scenes and non-interactive areas) are placed in the low-priority cache layer; this improves the response speed. In high-speed flight scenes, the focus area can be quickly transmitted and rendered to avoid loading delays when the user moves the view quickly.

[0054] Furthermore, the interaction point refers to a device that can deliver tactile feedback to the user and can be synchronized with the vision (such as a vibration handle). The tactile feedback and visual feedback are highly synchronized, avoiding the feedback lag problem in traditional solutions.

[0055] Furthermore, the synchronous adjustment method of vision and touch in step S3 is:

[0056] Classify and model the materials of objects in each focus area, and set the material model of each object as Mi , whose tactile feedback characteristic is M i ={E i ,H i ,R i}, where E i represents the elastic modulus of object i, H i represents the surface hardness of object i, R i represents the roughness of object i;

[0057] When a user interacts with an object, the tactile intensity is adjusted through force feedback at the interaction point. Assuming that the user interacts with object i at position p(t) at time t, the contact force F(t) at the interaction point is expressed as follows according to the elasticity and hardness of the object: Where F(t) represents the force generated when the user touches the object at time t, A represents the proportional constant used to adjust the amplitude of the feedback, and E i represents the elastic modulus of object i, indicating the degree of response of the object to pressure, Δx(t) represents the deformation of the object at time t, and H i Indicates the hardness parameter of an object;

[0058] The feedback pattern of the tactile device depends on the roughness R of the object. i The interactive contact force F(t) is dynamically adjusted, and the vibration frequency f(t) and vibration intensity V(t) are calculated as follows:

[0059]

[0060] V(t)=C·F(t)·R i , where f(t) is the vibration frequency at time t. The higher the roughness, the higher the R i The smaller the value, the higher the frequency. B is the frequency proportional constant associated with the tactile device. V(t) is the vibration intensity at time t. The feedback intensity is related to the contact force F(t) and the roughness R of the object surface. i Proportional to, C is the vibration intensity proportionality constant associated with the tactile device;

[0061] This simulates the situation where the tactile device adjusts its feedback strength and mode according to the physical properties and interaction force of the object's surface when it comes into contact with objects of different materials.

[0062] Furthermore, during the synchronous adjustment of vision and touch in step S3:

[0063] Calculate the time step Δt of visual update visual and the time step Δt of tactile feedback haptic , and keep the delay between them within the allowed range:

[0064] Δt=|Δt visual -Δthaptic |≤∈, where Δt visual represents the visual feedback frame rate, Δt haptic represents the tactile feedback update frequency, Δt is the time difference between vision and tactile, and ∈ is the maximum allowed delay;

[0065] By setting a maximum delay, tactile feedback is synchronized with visual feedback to avoid perceptual inconsistencies and enhance user immersion.

[0066] Furthermore, other areas refer to areas other than the focus point.

[0067] The beneficial effects of the present invention are:

[0068] The present invention predicts the user's current and next focus points through eye tracking and Kalman filtering, and allocates rendering resources to these areas preferentially. It utilizes a dynamic resource allocation mechanism to pre-allocate resources to the areas that the user is about to focus on, while reducing the processing priority of secondary areas, thereby avoiding a waste of computing resources.

[0069] The present invention predicts the movement trajectory of the user's gaze point through a Kalman filter and renders the user's next focus point in advance. When the perspective switches quickly, the rendering of the focus area is completed, reducing the image lag and blur felt by the user. This greatly improves the rendering efficiency and makes the user experience more smooth and natural, especially in high-speed dynamic environments (such as drone fast flight scenes). The effect is particularly significant.

[0070] The present invention adopts a multi-level cache architecture to place the user's current focus point and the predicted next focus point data in a fast response cache, while the data of the secondary area is placed in a low-priority cache layer. It can quickly respond and complete the rendering of the focus area in high-speed mobile scenarios, avoiding the loading delay problem in traditional methods and effectively improving the response speed of the system.

[0071] The present invention dynamically adjusts the intensity and mode of tactile feedback by modeling the material of the object (such as hardness, elasticity, and roughness) and combining it with the force feedback of the user interaction point. The tactile feedback of the object is highly consistent with its physical properties. The user feels stronger feedback when touching a hard object and produces a higher vibration frequency when touching a rough object. The authenticity of the tactile feedback is improved, and the tactile and visual feedback are highly synchronized, avoiding the inconsistency between the tactile and visual caused by fixed delays in traditional methods, thereby enhancing the overall immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0073] Figure 1 The figure is a flow chart of the virtual reality fusion method based on drone video of the present invention. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0075] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0076] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0077] Example 1, reference Figure 1 , this embodiment provides a virtual reality fusion method based on drone video, comprising the following steps:

[0078] Step S1, scene data mining,

[0079] Use convolutional neural networks to analyze drone videos, mark important areas and target objects in the videos, and pre-render them.

[0080] Eye tracking is used to locate the user's current focus point, that is, the current gaze area, and the next focus point is predicted based on the movement trajectory of the current focus point;

[0081] Here, important areas and target objects refer to the key content in the video, such as the subject captured by the drone (such as the tracked cat) and the environment in which it interacts (such as buildings, trees, etc.);

[0082] The method of marking important areas and target objects in the video in step S1 is:

[0083] Assume that the t-th frame in the video is represented by a matrix F(t), whose size is W×H, where W and H represent the width and height respectively, and the pixel values ​​of the three channels are represented by F rgb (t), scale normalization and color standardization:

[0084] in, is the normalized data matrix of the t-th frame, μ represents the mean of the pixel values ​​of the frame, σ represents the standard deviation of the pixel values ​​of the frame, both of which are the mean and standard deviation under the RGB three channels;

[0085] Use convolutional neural network to extract important features in each frame of image. Important features refer to important areas and target objects. Let F(t) be the input frame, then:

[0086] X l =f(W l *X l-1 +b l ), where X l is the output feature map of the lth layer, f is the Sigmoid activation function, W l is the convolution kernel weight matrix of the lth layer, * represents the convolution operation, X l-1 It is the input feature map of the previous layer, initially it is a video frame b l is the bias term of the lth layer;

[0087] The YOLO target detection model is used to detect the extracted important features and generate multiple bounding boxes to represent the location and area of ​​the target object. Assuming that there are N target objects in each frame F(t), the detection result is the bounding box B of each target. i and its confidence score S i :

[0088] B i =(x i ,y i ,w i ,h i ),

[0089] S i =P(c i |||F(t)), where B i represents the bounding box of the i-th target object, x i ,y i is the coordinate of the upper left corner of the bounding box, w i ,h i is the width and height, S i Represents the confidence score of the target object, defined as the target belonging to a certain category c i The probability P(c i |||F(t));

[0090] This step detects and calibrates important features in each frame. Bounding boxes and confidence scores are used to determine the location and importance of objects, respectively;

[0091] In step S1, important areas and target objects are pre-rendered:

[0092] Confidence score S based on target detection i , assign a priority to each target area and plan pre-rendering, priority P i Calculate based on the confidence of the target and the focus point G(t) of the user's line of sight:

[0093] P i =αS i +β·d(B i ,G(t)), where P i is the rendering priority of the i-th target area, α and β are weight coefficients, balancing the target confidence and distance weight, d(B i ,G(t)) represents the target bounding box B i The distance from the user's current focus point G(t) is calculated as:

[0094] Among them, (x g ,y g ) is the coordinate of the user's focus point G(t), (x i ,y i ) is the target bounding box B i The center coordinates of

[0095] This step integrates the object's confidence score and the user's gaze distance, so that the target objects near the focus point have a higher rendering priority;

[0096] Based on the priority P i , pre-render important areas and target objects, assuming that the rendering resources in the video frame are R and the total resource budget is R total , each target object is allocated rendering resources according to priority:

[0097] Among them, R i is the rendering resource allocated to the i-th target object, P i is the priority of the i-th target object, R total is the total available rendering resources, N is the total number of target objects detected in the current frame, and P j is the priority of the jth target object;

[0098] Thus more rendering resources can be allocated to important target objects and areas;

[0099] Specifically, in terms of scene data mining, convolutional neural networks are used to extract important features from drone videos, and key areas and target objects are identified through target detection models. Unlike traditional methods, this method not only calibrates the position of objects, but also integrates the importance of objects and the user's gaze area. Based on the object confidence score and the user's current visual focus, dynamic rendering priority is assigned to each area. Important objects near the focus point will obtain more rendering resources, so that the part that the user is concerned about has higher rendering accuracy and real-time performance.

[0100] The method for predicting the next focus point in step S1 is:

[0101] Assume that the user's focus point at time t is G(t), and its position is represented by a two-dimensional coordinate (x t ,y t ), record the position of the focus point in the past k time steps, and form a time series G(tk), G(t-k+1),…, G(t), G(t) = (x t ,y t ), where G(t) is the user’s focus point at time t, x t and t are the horizontal and vertical coordinates of the focus point in the current frame respectively;

[0102] Calculate the sight speed and acceleration based on the change in the position of the focus point in adjacent time steps:

[0103] Where v(t) is the user's line of sight speed at time t, G(t)-G(t-1)=(x t -x t-1 ,y t -y t-1 ), represents the difference between the positions of two adjacent focus points, and Δt is the time interval between adjacent time steps;

[0104] Where a(t) is the user's line of sight acceleration at time t, and v(t-1) is the line of sight velocity at time t-1;

[0105] The method of predicting the next focus point in step S1 also includes:

[0106] Based on the current position, velocity and acceleration of the point of interest, the Kalman filter is used to predict the next point of interest, and the position, velocity and acceleration of the point of interest are constructed as a state vector X(t):

[0107] Among them, x t ,y t is the coordinate of the current focus point, v x(t),v y (t) are the horizontal and vertical line of sight speeds, respectively, a x (t),a y (t) are the lateral and longitudinal line of sight accelerations, respectively;

[0108] Predict the location of the focus point at the next moment t+1:

[0109] X(t+1)=FX(t)+w(t), where X(t+1) is the state vector at time t+1, including the predicted position, velocity, and acceleration of the point of interest, and F is the state transfer matrix, which represents the state change of the system from time t to t+1: w(t) is the process noise vector;

[0110] Assume that the actual observed point of interest is Z(t+1), and use the Kalman gain K(t+1) to update the predicted value:

[0111] X(t+1)=X(t+1) pred +K(t+1)(Z(t+1)-HX(t+1) pred ),

[0112] Among them, X(t+1) pred is the predicted value of the Kalman filter, Z(t+1) is the actual observation value, i.e. the position of the user’s focus point at time t+1, and H is the observation matrix;

[0113] The Kalman gain K(t+1) is calculated as:

[0114] K(t+1)=P(t+1)H T (HP(t+1)H T +R) -1 ,

[0115] Where P(t+1) is the predicted state covariance matrix, R is the observed noise covariance matrix, and T represents the transposition operation;

[0116] Through the prediction and update process of the Kalman filter, the next point of interest is obtained;

[0117] Specifically, a dynamic resource allocation mechanism is introduced to monitor the user's gaze position in real time through eye tracking technology, and the Kalman filter model is used to predict the user's next gaze point. By analyzing and predicting the movement of the user's gaze in advance, rendering resources are allocated in advance to the areas that may be gazed at. Even in the case of rapid perspective switching, it can ensure that the user's visual focus area maintains high-definition rendering, avoid the occurrence of blurred or delayed images, and avoid excessive rendering of secondary areas, thereby reducing the waste of computing resources.

[0118] Step S2, dynamic resource allocation,

[0119] High-quality rendering is performed on the user's focus point, while reducing the processing priority of other areas. The focus points include the current focus point and the next focus point. Resources are allocated in advance based on the predicted next focus point.

[0120] Through dynamic resource allocation, efficient rendering of the focus point can be maintained in the drone video scene, and allocation is made based on the user's current visual focus and predicted visual focus. That is, resources are allocated in advance for the next view movement. Even if the user moves the view quickly, the details of the focus area can still be rendered in advance, avoiding delays or blurred images felt by users.

[0121] Other areas refer to areas outside the focus area;

[0122] The dynamic resource allocation in step S2 adopts a multi-level cache architecture:

[0123] The data of the focus area is placed in the fast response cache, and the data of other areas (such as background scenes and non-interactive areas) are placed in the low-priority cache layer; this improves the response speed. In high-speed flight scenes, the focus area can be quickly transmitted and rendered to avoid loading delays when the user moves the view quickly;

[0124] Specifically, in terms of resource management, this method adopts a multi-level cache architecture, storing data in key areas in a fast response cache, and placing data in other secondary areas in a low-priority cache layer. The layered cache design improves response speed and can quickly render the user's current and predicted focus areas, avoiding loading delays when the perspective moves quickly.

[0125] Step S3, synchronous adjustment of vision and touch,

[0126] Differentiate the tactile feedback of objects in the focus area and classify the object materials into hard, soft, smooth, and rough. When the user interacts with certain objects in the video, the tactile feedback intensity and mode of the interaction point are dynamically adjusted according to the user's interaction point and the object material.

[0127] Interaction points refer to devices that can deliver tactile feedback to users and can be synchronized with vision (such as vibrating controllers);

[0128] The tactile feedback is highly synchronized with the visual feedback, avoiding the feedback lag problem in traditional solutions.

[0129] The synchronous adjustment method of vision and touch in step S3 is:

[0130] Classify and model the materials of objects in each focus area, and set the material model of each object as M i , whose tactile feedback characteristic is Mi ={E i ,H i ,R i}, where E i represents the elastic modulus of object i, H i represents the surface hardness of object i, R i represents the roughness of object i;

[0131] When a user interacts with an object, the tactile intensity is adjusted through force feedback at the interaction point. Assuming that the user interacts with object i at position p(t) at time t, the contact force F(t) at the interaction point is expressed as follows according to the elasticity and hardness of the object: Where F(t) represents the force generated when the user touches the object at time t, A represents the proportional constant used to adjust the amplitude of the feedback, and E i represents the elastic modulus of object i, indicating the degree of response of the object to pressure, Δx(t) represents the deformation of the object at time t, and H i Indicates the hardness parameter of an object;

[0132] The feedback pattern of the tactile device depends on the roughness R of the object. i The interactive contact force F(t) is dynamically adjusted, and the vibration frequency f(t) and vibration intensity V(t) are calculated as follows:

[0133]

[0134] V(t)=C·F(t)·R i ,in,

[0135] f(t) is the vibration frequency at time t. The higher the roughness, the i The smaller the value, the higher the frequency. B is the frequency proportional constant associated with the tactile device. V(t) is the vibration intensity at time t. The feedback intensity is related to the contact force F(t) and the roughness R of the object surface. i Proportional to, C is the vibration intensity proportionality constant associated with the tactile device;

[0136] This simulates the situation when the haptic device adjusts its feedback strength and mode according to the physical properties and interaction force of the object's surface when in contact with objects of different materials.

[0137] During the synchronous adjustment of vision and touch in step S3:

[0138] Calculate the time step Δt of visual update visual and the time step Δt of tactile feedback haptic , and keep the delay between them within the allowed range:

[0139] Δt=|Δt visual -Δt haptic|≤∈, where Δt visual represents the visual feedback frame rate, Δt haptic represents the tactile feedback update frequency, Δt is the time difference between vision and tactile, and ∈ is the maximum allowed delay;

[0140] Synchronize tactile feedback with visual feedback by setting a maximum delay to avoid perceptual inconsistencies and enhance user immersion.

[0141] Specifically, when users interact with virtual objects, the intensity and pattern of tactile feedback are dynamically adjusted according to the material properties of the object (such as elasticity, hardness, and roughness). Hard objects will produce stronger tactile feedback, while rough objects will bring higher vibration frequencies. By calculating the contact force between the user and the object, a more realistic tactile experience is simulated. At the same time, a high degree of synchronization between visual feedback and tactile feedback is guaranteed, avoiding perceptual distortion caused by delays or mismatches, and enhancing the user's sense of immersion in the virtual reality environment.

[0142] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A virtual reality fusion method based on drone video, characterized by: include, Step S1, scene data mining, uses convolutional neural networks to analyze drone videos, mark important areas and target objects in the videos, and pre-render them. Eye tracking is used to locate the user's current focus point, that is, the current gaze area, and the next focus point is predicted based on the movement trajectory of the current focus point; Step S2, dynamic resource allocation, high-quality rendering of the user's focus point, while reducing the processing priority of other areas, the focus point includes the current focus point and the next focus point, and resources are allocated in advance based on the predicted next focus point; Step S3, synchronous adjustment of vision and touch, differentiated processing of tactile feedback for objects in the focus area, and classification of object materials into hard, soft, smooth, and rough. When the user interacts with certain objects in the video, the tactile feedback intensity and mode of the interaction point are dynamically adjusted according to the user's interaction point and the object material.

2. The virtual reality fusion method based on drone video according to claim 1 is characterized in that: The method of marking important areas and target objects in the video in step S1 is: Assume that the t-th frame in the video is represented by a matrix F(t), whose size is W×H, where W and H represent the width and height respectively, and the pixel values ​​of the three channels are represented by F rgb (t), scale normalization and color standardization: in, is the normalized data matrix of the t-th frame, μ represents the mean of the pixel values ​​of the frame, σ represents the standard deviation of the pixel values ​​of the frame, both of which are the mean and standard deviation under the RGB three channels; Use convolutional neural network to extract important features in each frame of image. Important features refer to important areas and target objects. Let F(t) be the input frame, then: X l =f(W l *X l-1 +b l ), where X l is the output feature map of the lth layer, f is the Sigmoid activation function, W l is the convolution kernel weight matrix of the lth layer, * represents the convolution operation, X l-1 It is the input feature map of the previous layer, initially it is a video frame b l is the bias term of the lth layer; The YOLO target detection model is used to detect the extracted important features and generate multiple bounding boxes to represent the location and area of ​​the target object. Assuming that there are N target objects in each frame F(t), the detection result is the bounding box B of each target. i and its confidence score S i : B i =(x i ,y i ,w i ,h i ), S i =P(c i |||F(t)), where B i represents the bounding box of the i-th target object, x i ,y i is the coordinate of the upper left corner of the bounding box, w i ,h i is the width and height, S i Represents the confidence score of the target object, defined as the target belonging to a certain category c i The probability P(c i ∣∣F(t)).

3. The virtual reality fusion method based on drone video according to claim 2 is characterized in that: In step S1, important areas and target objects are pre-rendered: Confidence score S based on target detection i , assign a priority to each target area and plan pre-rendering, priority P i Calculate based on the confidence of the target and the focus point G(t) of the user's line of sight: P i =αS i +β·d(B i ,G(t)), where P i is the rendering priority of the i-th target area, α and β are weight coefficients, balancing the target confidence and distance weight, d(B i ,G(t)) represents the target bounding box B i The distance from the user's current focus point G(t) is calculated as: Among them, (x g ,y g ) is the coordinate of the user's focus point G(t), (x i ,y i ) is the target bounding box B i The center coordinates of Based on the priority P i , pre-render important areas and target objects, assuming that the rendering resources in the video frame are R and the total resource budget is R total , each target object is allocated rendering resources according to priority: Among them, R i is the rendering resource allocated to the i-th target object, P i is the priority of the i-th target object, R total is the total available rendering resources, N is the total number of target objects detected in the current frame, and P j is the priority of the jth target object.

4. The virtual reality fusion method based on drone video according to claim 3 is characterized in that: The method for predicting the next focus point in step S1 is: Assume that the user's focus point at time t is G(t), and its position is represented by a two-dimensional coordinate (x t ,y t ), record the position of the focus point in the past k time steps, and form a time series G(tk), G(t-k+1),…, G(t), G(t) = (x t ,y t ), where G(t) is the user’s focus point at time t, x t and t are the horizontal and vertical coordinates of the focus point in the current frame respectively; Calculate the sight speed and acceleration based on the change in the position of the focus point in adjacent time steps: Where v(t) is the user's line of sight speed at time t, G(t)-G(t-1)=(x t -x t-1 ,y t -y t-1 ), represents the difference between the positions of two adjacent focus points, and Δt is the time interval between adjacent time steps; Where a(t) is the user's line of sight acceleration at time t, and v(t-1) is the line of sight velocity at time t-1.

5. The virtual reality fusion method based on drone video according to claim 4 is characterized in that: The method of predicting the next focus point in step S1 also includes: Based on the current position, velocity and acceleration of the point of interest, the Kalman filter is used to predict the next point of interest, and the position, velocity and acceleration of the point of interest are constructed as a state vector X(t): Among them, x t ,y t is the coordinate of the current focus point, v x (t),v y (t) are the horizontal and vertical line of sight speeds, respectively, a x (t),a y (t) are the lateral and longitudinal line of sight accelerations, respectively; Predict the location of the focus point at the next moment t+1: X(t+1)=FX(t)+w(t), where X(t+1) is the state vector at time t+1, including the predicted position, velocity, and acceleration of the point of interest, and F is the state transfer matrix, which represents the state change of the system from time t to t+1: w(t) is the process noise vector; Assume that the actual observed point of interest is Z(t+1), and use the Kalman gain K(t+1) to update the predicted value: X(t+1)=X(t+1) pred +K(t+1)(Z(t+1)-HX(t+1) pred ), Among them, X(t+1) pred is the predicted value of the Kalman filter, Z(t+1) is the actual observation value, i.e. the position of the user’s focus point at time t+1, and H is the observation matrix; The Kalman gain K(t+1) is calculated as: K(t+1)=P(t+1)H T (HP(t+1)H T +R) -1 , Where P(t+1) is the predicted state covariance matrix, R is the observation noise covariance matrix, and T represents the transpose operation.

6. The virtual reality fusion method based on drone video according to claim 5 is characterized in that: The dynamic resource allocation in step S2 adopts a multi-level cache architecture: Place the focus area data in the fast response cache and other area data in the low priority cache layer.

7. The virtual reality fusion method based on drone video according to claim 6 is characterized in that: Interaction points refer to devices that can deliver tactile feedback to users and can be synchronized with vision.

8. The virtual reality fusion method based on drone video according to claim 7 is characterized in that: The synchronous adjustment method of vision and touch in step S3 is: Classify and model the materials of objects in each focus area, and set the material model of each object as M i , whose tactile feedback characteristic is M i ={E i ,H i ,R i }, where E i represents the elastic modulus of object i, H i represents the surface hardness of object i, R i represents the roughness of object i; When a user interacts with an object, the tactile intensity is adjusted through force feedback at the interaction point. Assuming that the user interacts with object i at position p(t) at time t, the contact force F(t) at the interaction point is expressed as follows according to the elasticity and hardness of the object: Where F(t) represents the force generated when the user touches the object at time t, A represents the proportional constant used to adjust the amplitude of the feedback, and E i represents the elastic modulus of object i, indicating the degree of response of the object to pressure, Δx(t) represents the deformation of the object at time t, and H i Indicates the hardness parameter of an object; The feedback pattern of the tactile device depends on the roughness R of the object. i The interactive contact force F(t) is dynamically adjusted, and the vibration frequency f(t) and vibration intensity V(t) are calculated as follows: V(t)=C·F(t)·R i , where f(t) is the vibration frequency at time t. The higher the roughness, the higher the R i The smaller the value, the higher the frequency. B is the frequency proportional constant associated with the tactile device. V(t) is the vibration intensity at time t. The feedback intensity is related to the contact force F(t) and the roughness R of the object surface. i is proportional to, and C is the vibration intensity proportionality constant associated with the tactile device.

9. The virtual reality fusion method based on drone video according to claim 8, characterized in that: During the synchronous adjustment of vision and touch in step S3: Calculate the time step Δt of visual update visual and the time step Δt of tactile feedback haptic , and keep the delay between them within the allowed range: Δt=|Δt visual -Δt haptic |≤∈, where Δt visual represents the visual feedback frame rate, Δt haptic represents the haptic feedback update frequency, Δt is the time difference between vision and haptic, and ∈ is the maximum allowed delay.

10. The virtual reality fusion method based on drone video according to claim 9, characterized in that: Other areas refer to areas outside the focus area.

Images