A virtual reality fusion method based on drone video

Abstract

The present invention discloses a virtual reality fusion method based on drone video, which relates to the field of virtual reality technology. The present invention predicts the user's current and next focus points through eye tracking and Kalman filtering, and preferentially allocates rendering resources to these areas. A dynamic resource allocation mechanism is used to pre-allocate resources to areas that the user is about to focus on, while reducing the processing priority of secondary areas, thereby avoiding waste of computing resources. At the same time, the movement trajectory of the user's gaze point is predicted through the Kalman filter, and the user's next focus point is rendered in advance. When the perspective is quickly switched, the rendering of the focus area is completed, reducing the image lag and blurring felt by the user, thereby greatly improving the rendering efficiency and making the user experience more smooth and natural. The multi-level cache architecture is adopted to avoid the loading delay problem in traditional methods and effectively improve the response speed of the system.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a virtual reality fusion method based on drone video. Background Art

[0002] In a virtual reality environment, different scene areas have different impacts on the user's perception. Virtual reality mainly focuses on the user's immersive experience. For the user's immersive experience, the details in front of the eyes, such as target objects and foreground elements, have a greater impact on the immersion, while the distant view or secondary areas that are not of concern have less impact on the immersion. Therefore, how to effectively allocate computing resources to ensure high-quality rendering of key detail areas is an important direction for virtual reality software optimization.

[0003] In traditional virtual reality fusion methods, all areas of the scene are generally rendered with the same quality. Although this rendering method is simple, it will result in a waste of computing resources for highly complex scenes, such as drone videos of cities or natural environments with a large number of dynamic elements. The rendering of secondary areas takes up a lot of resources, and the rendering of key areas cannot guarantee the details. At the same time, the details cannot be adjusted dynamically according to the user's attention. In addition, the user's sense of immersion also requires a high degree of synchronization of tactile feedback and sound. When interacting with virtual objects, if the tactile feedback lags or does not match the vision, the user will clearly feel the illusion of being "out of reality", which greatly weakens the sense of immersion.

[0004] To address the issue of scene rendering, some traditional methods have introduced the concept of regional priority, prioritizing rendering of the area the user is focusing on while reducing detail in other areas. However, when the user's perspective shifts rapidly or during dynamic interactions, this method cannot promptly capture the new focus and quickly adjust rendering details, resulting in a broken experience, a decrease in overall scene consistency, and a poor sense of immersion. Regarding visual-tactile synchronization, traditional visual-tactile synchronization methods rely on a fixed-delay fusion strategy, which sets a fixed synchronization delay window between the visual and tactile devices to keep their feedback consistent. Such methods perform well in static or low-speed interactions, but perform poorly in highly dynamic environments, such as the fast-moving scenes in drone videos. When a drone suddenly accelerates or turns, the scene changes faster, and the fixed synchronization strategy cannot adjust in real time, causing tactile feedback to be out of sync with the visual performance, resulting in noticeable lag or advance. Although some traditional solutions attempt to reduce lag by accelerating the response time of the tactile device or increasing the data transmission rate, this is still not enough to solve the fundamental problem. Therefore, a virtual reality fusion method based on drone videos is urgently needed to address this issue. Summary of the Invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a virtual reality fusion method based on drone video to solve the problem that in drone video scenes, traditional virtual reality fusion methods cannot adjust rendering details in time according to dynamic changes in the scene when processing local scene details, resulting in waste of computing resources or missing details, and it is difficult to maintain consistency in the synchronization of vision and touch, which destroys the user's immersion and interactive experience.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] The present invention provides a virtual reality fusion method based on drone video, which includes:

[0009] Step S1, scene data mining,

[0010] Use convolutional neural networks to analyze drone videos, mark target objects in the videos, and pre-render them.

[0011] Eye tracking is used to locate the user's current focus point, that is, the current gaze area, and predict the next focus point based on the movement trajectory of the current focus point;

[0012] The target object here refers to the key content in the video, such as the subject captured by the drone (such as the tracked cat) and the environment in which it interacts (such as buildings, trees, etc.);

[0013] Step S2, dynamic resource allocation,

[0014] Provide high-quality rendering for the user's focus point while reducing the processing priority of other areas. The focus points include the current focus point and the next focus point. Resources are allocated in advance based on the predicted next focus point.

[0015] Dynamic resource allocation enables efficient rendering of points of interest in drone video scenes. This allocation is based on the user's current and predicted visual focus. This means resources are allocated in advance of the next view movement. Even when the user's view is rapidly shifting, detailed rendering of the desired area is still completed in advance, preventing users from experiencing delays or blurry images.

[0016] Step S3, synchronous adjustment of vision and touch,

[0017] Differentiated tactile feedback is performed on objects within the focus area, and the object materials are classified into hard, soft, smooth, and rough. When the user interacts with certain objects in the video, the tactile feedback intensity and mode of the interaction point are dynamically adjusted based on the user's interaction point and the object material.

[0018] Furthermore, the target object in the video is marked in step S1 as follows:

[0019] Assume that the t-th frame in the video is represented by a matrix F(t), whose size is W×H, where W and H represent width and height respectively, and the pixel values ​​of the three channels are represented by F rgb (t), scale normalization and color standardization:

[0020] in, is the normalized data matrix of the t-th frame, μ represents the mean of the pixel values ​​of the frame, σ represents the standard deviation of the pixel values ​​of the frame, both of which are the mean and standard deviation of the RGB three channels;

[0021] Use convolutional neural networks to extract important features from each frame of the image. The important features refer to the target object and its bounding box. Let F(t) be the input frame, then:

[0022] X l =f(W l *X l-1 +b l ), where X l is the output feature map of the lth layer, f is the Sigmoid activation function, W l is the convolution kernel weight matrix of the lth layer, * represents the convolution operation, X l-1 It is the input feature map of the previous layer, initially it is a video frame b l is the bias term of the lth layer;

[0023] The YOLO target detection model is used to detect the extracted important features and generate multiple bounding boxes to represent the location and area of ​​the target object. Assuming that there are N target objects in each frame F(t), the detection result is the bounding box B of each target. i and its confidence score S i :

[0024] B i =(x i ,y i ,w i ,h i ),

[0025] S i =P(c i |F(t)), where B i represents the bounding box of the i-th target object, x i ,y i is the center coordinate of the bounding box, w i ,h i is the width and height, S i Represents the confidence score of the target object, defined as the target belongs to a certain category c i The probability P(c i |F(t));

[0026] This step detects and localizes important features in each frame, and the bounding box and confidence score are used to determine the location and importance of the object, respectively.

[0027] Furthermore, in step S1, the target object is pre-rendered:

[0028] Confidence score S based on target detection i , assign priority to each target object area and perform pre-rendering planning, priority P i Calculate based on the confidence of the target and the focus point G(t) of the user's sight:

[0029] P i =αS i +β·d(B i ,G(t)), where P i is the rendering priority of the i-th target object, α and β are weight coefficients, balancing the target confidence and distance weight, d(B i ,G(t)) represents the target bounding box B i The distance from the user's current focus point G(t) is calculated as:

[0030] Among them, (x g ,y g ) is the coordinate of the user's focus point G(t), (x i ,y i ) is the target bounding box B i The center coordinates of

[0031] This step integrates the object's confidence score and the user's gaze distance, giving target objects near the focus point a higher rendering priority;

[0032] Based on the priority P i , pre-render the target object, set the rendering resource in the video frame to R, and the total resource budget to R total , each target object is allocated rendering resources according to priority:

[0033] Among them, R i is the rendering resource allocated to the i-th target object, P i is the priority of the i-th target object, R total is the total available rendering resources, N is the total number of target objects detected in the current frame, and P j is the priority of the j-th target object;

[0034] This allows more rendering resources to be allocated to important target objects and areas.

[0035] Furthermore, the next focus point is predicted in step S1 as follows:

[0036] Assume that the user's focus point at time t is G(t), and its position is represented by two-dimensional coordinates (x t ,y t ), record the position of the focus point in the past k time steps, and form a time series G(tk), G(t-k+1),…, G(t), G(t)=(x t ,y t ), where G(t) is the user’s focus point at time t, x t and y t are the horizontal and vertical coordinates of the focus point in the current frame respectively;

[0037] Calculate the sight speed and acceleration based on the change in the position of the focus point in adjacent time steps:

[0038] Where v(t) is the user’s line of sight velocity at time t, G(t)-G(t-1)=(x t -x t-1 ,y t -y t-1 ), represents the difference between the positions of two adjacent focus points, and Δt is the time interval between adjacent time steps;

[0039] Where a(t) is the user's line of sight acceleration at time t, and v(t-1) is the line of sight velocity at time t-1.

[0040] Furthermore, the method of predicting the next focus point in step S1 also includes:

[0041] Based on the current position, velocity, and acceleration of the point of interest, a Kalman filter is used to predict the next point of interest, and the position, velocity, and acceleration of the point of interest are constructed as a state vector X(t):

[0042] Among them, x t ,y t is the coordinate of the current focus point, v x (t),v y (t) are the horizontal and vertical line of sight speeds, respectively, a x (t),a y (t) are the lateral and longitudinal line of sight accelerations, respectively;

[0043] Predict the location of the focus point at the next moment t+1:

[0044] X(t+1)=FX(t)+w(t), where X(t+1) is the state vector at time t+1, which contains the predicted position, velocity, and acceleration of the point of interest. F is the state transition matrix, which represents the state change of the system from time t to t+1: w(t) is the process noise vector;

[0045] Assume that the actual observed point of interest is Z(t+1), and the Kalman gain K(t+1) is used to update the predicted value:

[0046] X(t+1)=X(t+1) pred +K(t+1)(Z(t+1)-HX(t+1) pred ),

[0047] Among them, X(t+1) pred is the predicted value of the Kalman filter, Z(t+1) is the actual observation value, that is, the position of the user's attention point at time t+1, and H is the observation matrix;

[0048] The Kalman gain K(t+1) is calculated as:

[0049] K(t+1)=P(t+1)H T (HP(t+1)H T +R) -1 ,

[0050] Where P(t+1) is the predicted state covariance matrix, R is the observation noise covariance matrix, and T represents the transpose operation;

[0051] Through the prediction and update process of the Kalman filter, the next point of interest is obtained.

[0052] Furthermore, a multi-level cache architecture is used in the dynamic resource allocation in step S2:

[0053] The data of the focus area is placed in the fast response cache, and the data of other areas (such as background scenes and non-interactive areas) is placed in the low-priority cache layer; this improves the response speed. In high-speed flight scenes, the focus area can be quickly transmitted and rendered, avoiding loading delays when the user quickly moves the perspective.

[0054] Furthermore, the interaction point refers to a tactile feedback device, which is used to transmit touch (such as a vibrating handle) when the user interacts with a virtual object. The tactile feedback is highly synchronized with the visual feedback, avoiding the feedback lag problem in traditional solutions.

[0055] Furthermore, the synchronous adjustment method of vision and touch in step S3 is:

[0056] Classify and model the materials of objects in each focus area, and set the material model of each object as Mi , its tactile feedback characteristic is M i ={E i ,H i ,R i}, where E i represents the elastic modulus of object i, H i represents the surface hardness of object i, R i represents the roughness of object i;

[0057] When a user interacts with an object, the tactile intensity is adjusted through force feedback at the interaction point. Assuming that the user interacts with object i at position p(t) at time t, the contact force F at the interaction point is contact (t) is expressed as follows based on the elasticity and hardness of the object: Among them, F contact (t) represents the force generated when the user touches the object at time t, A represents the proportional constant used to adjust the amplitude of the feedback, and E i represents the elastic modulus of object i, indicating the degree of response of the object to pressure, Δx(t) represents the deformation of the object at time t, and H i Indicates the hardness parameter of the object;

[0058] The feedback pattern of the tactile device depends on the roughness R of the object. i The interactive contact force F(t) is dynamically adjusted, and the vibration frequency f(t) and vibration intensity V(t) are calculated as follows:

[0059]

[0060] V(t)=C·F contact (t)·R i , where f(t) is the vibration frequency at time t. The higher the roughness R i The smaller the value, the higher the frequency. B is the frequency proportional constant related to the tactile device. V(t) is the vibration intensity at time t. The feedback intensity is related to the contact force F. contact (t) and the roughness R of the surface i Proportional to, C is the proportionality constant of the vibration intensity associated with the tactile device;

[0061] This simulates the situation when the tactile device adjusts its feedback intensity and mode according to the physical properties of the object's surface and the interaction force when it comes into contact with objects of different materials.

[0062] Furthermore, during the synchronous adjustment of vision and touch in step S3:

[0063] Compute visual update cycle T visual and tactile feedback update period T haptic , and keep the delay between them within the allowed range:

[0064] δt=|T visual -T haptic |≤∈, where T visual Represents the computational vision update cycle, T haptic represents the tactile feedback update period, δt is the time difference between vision and tactile, and ∈ is the maximum allowed delay;

[0065] By setting a maximum delay, tactile feedback is synchronized with visual feedback to avoid perceptual inconsistencies and enhance user immersion.

[0066] The beneficial effects of the present invention are:

[0067] The present invention predicts the user's current and next focus points through eye tracking and Kalman filtering, and prioritizes the allocation of rendering resources to these areas. It utilizes a dynamic resource allocation mechanism to pre-allocate resources to the areas that the user is about to focus on, while reducing the processing priority of secondary areas, thereby avoiding the waste of computing resources.

[0068] The present invention predicts the movement trajectory of the user's gaze point through a Kalman filter and renders the user's next focus point in advance. When the perspective switches quickly, the rendering of the focus area is completed, reducing the image lag and blur felt by the user. This greatly improves the rendering efficiency and makes the user experience smoother and more natural, especially in high-speed dynamic environments (such as drone fast flight scenes). The effect is particularly significant.

[0069] The present invention adopts a multi-level cache architecture, placing the user's current focus point and predicted next focus point data in a fast response cache, while the data of secondary areas is placed in a low-priority cache layer. In high-speed mobile scenarios, it can quickly respond and complete the rendering of the focus area, avoiding the loading delay problem in traditional methods and effectively improving the response speed of the system.

[0070] The present invention dynamically adjusts the intensity and pattern of tactile feedback by modeling the material of the object (such as hardness, elasticity, and roughness) and combining it with force feedback at the user interaction point. The tactile feedback of the object is highly consistent with its physical properties. The user feels stronger feedback when touching a hard object and a higher vibration frequency when touching a rough object. This improves the authenticity of the tactile feedback, maintains a high degree of synchronization between tactile and visual feedback, avoids the inconsistency between tactile and visual caused by fixed delays in traditional methods, and enhances the overall sense of immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0072] Figure 1 The figure is a flow chart of the virtual reality fusion method based on drone video of the present invention. DETAILED DESCRIPTION

[0073] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0074] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0075] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0076] Example 1, with reference to Figure 1 This embodiment provides a virtual reality fusion method based on drone video, comprising the following steps:

[0077] Step S1, scene data mining,

[0078] Use convolutional neural networks to analyze drone videos, mark target objects in the videos, and pre-render them.

[0079] Eye tracking is used to locate the user's current focus point, that is, the current gaze area, and predict the next focus point based on the movement trajectory of the current focus point;

[0080] The target object here refers to the key content in the video, such as the subject captured by the drone (such as the tracked cat) and the environment in which it interacts (such as buildings, trees, etc.);

[0081] The method of marking the target object in the video in step S1 is:

[0082] Assume that the t-th frame in the video is represented by a matrix F(t), whose size is W×H, where W and H represent width and height respectively, and the pixel values ​​of the three channels are represented by Frgb (t), scale normalization and color standardization:

[0083] in, is the normalized data matrix of the t-th frame, μ represents the mean of the pixel values ​​of the frame, σ represents the standard deviation of the pixel values ​​of the frame, both of which are the mean and standard deviation of the RGB three channels;

[0084] Use convolutional neural networks to extract important features from each frame of the image. The important features refer to the target object and its bounding box. Let F(t) be the input frame, then:

[0085] X l =f(W l *X l-1 +b l ), where X l is the output feature map of the lth layer, f is the Sigmoid activation function, W l is the convolution kernel weight matrix of the lth layer, * represents the convolution operation, X l-1 It is the input feature map of the previous layer, initially it is a video frame b l is the bias term of the lth layer;

[0086] The YOLO target detection model is used to detect the extracted important features and generate multiple bounding boxes to represent the location and area of ​​the target object. Assuming that there are N target objects in each frame F(t), the detection result is the bounding box B of each target. i and its confidence score S i :

[0087] B i =(x i ,y i ,w i ,h i ),

[0088] S i =P(c i |F(t)), where B i represents the bounding box of the i-th target object, x i ,y i is the center coordinate of the bounding box, w i ,h i is the width and height, S i Represents the confidence score of the target object, defined as the target belongs to a certain category c i The probability P(c i |F(t));

[0089] This step detects and calibrates important features in each frame. Bounding boxes and confidence scores are used to determine the location and importance of objects, respectively.

[0090] In step S1, the target object is pre-rendered:

[0091] Confidence score S based on target detection i , assign priority to each target object area and perform pre-rendering planning, priority P i Calculate based on the confidence of the target and the focus point G(t) of the user's sight:

[0092] P i =αS i +β·d(B i ,G(t)), where P i is the rendering priority of the i-th target object, α and β are weight coefficients, balancing the target confidence and distance weight, d(B i ,G(t)) represents the target bounding box B i The distance from the user's current focus point G(t) is calculated as:

[0093] Among them, (x g ,y g ) is the coordinate of the user's focus point G(t), (x i ,y i ) is the target bounding box B i The center coordinates of

[0094] This step integrates the object's confidence score and the user's gaze distance, giving target objects near the focus point a higher rendering priority;

[0095] Based on the priority P i , pre-render the target object, set the rendering resource in the video frame to R, and the total resource budget to R total , each target object is allocated rendering resources according to priority:

[0096] Among them, R i is the rendering resource allocated to the i-th target object, P i is the priority of the i-th target object, R total is the total available rendering resources, N is the total number of target objects detected in the current frame, and P j is the priority of the j-th target object;

[0097] Thus more rendering resources are allocated to important target objects and areas;

[0098] Specifically, in terms of scene data mining, convolutional neural networks are used to extract important features from drone videos, and target detection models are used to identify key areas and target objects. Unlike traditional methods, this method not only calibrates the position of objects, but also integrates the importance of objects and the user's gaze area. Based on the object confidence score and the user's current visual focus, dynamic rendering priority is assigned to each area. Important objects near the focus point will obtain more rendering resources, so that the part that the user is concerned about has higher rendering accuracy and real-time performance.

[0099] The method for predicting the next focus point in step S1 is:

[0100] Assume that the user's focus point at time t is G(t), and its position is represented by two-dimensional coordinates (x t ,y t ), record the position of the focus point in the past k time steps, and form a time series G(tk), G(t-k+1),…, G(t), G(t)=(x t ,y t ), where G(t) is the user’s focus point at time t, x t and y t are the horizontal and vertical coordinates of the focus point in the current frame respectively;

[0101] Calculate the sight speed and acceleration based on the change in the position of the focus point in adjacent time steps:

[0102] Where v(t) is the user’s line of sight velocity at time t, G(t)-G(t-1)=(x t -x t-1 ,y t -y t-1 ), represents the difference between the positions of two adjacent focus points, and Δt is the time interval between adjacent time steps;

[0103] Where a(t) is the user's line of sight acceleration at time t, and v(t-1) is the line of sight velocity at time t-1;

[0104] The method of predicting the next focus point in step S1 also includes:

[0105] Based on the current position, velocity, and acceleration of the point of interest, a Kalman filter is used to predict the next point of interest, and the position, velocity, and acceleration of the point of interest are constructed as a state vector X(t):

[0106] Among them, x t ,y t is the coordinate of the current focus point, v x (t),vy (t) are the horizontal and vertical line of sight speeds, respectively, a x (t),a y (t) are the lateral and longitudinal line of sight accelerations, respectively;

[0107] Predict the location of the focus point at the next moment t+1:

[0108] X(t+1)=FX(t)+w(t), where X(t+1) is the state vector at time t+1, which contains the predicted position, velocity, and acceleration of the point of interest. F is the state transition matrix, which represents the state change of the system from time t to t+1: w(t) is the process noise vector;

[0109] Assume that the actual observed point of interest is Z(t+1), and the Kalman gain K(t+1) is used to update the predicted value:

[0110] X(t+1)=X(t+1) pred +K(t+1)(Z(t+1)-HX(t+1) pred ),

[0111] Among them, X(t+1) pred is the predicted value of the Kalman filter, Z(t+1) is the actual observation value, that is, the position of the user's attention point at time t+1, and H is the observation matrix;

[0112] The Kalman gain K(t+1) is calculated as:

[0113] K(t+1)=P(t+1)H T (HP(t+1)H T +R) -1 ,

[0114] Where P(t+1) is the predicted state covariance matrix, R is the observation noise covariance matrix, and T represents the transpose operation;

[0115] Through the prediction and update process of the Kalman filter, the next point of interest is obtained;

[0116] Specifically, a dynamic resource allocation mechanism is introduced to monitor the user's gaze position in real time through eye tracking technology, and the Kalman filter model is combined to predict the user's next gaze point. By analyzing and predicting the movement of the user's gaze in advance, rendering resources are allocated in advance to areas that may be gazed at. Even in the case of rapid perspective switching, the user's visual focus area can be ensured to maintain high-definition rendering, avoiding the occurrence of blurred or delayed images, and avoiding excessive rendering of secondary areas, thereby reducing the waste of computing resources.

[0117] Step S2, dynamic resource allocation,

[0118] Provide high-quality rendering for the user's focus point while reducing the processing priority of other areas. The focus points include the current focus point and the next focus point. Resources are allocated in advance based on the predicted next focus point.

[0119] Dynamic resource allocation enables efficient rendering of points of interest in drone video scenes. This allocation is based on the user's current and predicted visual focus. This means resources are allocated in advance of the next view movement. Even when the user's view is rapidly shifting, detailed rendering of the desired area is still completed in advance, preventing users from experiencing delays or blurry images.

[0120] The dynamic resource allocation in step S2 adopts a multi-level cache architecture:

[0121] The focus area data is placed in a fast-response cache, while other area data (such as background scenes and non-interactive areas) are placed in a low-priority cache layer. This improves response speed. In high-speed flight scenarios, the focus area can be quickly transmitted and rendered, avoiding loading delays when the user quickly moves the view.

[0122] Specifically, in terms of resource management, this method adopts a multi-level cache architecture, storing data in key areas in a fast-response cache and data in other secondary areas in a low-priority cache layer. The layered cache design improves response speed and can quickly render the user's current and predicted focus areas, avoiding loading delays when the perspective moves quickly.

[0123] Step S3, synchronous adjustment of vision and touch,

[0124] Differentiate tactile feedback for objects within the focus area, classifying object materials as hard, soft, smooth, and rough. When a user interacts with certain objects in the video, the tactile feedback intensity and pattern are dynamically adjusted based on the user's interaction point and the object's material.

[0125] Interaction points refer to tactile feedback devices that deliver tactile sensations (such as vibrating controllers) when users interact with virtual objects.

[0126] The tactile feedback is highly synchronized with the visual feedback, avoiding the feedback lag problem in traditional solutions. The synchronization adjustment method of vision and tactile in step S3 is:

[0127] Classify and model the materials of objects in each focus area, and set the material model of each object as M i , its tactile feedback characteristic is M i ={E i ,H i ,R i}, where E irepresents the elastic modulus of object i, H i represents the surface hardness of object i, R i represents the roughness of object i;

[0128] When a user interacts with an object, the tactile intensity is adjusted through force feedback at the interaction point. Assuming that the user interacts with object i at position p(t) at time t, the contact force F at the interaction point is contact (t) is expressed as follows based on the elasticity and hardness of the object: Among them, F contact (t) represents the force generated when the user touches the object at time t, A represents the proportional constant used to adjust the amplitude of the feedback, and E i represents the elastic modulus of object i, indicating the degree of response of the object to pressure, Δx(t) represents the deformation of the object at time t, and H i Indicates the hardness parameter of the object;

[0129] The feedback pattern of the tactile device depends on the roughness R of the object. i The interactive contact force F(t) is dynamically adjusted, and the vibration frequency f(t) and vibration intensity V(t) are calculated as follows:

[0130]

[0131] V(t)=C·F contact (t)·R i ,in,

[0132] f(t) is the vibration frequency at time t. The higher the roughness R i The smaller the value, the higher the frequency. B is the frequency proportional constant related to the tactile device. V(t) is the vibration intensity at time t. The feedback intensity is related to the contact force F. contact (t) and the roughness R of the surface i Proportional to, C is the proportionality constant of the vibration intensity associated with the tactile device;

[0133] This simulates the situation when the haptic device adjusts its feedback intensity and mode according to the physical properties of the object's surface and the interaction force when it comes into contact with objects of different materials.

[0134] During the synchronous adjustment of vision and touch in step S3:

[0135] Compute visual update cycle T visual and tactile feedback update period T haptic , and keep the delay between them within the allowed range:

[0136] δt=|T visual -T haptic |≤∈, where T visualRepresents the computational vision update cycle, T haptic represents the tactile feedback update period, δt is the time difference between vision and tactile, and ∈ is the maximum allowed delay;

[0137] Synchronize tactile feedback with visual feedback by setting a maximum delay, thus avoiding perceptual inconsistencies and enhancing user immersion.

[0138] Specifically, when users interact with virtual objects, the intensity and pattern of tactile feedback are dynamically adjusted according to the material properties of the object (such as elasticity, hardness, and roughness). Hard objects will produce stronger tactile feedback, while rough objects will bring higher vibration frequencies. By calculating the contact force between the user and the object, a more realistic tactile experience is simulated. At the same time, a high degree of synchronization between visual feedback and tactile feedback is guaranteed, avoiding perceptual distortion caused by delays or mismatches, and enhancing the user's immersion in the virtual reality environment.

[0139] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A virtual reality fusion method based on drone video, characterized by: include, Step S1, scene data mining, uses convolutional neural networks to analyze drone videos, mark target objects in the video, and pre-render them. Eye tracking is used to locate the user's current focus point, that is, the current gaze area, and predict the next focus point based on the movement trajectory of the current focus point; Step S2: Dynamic resource allocation: Render the user's focus point with high quality while reducing the processing priority of other areas. The focus points include the current focus point and the next focus point. Resources are allocated in advance based on the predicted next focus point. Step S3: Synchronous adjustment of vision and touch. Differentiated tactile feedback is performed on objects within the focus area, with the object material classified as hard, soft, smooth, or rough. When the user interacts with certain objects in the video, the tactile feedback intensity and pattern are dynamically adjusted based on the user's interaction point and the object material. Interaction points refer to tactile feedback devices that deliver tactile sensations when users interact with virtual objects; The synchronous adjustment method of vision and touch in step S3 is: Classify and model the materials of objects in each focus area, and set the material model of each object as M i , its tactile feedback characteristic is M i ={E i ,H i ,R i }, where E i represents the elastic modulus of object i, H i represents the surface hardness of object i, R i represents the roughness of object i; When a user interacts with an object, the tactile intensity is adjusted through force feedback at the interaction point. Assuming that the user interacts with object i at position p(t) at time t, the contact force F at the interaction point is contact (t) is expressed as follows based on the elasticity and hardness of the object: Among them, F contact (t) represents the force generated when the user touches the object at time t, A represents the proportional constant used to adjust the amplitude of the feedback, and E i represents the elastic modulus of object i, indicating the degree of response of the object to pressure, Δx(t) represents the deformation of the object at time t, and H i Indicates the hardness parameter of the object; The feedback pattern of the tactile device depends on the roughness R of the object. i and the interaction contact force F contact (t) is dynamically adjusted, and the vibration frequency f(t) and vibration intensity V(t) are calculated as follows: V(t)=C·F contact (t)·R i , where f(t) is the vibration frequency at time t. The higher the roughness R i The smaller the value, the higher the frequency. B is the frequency proportional constant related to the tactile device. V(t) is the vibration intensity at time t. The feedback intensity is related to the contact force F. contact (t) and the roughness R of the surface i Proportional to, C is the proportionality constant of the vibration intensity associated with the tactile device; During the synchronous adjustment of vision and touch in step S3: Compute visual update cycle T visual and tactile feedback update period T haptic , and keep the delay between them within the allowed range: δt=|T visual -T haptic |≤∈, where T visual Represents the computational vision update cycle, T haptic represents the haptic feedback update period, δt is the time difference between vision and haptic, and ∈ is the maximum allowed delay.

2. The virtual reality fusion method based on drone video according to claim 1, characterized in that: The method of marking the target object in the video in step S1 is: Assume that the t-th frame in the video is represented by a matrix F(t), whose size is W×H, where W and H represent width and height respectively, and the pixel values ​​of the three channels are represented by F rgb (t), scale normalization and color standardization: in, is the normalized data matrix of the t-th frame, μ represents the mean of the pixel values ​​of the frame, σ represents the standard deviation of the pixel values ​​of the frame, both of which are the mean and standard deviation of the RGB three channels; Use convolutional neural networks to extract important features from each frame of the image. The important features refer to the target object and its bounding box. Let F(t) be the input frame, then: X l =f(W l *X l-1 +b l ), where X l is the output feature map of the lth layer, f is the Sigmoid activation function, W l is the convolution kernel weight matrix of the lth layer, * represents the convolution operation, X l-1 It is the input feature map of the previous layer, initially it is a video frame b l is the bias term of the lth layer; The YOLO target detection model is used to detect the extracted important features and generate multiple bounding boxes to represent the location and area of ​​the target object. Assuming that there are N target objects in each frame F(t), the detection result is the bounding box B of each target. i and its confidence score S i : B i =(x i ,y i ,w i ,h i ), S i =P(c i |F(t)), where B i represents the bounding box of the i-th target object, x i ,y i is the center coordinate of the bounding box, w i ,h i is the width and height, S i Represents the confidence score of the target object, defined as the target belongs to a certain category c i The probability P(c i |F(t)).

3. The virtual reality fusion method based on drone video according to claim 2, characterized in that: In step S1, the target object is pre-rendered: Confidence score S based on target detection i , assign priority to each target object area and perform pre-rendering planning, priority P i Calculate based on the confidence of the target and the focus point G(t) of the user's sight: P i =αS i +β·d(B i ,G(t)), where P i is the rendering priority of the i-th target object, α and β are weight coefficients, balancing the target confidence and distance weight, d(B i ,G(t)) represents the target bounding box B i The distance from the user's current focus point G(t) is calculated as: Among them, (x g ,y g ) is the coordinate of the user's focus point G(t), (x i ,y i ) is the target bounding box B i The center coordinates of Based on the priority P i , pre-render the target object, set the rendering resource in the video frame to R, and the total resource budget to R total , each target object is allocated rendering resources according to priority: Among them, R i is the rendering resource allocated to the i-th target object, P i is the priority of the i-th target object, R total is the total available rendering resources, N is the total number of target objects detected in the current frame, and P j is the priority of the j-th target object.

4. The virtual reality fusion method based on drone video according to claim 3 is characterized in that: The method for predicting the next focus point in step S1 is: Assume that the user's focus point at time t is G(t), and its position is represented by two-dimensional coordinates (x t ,y t ), record the position of the focus point in the past k time steps, and form a time series G(tk), G(t-k+1),…, G(t), G(t)=(x t ,y t ), where G(t) is the user’s focus point at time t, x t and y t are the horizontal and vertical coordinates of the focus point in the current frame respectively; Calculate the sight speed and acceleration based on the change in the position of the focus point in adjacent time steps: Where v(t) is the user’s line of sight velocity at time t, G(t)-G(t-1)=(x t -x t-1 ,y t -y t-1 ), represents the difference between the positions of two adjacent focus points, and Δt is the time interval between adjacent time steps; Where a(t) is the user's line of sight acceleration at time t, and v(t-1) is the line of sight velocity at time t-1.

5. The virtual reality fusion method based on drone video according to claim 4, characterized in that: The method of predicting the next focus point in step S1 also includes: Based on the current position, velocity, and acceleration of the point of interest, a Kalman filter is used to predict the next point of interest, and the position, velocity, and acceleration of the point of interest are constructed as a state vector X(t): Among them, x t ,y t is the coordinate of the current focus point, v x (t),v y (t) are the horizontal and vertical line of sight speeds, respectively, a x (t),a y (t) are the lateral and longitudinal line of sight accelerations, respectively; Predict the location of the focus point at the next moment t+1: X(t+1)=FX(t)+w(t), where X(t+1) is the state vector at time t+1, which contains the predicted position, velocity, and acceleration of the point of interest. F is the state transition matrix, which represents the state change of the system from time t to t+1: w(t) is the process noise vector; Assume that the actual observed point of interest is Z(t+1), and the Kalman gain K(t+1) is used to update the predicted value: X(t+1)=X(t+1) pred +K(t+1)(Z(t+1)-HX(t+1) pred ), Among them, X(t+1) pred is the predicted value of the Kalman filter, Z(t+1) is the actual observation value, that is, the position of the user's attention point at time t+1, and H is the observation matrix; The Kalman gain K(t+1) is calculated as: K(t+1)=P(t+1)H T (HP(t+1)H T +R) -1 , Where P(t+1) is the predicted state covariance matrix, R is the observation noise covariance matrix, and T represents the transpose operation.

6. The virtual reality fusion method based on drone video according to claim 5, characterized in that: The dynamic resource allocation in step S2 adopts a multi-level cache architecture: Place the focus area data in the fast response cache and other area data in the low-priority cache layer.

Images