A first-person view drone video saliency prediction method and system
By combining camera motion information and first-person perspective priors, and employing a cascaded architecture approach, the applicability of traditional video saliency prediction methods in first-person perspective UAV videos is addressed, achieving higher prediction accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2023-12-18
- Publication Date
- 2026-07-31
AI Technical Summary
Existing video saliency prediction methods are mainly designed for traditional videos and are difficult to adapt to the camera motion effects and viewing habits of first-person perspective drone videos, resulting in insufficient prediction accuracy.
A cascaded architecture based on camera motion information and first-person view prior information is adopted. The rotation matrix is calculated through motion reconstruction structure, and a saliency prediction map is generated by combining saliency basic feature extraction, spatiotemporal feature extraction, camera motion module and first-person view prior module.
It improves the accuracy of saliency prediction in first-person view drone videos, effectively integrates camera motion and visual features, and generates smoother and more accurate saliency prediction results.
Smart Images
Figure CN117671538B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video saliency prediction, and in particular to a first-person perspective UAV video saliency prediction technology based on camera motion information and first-person perspective prior information. Background Technology
[0002] When observing a scene, the human visual system quickly and selectively identifies the most attention-grabbing parts from a large amount of received information, rather than processing information from all areas. Even in complex environments, humans can quickly focus their attention on the important parts of the scene. This is called the visual attention mechanism, and the areas in a scene that attract the attention of the visual system are called saliency regions. Understanding and modeling this mechanism to predict which areas in a video will attract more human attention is the research content of video saliency. By applying the human visual attention mechanism to the field of video processing, computers can be assisted in selectively processing information in videos. It enables computers to prioritize the allocation of limited computing resources to more important areas, which will greatly improve the efficiency of video processing methods.
[0003] Over the past few decades, drone technology has developed rapidly and has been widely applied in key areas such as fire detection, wildlife statistics, environmental protection, and news reporting. Consequently, the demand for computer vision research analyzing drone images and videos is constantly increasing. Compared to traditional video, drone video has several distinct characteristics. First, most drone videos are shot from a bird's-eye view, resulting in significant differences in the appearance, shape, size, and other features of objects compared to traditional video. Second, drone videos typically feature complex backgrounds and rich scene information, encompassing diverse landscapes. Third, drone videos usually involve less manual editing and scene transitions, with relatively small changes between adjacent frames. These characteristics of drone video can potentially influence human attention to the content within the video.
[0004] Recently, a new type of drone has become increasingly popular, known as first-person perspective drones. These drones are equipped with real-time image transmission capabilities, allowing users to wear flight goggles (a type of head-mounted display device) and experience an immersive feeling of controlling the drone as if they were flying in the air themselves. First-person drone videos exhibit significant differences compared to traditional drone videos: (1) Shooting angle: Unlike the typical bird's-eye view angle of traditional drone videos, first-person drone videos offer a wider variety of perspectives, including eye-level and upward-looking views. (2) Camera motion modes: Traditional drones typically offer limited camera motion modes. In contrast, first-person drone videos showcase a wider range of camera motion modes, such as acceleration, braking, rotation, and ascent / descend. First-person perspective is currently widely used in movies, allowing viewers to observe scenes from the actors' point of view.
[0005] Over the past few decades, video saliency prediction has seen significant improvements in accuracy, evolving from traditional algorithms using handcrafted features to models leveraging deep learning methods. However, current mainstream video saliency prediction methods are designed for traditional videos played on LCD screens, which differs from the shooting and viewing experience of first-person perspective drone videos. Furthermore, most current video saliency prediction methods rely on a center prior. Analysis of eye-tracking experiments on first-person perspective drone videos reveals that viewers tend to focus on the lower center of the image, and the variances in the horizontal and vertical directions differ, rendering the center prior inapplicable. Camera motion also influences attention; for example, when the drone rotates left, viewers tend to focus on the left half of the image, and vice versa. Therefore, video saliency prediction methods based on camera motion and first-person perspective prior information will provide predictions that better reflect human viewing habits in real-world scenarios. Compared to mature traditional video saliency prediction methods, research on saliency prediction methods for first-person perspective drone videos is still in its early stages. Furthermore, the application of camera motion information in video saliency tasks is still limited. Therefore, how to reasonably utilize similar motions for first-person perspective UAV video saliency prediction is a significant challenge in this technical field. Summary of the Invention
[0006] Because people immerse themselves in first-person perspective drone videos by wearing VR headsets, and most current video saliency prediction methods are only applicable to traditional videos played on LCD screens, these methods can predict regions of interest well in traditional videos such as movies and short videos, but they struggle to predict first-person perspective drone videos. Furthermore, camera motion in first-person drone videos affects the human gaze area. While camera motion information can aid saliency prediction, existing methods often employ a center prior, which is not applicable to first-person perspective videos. This invention addresses this issue by providing a first-person drone video saliency prediction technology based on camera motion and first-person perspective prior information.
[0007] The present invention provides a first-person perspective UAV video saliency prediction method. Based on camera motion information and first-person perspective prior information, the method performs first-person UAV video saliency prediction. The implementation includes: firstly, calculating the rotation matrix of each frame using a motion reconstruction structure based on several consecutive input video frames; simultaneously, feeding the video frames into a saliency basic feature extraction module, and then feeding the output into the temporal encoder and spatial encoder in the spatiotemporal feature extraction module; adding the outputs of the two encoders and integrating them into a 2D convolutional layer, then adding this result to the output of the saliency basic feature extraction module to obtain the output of the spatiotemporal feature extraction module; using the output of the spatiotemporal feature extraction module and the rotation matrix calculated by the motion reconstruction structure as input to a camera motion module, and feeding the output of the camera motion module into a first-person perspective prior module; feeding the output of the first-person perspective prior module into a temporal averaging module, and then feeding the result into a decoder to finally obtain a saliency prediction map.
[0008] Furthermore, the process of achieving saliency prediction in first-person drone video includes the following steps.
[0009] Step 1: Obtain the quaternion of camera motion from the video frames to be predicted using the motion reconstruction structure, and further process it to obtain the rotation matrix of each frame, thus preprocessing the video frames.
[0010] Step 2: Construct a salient basic feature extraction module to extract the basic salient features of video frames;
[0011] Step 3: Construct a spatiotemporal feature extraction module. In the spatiotemporal feature extraction module, the time encoder extracts the inter-frame temporal features from multiple consecutive video frames, the spatial encoder extracts the spatial features from the video frames, and the temporal features and spatial features are fused to obtain the spatiotemporal features.
[0012] Step 4: Construct a camera motion module. Input the rotation matrix obtained in Step 1 through the motion recovery structure into the camera motion module to extract camera motion information, and adaptively fuse it with the spatiotemporal features obtained in Step 3.
[0013] Step 5: Construct a first-person perspective prior module. In the first-person perspective prior module, a first-person perspective prior map is generated by statistically analyzing the data, and the multi-scale prior map is fused with the spatiotemporal saliency features based on camera motion.
[0014] Step 6: Construct a time-series averaging module to make the predicted saliency map smoother in the time dimension;
[0015] Step 7: Use the decoder to integrate the saliency features into a single-channel saliency map;
[0016] Step 8: Train the overall model constructed from steps 2-6;
[0017] Step 9: Use the trained overall model to predict visual saliency.
[0018] Furthermore, the implementation method of constructing the salient basic feature extraction module is as follows:
[0019] An improvement upon the MobileNet-V2 model is made by replacing the final spatial pooling layer with a perforated pyramidal pooling layer to obtain multi-scale features for extracting features of salient objects of different sizes. Then, features from the last three convolutional layers of MobileNet-V2 are used for feature fusion to obtain the salient base features F. x , where x is the input video frame number.
[0020] Furthermore, the spatiotemporal feature extraction module extracts spatiotemporal features in the following way:
[0021] When video frame X i Feature F is obtained after the saliency-based basic feature extraction module. i Then, the input is to the spatial encoder, where F is integrated through depthwise separable convolutional layers. i Spatial features at different levels of abstraction are used to obtain spatial saliency features.
[0022] When n consecutive video frames X1, X2, ..., X n After being input into the temporal encoder via n parallel saliency feature extraction modules, the temporal encoder concatenates the n corresponding features obtained from the saliency feature extraction modules. First, a convolutional layer is used for dimensionality reduction. Next, the Bi-Sub operator is used to calculate the difference between the current frame features and the features of the preceding and following frames, obtaining the temporal difference features. A final convolutional layer is then used for feature dimensionality enhancement, ensuring that the input and output feature dimensions are the same, ultimately outputting the temporal saliency features. X1,X2,…,X n The time encoder outputs share this single output;
[0023] Calculated time-domain features and spatial domain features Subsequently, feature fusion is performed using element-wise addition, and a 1×1 convolutional layer is used to further integrate spatiotemporal features. These features are then superimposed onto the saliency base feature F using residual connections. i Spatiotemporal saliency features were obtained.
[0024] Moreover, in the camera motion module,
[0025] The camera motion features are obtained by passing the rotation matrix of each frame through two consecutive fully connected layers. The spatiotemporal saliency features are then globally averaged and pooled in the spatial dimension to obtain pooled features. Camera motion characteristics With pooling characteristics The layers are stacked along the channel dimension, and a fully connected layer and a sigmoid activation function are used to obtain the weights W corresponding to each channel; finally, the spatiotemporal saliency features are... Multiplying the weight W channel by channel yields the spatiotemporal saliency features based on camera motion.
[0026] Moreover, in the first-person perspective prior module,
[0027] First, the fixation points in the dataset are statistically analyzed to obtain the mean μ of the fixation point position coordinates in the horizontal and vertical directions. x ,μ y and variance σ x ,σ y Based on the statistical results, eight prior images with the same mean but different variances are generated. Two 2D convolutional layers are used to integrate the prior images, which are then superimposed on the spatiotemporal saliency features based on camera motion along the channel dimension. Finally, a single 2D convolutional layer is used to integrate the features, resulting in spatiotemporal saliency features based on camera motion and first-person perspective priors.
[0028] Moreover, in the time-series averaging module,
[0029] The spatiotemporal saliency features at the current time and the output of the temporal averaging module at the previous time are fed into a convolutional layer, and the outputs of the convolutional layers are added together. Then, the weight W is obtained by using the sigmoid activation function. Finally, the weight is multiplied element-wise by the spatiotemporal saliency features at the current time, and (1-W) is multiplied element-wise by the output of the temporal averaging module at the previous time. The results of the two multiplications are then added together to obtain the output of the temporal averaging module at the current time.
[0030] On the other hand, the present invention also provides a first-person perspective UAV video saliency prediction system for implementing the first-person perspective UAV video saliency prediction method described above.
[0031] Alternatively, it may include a processor and a memory, with the memory used to store program instructions and the processor used to call the stored instructions in the memory to execute a first-person perspective UAV video saliency prediction method as described above.
[0032] Alternatively, it may include a readable storage medium on which a computer program is stored, which, when executed, implements a first-person perspective UAV video saliency prediction method as described above.
[0033] The first-person perspective (FPPS) UAV video saliency prediction scheme proposed in this invention, based on camera motion and first-person perspective prior information, employs a cascaded architecture to improve the accuracy of saliency prediction. To effectively extract saliency-related visual temporal and visual spatial features and utilize camera motion as prior information to enhance prediction accuracy, a novel structure is proposed. This structure includes a saliency feature extraction module that extracts basic saliency features from video frames, and a spatiotemporal feature extraction module consisting of two encoders: a temporal encoder that extracts inter-frame temporal features from multiple consecutive video frames, and a spatial encoder that extracts spatial features from video frames. A temporal averaging module is also designed to make the predicted saliency map smoother in the temporal dimension. To effectively utilize camera motion information to improve the accuracy of visual saliency prediction, a camera motion module is also proposed. This module uses a motion reconstruction structure method and some post-processing to calculate the rotation matrix for each frame, extracts camera motion features based on the rotation matrix, and fuses them with the spatiotemporal saliency features. To effectively utilize prior information for more accurate prediction, a first-person perspective prior module is proposed. This module statistically generates a prior map suitable for viewing first-person perspective videos and fuses it with the features.
[0034] The beneficial effects of this invention are as follows:
[0035] We effectively model the relationship between human visual attention and camera motion by using a novel method to generate prior maps based on camera motion. By adding a camera motion module that adaptively fuses camera motion into the model, we more effectively integrate camera motion information and salient features. Attached Figure Description
[0036] Figure 1 This is a general framework diagram of an embodiment of the present invention.
[0037] Figure 2 This is a framework diagram of the salient basic feature extraction module in an embodiment of the present invention.
[0038] Figure 3 This is a framework diagram of the timing averaging module in an embodiment of the present invention.
[0039] Figure 4 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation
[0040] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0041] This invention discloses a first-person UAV video saliency prediction method based on camera motion and first-person perspective prior information. First, the rotation matrix of the UAV during the capture of each video frame is obtained using a structure-of-motion (COMO) method. For each video frame, spatial and temporal features are extracted using a saliency feature extraction module and a spatiotemporal feature extraction module. Then, a camera motion module extracts camera motion features based on the rotation matrix and adaptively fuses them with the spatiotemporal saliency features. Subsequently, a first-person perspective prior module fuses the saliency features with a multi-scale prior map to aid in more accurate prediction. Finally, a temporal averaging module smooths the generated saliency prediction map over time. The features are then decoded to obtain the saliency prediction map of the video frame.
[0042] See Figure 1 The overall scheme proposed in this embodiment of the invention is as follows: First, based on several (n) consecutive input video frames, the rotation matrix of each frame is calculated using the motion reconstruction structure algorithm. Simultaneously, the video frames are fed into the saliency basic feature extraction module, and the output is further fed into the temporal encoder and spatial encoder in the spatiotemporal feature extraction module. The outputs of the two encoders are added and integrated into a 2D convolutional layer, then added to the output of the saliency basic feature extraction module to obtain the output of the spatiotemporal feature extraction module. The output of the spatiotemporal feature extraction module and the rotation matrix calculated by the motion reconstruction structure are used as the input to the camera motion module, and the output of the camera motion module is fed into the first-person perspective prior module. The output of the first-person perspective prior module is fed into the temporal averaging module, and the result is then fed into the decoder to finally obtain the saliency prediction map. In specific implementation, n is the batch size.
[0043] like Figure 4 As shown in the figure, the first-person perspective UAV video saliency prediction method provided by the present invention includes the following steps:
[0044] Step 1: Obtain the rotation matrix of the camera motion and preprocess the data.
[0045] This step first obtains the quaternions of camera motion from the video frames to be predicted using the structure-of-motion motion (SCOP) method, and then further processes them to obtain the rotation matrix for each frame. The video frames are then preprocessed.
[0046] Step 1 of the embodiment is preferably implemented using the following sub-steps:
[0047] Step 1.1: First, process the video to be predicted by cropping it into video frames according to the original video frame rate.
[0048] Step 1.2: Frame extraction is performed on the video, preferably one frame every three frames. The motion reconstruction structure method is used on the sample frames to obtain a set of quaternions for each frame. These quaternions are then converted into rotation matrices relative to the previous sampled frame.
[0049] The motion recovery structure method is Colmap, where the quaternion includes (w,x,y,z), where w represents the imaginary part and x,y,z represent the real part. The rotation matrix of the transformation is used to represent the rotation of the camera.
[0050] Step 1.3, the preprocessing of the video frames includes: reading the image and converting it to RGB format; adjusting the image resolution to W×H; normalizing the image; and converting the data type to Tensor type for subsequent input into the neural network for computation; where W is the width of the adjusted image and H is the height of the adjusted image.
[0051] Step 2: Construct a salient basic feature extraction module.
[0052] This invention constructs a salient basic feature extraction module for extracting basic salient features of video frames.
[0053] See Figure 2 The module in this embodiment is an improvement on the MobileNet-V2 model, as detailed below:
[0054] First, video frames are fed into several consecutive convolutional blocks (five convolutional blocks are used in this example, denoted as convolutional blocks 1-5). The final spatial pooling layer in the MobileNet-V2 model is replaced with a perforated pyramid pooling layer to obtain multi-scale features for extracting features of salient objects of different sizes. Further, this perforated pyramid pooling layer consists of K parallel perforated convolutional layers (K is preferably 4). Figure 2 The convolutional blocks are denoted as dilated convolutional layers 1-4 respectively. The features obtained through the K dilated convolutional layers are concatenated together as the output of the last convolutional block (convolutional block 5) of the MobileNet-V2 model.
[0055] The second step involves concatenating the outputs of the last three convolutional blocks (convolutional blocks 3, 4, and 5) of MobileNet-V2. This includes using 2D convolutional layers to adjust the channel dimensions of the output features from each of the three convolutional blocks, and upsampling the features from convolutional blocks 4 and 5 to make the three feature maps the same size. The three features are then concatenated along the channel direction and passed through another convolutional layer to obtain the visual saliency base feature F. x , where x is the input video frame number.
[0056] Step 3: Construct a spatiotemporal feature extraction module.
[0057] The spatiotemporal feature extraction module constructed in this invention includes two encoders: a time encoder extracts inter-frame temporal features from multiple consecutive video frames, and a spatial encoder extracts spatial features from video frames, and fuses the temporal and spatial features.
[0058] Step 3 of the embodiment is preferably implemented using the following sub-steps:
[0059] Step 3.1: Construct a spatial encoder. When video frame X... i Feature F is obtained after the saliency-based basic feature extraction module. i Then, the input is to the spatial encoder, where F is integrated through a depthwise separable convolutional layer. i Spatial features at different levels of abstraction are used to obtain spatial saliency features. Where the labels i = 1, 2, ..., n. For example, when batchsize is 5, n = 5.
[0060] Step 3.2: Construct a temporal encoder. When n consecutive video frames (X1, X2, ..., X...) are generated... n The temporal encoder, through n parallel saliency feature extraction modules, concatenates the corresponding features obtained from the saliency feature extraction modules. First, a convolutional layer reduces the dimensionality. Then, the Bi-Sub operator calculates the difference between the current frame's features and those of the preceding and following frames, yielding the temporal difference features. A final convolutional layer is used for feature dimensionality enhancement, ensuring that the input and output feature dimensions are the same. The final output is the temporal saliency feature. X1,X2,…,X n The time encoder outputs share this single output.
[0061] The Bi-Sub operator can be found in the literature "K. Zhang, Z. Chen, S. Li, S. Liu, An efficient saliencyprediction model for unmanned aerial vehicle video, ISPRS Journal of Photogrammetry and Remote Sensing 194(2022)152-166", which will not be elaborated upon in this invention.
[0062] Step 3.3: Calculate the time-domain features and spatial domain features Subsequently, feature fusion is performed using element-wise addition, and a (1×1) convolutional layer is used to further integrate spatiotemporal features. These features are then superimposed onto the saliency base feature F using residual connections. i Spatiotemporal saliency features were obtained. The formula is expressed as follows, where W represents a (1×1) convolutional layer:
[0063]
[0064] Step 4: Build the camera motion module.
[0065] The rotation matrix obtained in step 1 using the motion recovery structure method is fed into the camera motion module to extract camera motion information and adaptively fused with the spatiotemporal features obtained in step 3.
[0066] In the camera motion module of this embodiment, the rotation matrix of each frame is passed through two consecutive fully connected layers to obtain camera motion features. The spatiotemporal saliency features are then globally averaged and pooled in the spatial dimension to obtain pooled features. Then camera motion features With pooling characteristics The layers are stacked along the channel dimension, and a fully connected layer and a sigmoid activation function are used to obtain the weights W corresponding to each channel; finally, the spatiotemporal saliency features are... Multiplying the weight W channel by channel yields the spatiotemporal saliency features based on camera motion.
[0067] Step 5: Construct a first-person perspective prior module. Statistical analysis of the data generates a first-person perspective prior map, and the multi-scale prior map is fused with spatiotemporal saliency features based on camera motion.
[0068] In the first-person perspective prior module of the embodiment, the gaze points in the dataset are first statistically analyzed to obtain the mean μ of the gaze point position coordinates in the horizontal and vertical directions. x ,μ y and variance σ x , σ y Based on the statistical results, eight prior images with the same mean but different variances are generated. Two 2D convolutional layers are used to integrate the multi-scale prior images, which are then superimposed on the spatiotemporal saliency features based on camera motion along the channel dimension. Finally, a single 2D convolutional layer is used for feature integration to obtain the spatiotemporal saliency features based on camera motion and first-person perspective priors.
[0069] Step 6: Construct the time series averaging module.
[0070] This invention uses a time-series averaging module to make the predicted saliency map smoother in the time dimension.
[0071] See Figure 3 In the temporal averaging module of the embodiment, the spatiotemporal saliency features at the current moment are... The output of the time series averaging module at the previous time step is fed into a convolutional layer, and the outputs of the convolutional layers are added together. Then, the weight W is obtained by using the sigmoid activation function. Finally, the weight is multiplied element-wise by the spatiotemporal saliency feature at the current time step. (1-W) is multiplied element-wise by the output of the time series averaging module at the previous time step. The two parts are then added element-wise to obtain the output of the time series averaging module at the current time step.
[0072] Step 7: Construct the decoder. Use the decoder to integrate the saliency features into a single-channel saliency map.
[0073] In the embodiment, the saliency features based on camera motion priors at the current moment are obtained. This needs to be integrated into a single-channel saliency map. It is preferred to generate the single-channel saliency map using a simple decoder. In this embodiment of the invention, a convolutional layer with a kernel size of 3×3 and an output feature channel of 1 is used to generate the saliency feature. Integrate to generate a single-channel saliency map S i .
[0074] Step 8: Train the overall model consisting of steps 2-6;
[0075] This example uses weights trained on the AVS1K dataset to initialize the salient feature extraction module and the spatiotemporal feature module. The UFPV200 dataset is used for overall model training. During training, the weights of the salient feature extraction module are not updated. Stochastic gradient descent is used for training.
[0076] Step 9: Use the trained overall model to predict visual saliency.
[0077] For ease of implementation and reference, the model training details of the embodiment are provided:
[0078] The proposed model is implemented using PyTorch on an NVIDIA TITAN X GPU. The parameters in the saliency base feature extraction module and the spatiotemporal feature extraction module are initialized using public weights trained on the large dataset AVS1K.
[0079] After weight initialization, the entire model is trained on the UFPV200 dataset. During training, the weights of the saliency base extraction module are not updated. Stochastic gradient descent is used for training, with an initial learning rate set to 10. -3 The momentum is set to 0.9, and the weight decay factor is set to 0.0005. The batch size is set to 2, and each batch consists of 5 consecutive video frames.
[0080] The loss function used during training is shown in the following formula:
[0081] L(S,M,F)=α1L kl (S,M)+α2L cc (S,M)+α3L nss (S,F)
[0082] Where S represents the single-channel saliency prediction map output by the model, M represents the saliency density map in the true data, F represents the fixation map in the true data, and α1, α2, and α3 represent the corresponding weights. L kl (), L cc (), L nss () represent the formulas for calculating KL divergence, linear correlation coefficient, and standardized path scan significance, respectively, as follows:
[0083]
[0084] Where i represents the i-th pixel, M i S is the value of the i-th pixel in the saliency density map. i It is the value of the i-th pixel in the saliency prediction map.
[0085]
[0086] Where cov(S,M) and σ(S,M) represent the covariance and standard deviation between the significance prediction results obtained by the model and the actual significance density map, respectively.
[0087]
[0088] Where μ(S) and σ(S) represent the mean and variance of the saliency prediction result map, respectively, i represents the i-th pixel, N represents the number of human eye fixation points, and F i It is the value of the i-th pixel in the fixation map. The larger the NSS value, the better the performance of the saliency prediction model.
[0089] In practice, α1, α2, and α3 are weights preset by the user. These weights can be preset through experiments and are preferably set to 1, -0.2, and -0.1 respectively in this embodiment.
[0090] In specific implementation, the method proposed in the technical solution of this invention can be automatically executed by those skilled in the art using computer software technology. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of this invention and computer equipment including the computer program running the corresponding computer program, should also be within the protection scope of this invention.
[0091] In some possible embodiments, a first-person perspective drone video saliency prediction system is provided, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a first-person perspective drone video saliency prediction method as described above.
[0092] In some possible embodiments, a first-person perspective drone video saliency prediction system is provided, including a readable storage medium on which a computer program is stored, wherein when the computer program is executed, it implements a first-person perspective drone video saliency prediction method as described above.
[0093] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, but without departing from the spirit of the invention or exceeding the scope defined by the appended claims.
Claims
1. A method for predicting the saliency of first-person perspective UAV videos, characterized in that: Based on camera motion information and first-person perspective prior information, saliency prediction of first-person UAV videos is performed. The implementation method includes: firstly, based on several consecutive input video frames, calculating the rotation matrix of each frame using a motion reconstruction structure; simultaneously, the video frames are fed into a saliency basic feature extraction module, and the output of the obtained output is further fed into the temporal encoder and spatial encoder in the spatiotemporal feature extraction module. The outputs of the two encoders are added and fed into a 2D convolutional layer for integration, and then added to the output of the saliency basic feature extraction module to obtain the output of the spatiotemporal feature extraction module; the output of the spatiotemporal feature extraction module and the rotation matrix calculated by the motion reconstruction structure are used as the input of the camera motion module, and the output of the camera motion module is fed into the first-person perspective prior module. The output of the first-person perspective prior module is fed into the temporal averaging module, and the result is then fed into the decoder to finally obtain the saliency prediction map. The process of achieving saliency prediction in first-person drone video includes the following steps. Step 1: Obtain the quaternion of camera motion from the video frames to be predicted using the motion reconstruction structure, and further process it to obtain the rotation matrix of each frame, thus preprocessing the video frames. Step 2: Construct a salient basic feature extraction module to extract the basic salient features of video frames; Step 3: Construct a spatiotemporal feature extraction module. In the spatiotemporal feature extraction module, the time encoder extracts the inter-frame temporal features from multiple consecutive video frames, the spatial encoder extracts the spatial features from the video frames, and the temporal features and spatial features are fused to obtain the spatiotemporal features. Step 4: Construct a camera motion module. Input the rotation matrix obtained in Step 1 through the motion recovery structure into the camera motion module to extract camera motion information, and adaptively fuse it with the spatiotemporal features obtained in Step 3. Step 5: Construct a first-person perspective prior module. In the first-person perspective prior module, a first-person perspective prior map is generated by statistically analyzing the data, and the multi-scale prior map is fused with the spatiotemporal saliency features based on camera motion. Step 6: Construct a time-series averaging module to make the predicted saliency map smoother in the time dimension; Step 7: Use the decoder to integrate the saliency features into a single-channel saliency map; Step 8: Train the overall model constructed from steps 2-6; Step 9: Use the trained overall model to predict visual saliency; The implementation method of constructing the salient basic feature extraction module is as follows: An improvement upon the MobileNet-V2 model is made by replacing the final spatial pooling layer with a perforated pyramidal pooling layer to obtain multi-scale features for extracting features of salient objects of different sizes. Then, features from the last three convolutional layers of MobileNet-V2 are used for feature fusion to obtain the salient base features. , where x is the input video frame number.
2. The first-person perspective UAV video saliency prediction method according to claim 1, characterized in that: The spatiotemporal feature extraction module extracts spatiotemporal features in the following way: When video frames Features are obtained after the saliency-based basic feature extraction module. Then, it is input into the spatial encoder, where it is integrated using depthwise separable convolutional layers. Spatial features at different levels of abstraction are used to obtain spatial saliency features. ; when A series of video frames pass A parallel salient basic feature extraction module is input into the temporal encoder, and the temporal encoder then extracts the salient basic feature extraction module's features. The corresponding features are concatenated, and firstly, a convolutional layer is used to reduce the dimensionality. Next, the Bi-Sub operator is used to calculate the difference between the current frame features and the features of the preceding and following frames to obtain the temporal difference features. A final convolutional layer is set to increase the dimensionality of the features, so that the input and output features have the same dimension, and finally outputs the temporal saliency features. , The time encoder outputs share this single output; Calculated time-domain features , i =1,2,… n and spatial domain features Subsequently, feature fusion was performed using element-wise addition, and a 1 × 1 convolutional layer was used to further integrate spatiotemporal features. These features were then superimposed onto the salient base features using residual connections. Spatiotemporal saliency features were obtained. .
3. The first-person perspective UAV video saliency prediction method according to claim 1, characterized in that: In the camera motion module The camera motion features are obtained by passing the rotation matrix of each frame through two consecutive fully connected layers. Furthermore, the spatiotemporal saliency features are globally averaged and pooled in the spatial dimension to obtain pooled features. ; camera motion characteristics With pooling characteristics The layers are stacked along the channel dimension, and a fully connected layer and a sigmoid activation function are used to obtain the weights corresponding to each channel. Finally, the spatiotemporal saliency features are... With weight By multiplying each channel, we obtain the spatiotemporal saliency features based on camera motion. .
4. The first-person perspective UAV video saliency prediction method according to claim 1, characterized in that: In the first-person perspective prior module First, the fixation points in the dataset are statistically analyzed to obtain the mean values of the fixation point position coordinates in the horizontal and vertical directions. and variance Based on the statistical results, eight prior images with the same mean but different variances are generated. Two 2D convolutional layers are used to integrate the prior images, which are then superimposed on the spatiotemporal saliency features based on camera motion along the channel dimension. Finally, a single 2D convolutional layer is used to integrate the features, resulting in spatiotemporal saliency features based on camera motion and first-person perspective priors. .
5. The first-person perspective UAV video saliency prediction method according to claim 1, characterized in that: In the time-series averaging module The spatiotemporal saliency features at the current time step and the output of the temporal averaging module at the previous time step are each fed into a convolutional layer, and the outputs of the convolutional layers are summed. Then, the weights are obtained using the sigmoid activation function. W Finally, the weights are multiplied element-wise by the spatiotemporal saliency features at the current moment. The current time-series average module output is obtained by multiplying the two parts element by element with the output of the time-series average module from the previous time step, and then adding the results of the two multiplications together.
6. A first-person perspective UAV video saliency prediction system, characterized in that: This method is used to implement a first-person perspective drone video saliency prediction method as described in any one of claims 1-5.
7. The first-person perspective UAV video saliency prediction system according to claim 6, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the first-person perspective UAV video saliency prediction method as described in any one of claims 1-5.
8. The first-person perspective UAV video saliency prediction system according to claim 6, characterized in that: The method includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements a first-person perspective UAV video saliency prediction method as described in any one of claims 1-5.