A video saliency map generation method, device and storage medium
By using a brain-inspired computer algorithm to calculate and fuse static and dynamic saliency maps separately, the problem of poor interpretability and low generation efficiency in existing technologies is solved, achieving interpretability and high efficiency in saliency map generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINESE INST FOR BRAIN RES BEIJING
- Filing Date
- 2026-01-06
- Publication Date
- 2026-06-05
AI Technical Summary
Existing saliency map generation methods suffer from poor interpretability and low generation efficiency. Deep learning-based algorithms lack interpretability, while traditional computer vision techniques are slow and ineffective.
A brain-inspired computer algorithm is used to calculate static and dynamic saliency maps separately. Then, a modality fusion algorithm is used to calculate the fused saliency map. Floating-point addition and multiplication are used for the operation to reduce computational complexity.
It achieves interpretability and high efficiency in saliency map generation, with simple and fast computation, thus improving the generation speed.
Smart Images
Figure CN122157077A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus and storage medium for generating video saliency maps. Background Technology
[0002] Currently, there are two main technical approaches to saliency map prediction. One approach uses deep neural networks for computation, such as the SalFoM model (SalFoM: Dynamic Saliency Prediction with Video Foundation Models). This model uses an unmasked teacher visual attention model as the encoder to encode features, and then uses a multi-view heterogeneous decoder to extract information at different temporal and spatial scales through convolution, thereby ultimately generating the saliency map of the video.
[0003] Another approach uses traditional computer vision algorithms, with representative examples being saliency detection algorithms and the GBVS algorithm. The saliency detection algorithm calculates the differences in brightness, color, and line orientation between the center and surrounding areas of a single image, ultimately integrating multi-scale center-periphery difference information to create a saliency map. The GBVS algorithm first segments the image into multiple small blocks and calculates histograms for each block. Then, it compares the similarity of each block with other blocks to construct a fully connected graph, where each block represents a node. The weight between two nodes is determined by the product of the Kullback-Leibler divergence distance and the negative exponent of that distance. The initial values of the saliency map are set to be equal at each node. Through iteration over the Markov chain on the fully connected graph, the graph eventually converges to a state where the node with the largest Kullback-Leibler divergence distance to its surroundings has the maximum weight, thus forming the saliency map.
[0004] Existing saliency map generation methods suffer from poor interpretability. Deep learning-based algorithms, in particular, lack interpretability due to their large number of parameters, the inherent difficulty in interpreting matrix calculations, and the inherent complexity of matrix computations. Furthermore, deep learning's extrapolation performance remains poor, and it's challenging to determine whether the target scene falls within the extrapolation range, leading to unreliable generation results. Deep learning-based saliency detection techniques are largely limited to static saliency detection because traditionally, datasets for static saliency detection are far more abundant than those for dynamic video saliency detection. Traditional computer vision techniques are slow and ineffective because they require numerous time-consuming floating-point multiplications and divisions, as well as complex nested operations such as 2D convolutions and graph-based iterative algorithms, resulting in slow generation speeds. Summary of the Invention
[0005] This invention provides a method, device, and storage medium for generating video saliency maps, with the aim of improving the efficiency of saliency map generation.
[0006] In a first aspect, embodiments of the present invention provide a video saliency map generation method, comprising: Based on multiple preset feature channel information of video frames, a Gaussian pyramid containing multiple different scales is constructed for each preset feature channel; wherein, the preset feature channels include brightness, color, and edge; Based on the Gaussian pyramids of the multiple preset feature channels, a first saliency map is determined for each preset feature channel; Based on the brightness information of multiple video frames, motion information in a preset dimension is calculated by inter-frame difference after image displacement at different levels to obtain simulated optical flow; wherein, the preset dimension includes multiple preset directions, multiple preset velocities and multiple preset scales; Based on optical flow-velocity information, a second saliency map of the preset direction and the preset velocity is extracted using the center-edge difference method; A video saliency map is determined based on the static saliency map synthesized from the first saliency map and the dynamic saliency map synthesized from the second saliency map.
[0007] In a second aspect, embodiments of the present invention provide an electronic device, comprising: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video saliency generation method provided in any embodiment of the present invention.
[0008] Thirdly, embodiments of the present invention provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a video saliency generation method as provided in any embodiment of the present invention.
[0009] The present invention provides a video saliency map generation method, device and storage medium that uses a brain-inspired computer algorithm to calculate static saliency maps and dynamic saliency maps respectively, and then calculates the fused saliency map through a modality fusion algorithm. It has complete interpretability and does not require training with a dataset. Moreover, the algorithm mostly uses floating-point addition and multiplication, which makes the operation simple and fast, reduces the computational complexity and improves the operation speed. Attached Figure Description
[0010] Figure 1 This is a flowchart of a video saliency map generation method provided in Embodiment 1 of the present invention; Figure 2This is a schematic diagram of a video saliency generation device provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention; Figure 4 This is a flowchart for generating video saliency maps provided in an embodiment of the present invention. Detailed Implementation
[0011] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.
[0012] Figure 1 This is a flowchart of a video saliency map generation method provided in Embodiment 1 of the present invention. This embodiment is applicable to video analysis to obtain visual saliency maps, and can be applied to application scenarios such as traffic sign design and placement, advertising design, visual target detection, autonomous driving, and research on biological attention. This method can be executed by a video saliency map generation device, which can be implemented by hardware and / or software and is generally integrated into electronic devices, such as computer devices. The method specifically includes: Step 110: Based on the multiple preset feature channel information of the video frame, construct a Gaussian pyramid containing multiple different scales for each preset feature channel.
[0013] The preset feature channels include brightness, color, and edge. The algorithm simulates feature integration theory, first extracting the brightness, color, and edge feature channel information of the current video frame, then filtering and constructing a Gaussian pyramid for each channel. For example, Gaussian pyramids of different scales with 7 or 8 layers are constructed.
[0014] In one possible implementation, such as Figure 4 The flowchart shown illustrates the generation of a video saliency map. The pre-defined model receives a three-channel image input and then constructs a Gaussian pyramid by applying multiple Gaussian filters and downsampling to the three different feature channels: color, brightness, and orientation. Orientation can be understood as the detection of the direction of lines in the image. For color, values from the four channels (red, green, blue, and yellow) are used. Let these represent the four colors in order, where red, green, and blue are constructed from the RGB channels used in image encoding, and are respectively: The Y channel corresponding to yellow is calculated using the red, green, and blue channels, satisfying the following: In the formula The value indicates a positive result. The lowercase letters r, g, and b represent the RGB channel values of the image input. RGBY represents the calculated pixel's bias or saturation along the red, green, blue, and yellow axes; the larger the value, the more it leans towards the corresponding color. After calculating the color encoding, a corresponding Gaussian pyramid can be constructed. ,in , representing the number of downsampling operations during construction, and then selecting We simulate a center-periphery mechanism to calculate the feature maps corresponding to different channels. 's' represents 'surround,' corresponding to the higher levels of the Gaussian pyramid, and 'c' represents 'center,' corresponding to the lower levels of the Gaussian pyramid. This represents the difference between s and c.
[0015] Step 120: Determine the first saliency map for each preset feature channel based on the Gaussian pyramid of multiple preset feature channels.
[0016] After constructing multiple Gaussian pyramids of different scales, edge information is extracted using the Gaussian pyramids built with brightness information. The edge information extraction uses a kernel-based decomposition algorithm to accelerate the one-dimensional convolution of the image using the Gabor operator.
[0017] Biologically rational bilateral rectified linear units (ReLUs) are used to separate antagonistic visual stimuli. The ReLUs... + and ReLU - The linear rectified functions representing the forward and reverse directions can be used to calculate the channel saliency maps corresponding to red, green, blue, and yellow. The following formula This represents the calculation of the difference across scales after interpolating and scaling the image, resulting in a feature map size equal to the larger of the channels (i.e., at scale 1). (of) consistent.
[0018] in, The feature maps calculated from the two opposing channels (red and green) The feature maps calculated for the two opposing channels (yellow and blue); , , , This represents the saliency map of each individual channel (red, green, blue, and yellow) after ReLU filtering. After calculating the feature maps, the saliency maps of the corresponding color channels can be calculated.
[0019] As for the saliency map of color channels Using nonlinear functions Then it can be calculated that like: Where, in the formula This represents addition across scales, and the final result is consistent with the larger values in the channel. Nonlinear function. It can be: Representing a nonlinear function, the pixel arrangement of a two-dimensional image can be viewed as a two-dimensional matrix, where... This represents the average of all local maxima in a two-dimensional image M. It can be used to calculate normalized channels that highlight global maxima that are significantly higher than other local maxima.
[0020] Define channel and They together satisfy similar antagonistic properties. The channel itself is calculated by taking the average of the r, g, and b channels, so we take a similar value. The difference between the maximum image brightness and the brightness at that point is as follows.
[0021] Defined as such The channel mimics the computation of on / off bipolar cells in the retina, and the luminance feature map can be calculated using a method similar to that used for color feature map computation. and As shown in the following formula.
[0022] The saliency map of the luminance channel can be calculated. like Step 130: Based on the brightness information of multiple video frames, calculate the motion information in the preset dimension through the inter-frame difference after image displacement at different levels to obtain the simulated optical flow.
[0023] The preset dimensions include multiple preset directions, multiple preset speeds, and multiple preset scales. The preset direction is a relative direction, such as up / down or left / right.
[0024] Step 140: Extract the second saliency map with preset direction and preset velocity based on the center-edge difference method according to the optical flow-velocity information.
[0025] The algorithm simulates the computational mechanism of retinal direction-selective cells, extracts brightness information from video frames, and simulates optical flow by calculating motion information at different directions, speeds, and scales through inter-frame difference analysis after displacement. Finally, it extracts a motion saliency map from the optical flow-velocity information using a center-edge difference method. The inter-frame difference analysis using displacement calculates the similarity between two spatially adjacent pixels each time. For example, for a point (x, y), different directions correspond to comparisons with adjacent points (x, y ± 1) and (x ± 1, y); different speeds correspond to the time length between the two frames being searched.
[0026] Video is a sequence of images, with each adjacent image played at a time interval of a specific duration, typically 1 / 24 of a second. Within images of the same scale, shifting them by one pixel maintains the same distance. Using images at different intervals is equivalent to using different time intervals. Distance divided by time equals speed, resulting in different speeds. For example, using adjacent images (separated by one frame), images separated by three frames, images separated by seven frames, etc., will produce different speeds.
[0027] For different scales, the images are processed using a Gaussian pyramid, resulting in approximately I(σ). Each higher-level image corresponds to a 2x downsampling; that is, one pixel in a higher-level image corresponds to a pixel in the lower-level image. Each pixel is used to calculate the displacement-inter-frame difference, which essentially involves moving one pixel at a time. For inputs of different Gaussian pyramid heights (i.e., different scales), the same operation is performed, thus enabling optical flow extraction at different scales.
[0028] Step 150: Determine the video saliency map based on the static saliency map synthesized from the first saliency map and the dynamic saliency map synthesized from the second saliency map.
[0029] The modal alignment method is used to calculate and output the final video saliency map using static and dynamic saliency maps. From the video input, a Gaussian pyramid of different channels is constructed for each new frame and added to the buffer. For each new frame, a static saliency map is calculated, and for the current frame and a few selected previous frames, dynamic saliency maps are calculated. Finally, these are aligned and merged to obtain the video saliency map.
[0030] The technical solution in this embodiment uses a brain-inspired computer algorithm, which is fully interpretable and does not require training with a dataset. The algorithm primarily uses floating-point addition and multiplication, making it simple and fast. Furthermore, it can use one-dimensional convolution instead of two-dimensional convolution, reducing computational complexity and improving speed. Saliency map prediction simulates the bottom-up visual attention process in biology, which is of great significance for applications such as traffic sign design and placement, advertising design, visual object detection, autonomous driving, and research on biological attention.
[0031] Optionally, before constructing a Gaussian pyramid containing multiple different scales for each preset feature channel based on multiple preset feature channel information of the video frame, the following steps are also included: The video frame is acquired, and the feature channel information of the brightness, color, and edges of the video frame is extracted.
[0032] Optionally, based on multiple preset feature channel information of the video frame, a Gaussian pyramid containing multiple different scales is constructed for each preset feature channel, including: For the edge feature channel, edge information is extracted using a Gaussian pyramid constructed based on brightness information using a preset filter.
[0033] Specifically, edge information is extracted based on a Gaussian pyramid constructed from brightness information using a preset filter, including: The edge information is extracted based on the Gabor filter. The kernel decomposition algorithm is used to accelerate the Gabor operator to perform one-dimensional convolution on the brightness information of the video frame to obtain the edge information.
[0034] The specific principle is that the filtering process can be written as follows: This represents the wavelength of the trigonometric function term in the Gabor filter. F Represents the convolution operation. These represent the frequency, direction, and variance of the Gaussian kernel, respectively. Furthermore, the x and y values in this formula can be separated and convolved independently: The separated results along the x-axis and y-axis are convolved separately. The Gabor operator convolution operation based on kernel decomposition is used for... Images and For a two-dimensional convolution operator, the original time complexity is... The two-dimensional convolution algorithm becomes One-dimensional convolution can reduce the amount of computation.
[0035] Optionally, based on the Gaussian pyramids of multiple preset feature channels, determine the first saliency map for each preset feature channel, including: For each preset feature channel, the Gaussian pyramid is calculated using a cross-scale center-edge difference algorithm to obtain the first saliency map corresponding to the preset feature channel.
[0036] After constructing a Gaussian pyramid with multiple channels, a cross-scale center-edge difference algorithm is used to calculate the saliency map of the corresponding feature channels as the first saliency map. The center-edge difference algorithm essentially upsamples the downsampled low-resolution image to the same size as the high-resolution image and then subtracts them, thereby simulating the center-edge mechanism of biological vision and extracting high-frequency and saliency information.
[0037] Optionally, based on the brightness information of multiple video frames, motion information in a preset dimension is calculated through inter-frame differences after image displacement at different levels to obtain simulated optical flow, including: Select adjacent video frames or select multiple video frames with an exponentially increasing interval, and shift the images of different levels of brightness information corresponding to the video frames in a preset direction to obtain a shifted image; Subtracting the displacement image from the hierarchical image determines the image similarity, resulting in simulated optical flow with different preset directions, speeds, and scales.
[0038] The selected video frames can be either two adjacent frames or multiple frames of exponential growth; the former is faster, while the latter can capture less motion information. The inter-frame difference calculation after displacement is a computational simulation of direction-selective cells in biological visual neurons. Similarity is measured by subtracting images from each level of the Gaussian pyramid after displacement by one pixel. Mutual inhibition between vertical and horizontal movements is considered to enhance motion detection capabilities, ultimately outputting saliency maps for different directions.
[0039] In calculating optical flow-velocity information, the algorithm employs a computational method mimicking the orientation-selective cells of the biological retina. Through an improved inter-frame difference method, using only simple matrix addition and multiplication, it can quickly and accurately reconstruct dense optical flow, outperforming traditional dense optical flow algorithms based on solving overdetermined equations. Traditional inter-frame difference algorithms can only detect the presence of motion, while this invention can reconstruct dense optical flow in a computation time comparable to that of traditional inter-frame difference algorithms.
[0040] In calculating optical flow-velocity information, this invention also uses a calculation method with exponential terms spaced multiple frames apart. This allows it to capture slow motion information that traditional optical flow algorithms cannot capture between adjacent frames. This invention also considers the persistence of vision effect; each time the video saliency map is updated, it can be combined with previously generated saliency maps to update the new saliency map, making the saliency map update more stable and preventing noise from affecting the generation of the saliency map.
[0041] In terms of reducing computational load, the element-wise addition and multiplication of matrices used in this invention are already among the simplest operations, and optimizing two-dimensional convolution into one-dimensional convolution has significantly improved computational speed. From the perspective of reducing computational load, it is difficult to find a simpler operation.
[0042] To reconstruct the information of moving optical flow, we first assume that the physical world is continuous and that no large objects suddenly appear or disappear, that is... ( It is the partial derivative of brightness with respect to the x-direction. (This refers to the partial derivative of brightness with respect to the y-direction) It exists almost everywhere. Similar to T4-T5 cells in the retina, it exists in the four directions of up, down, left, and right. For two time intervals of Two frames First, the corresponding Gaussian pyramid of brightness can be calculated using the formula for calculating the brightness pyramid. ,in Considering that human bottom-up vision is not sensitive to long-range motion, the number of layers in the constructed Gaussian pyramid is slightly less than that calculated for static saliency.
[0043] After calculating the brightness Gaussian pyramid, the Gaussian pyramid for each direction and each scale can be calculated, and the directional pyramid can also be calculated. For the directional pyramid with a height of For a given layer, the corresponding speed of motion is equivalent to This allows us to capture information about the point moving at different speeds in different directions.
[0044] After calculating the orientation pyramid, for The corresponding directional feature map can be calculated. Where d can take any of the following values: This represents two opposite directions. Ultimately, this applies to different center-periphery differences, different directions, and different time differences. After performing nonlinear processing, linear summation yields the dynamic significance map. ,in Representing the frame distance between frames, empirically using an exponential distance, such as... The results were quite good. Furthermore, a larger frame interval and different Gaussian downsampling times helped the model capture minute movements.
[0045] While this method of calculating motion saliency is slightly more computationally challenging than simple inter-frame differencing, it completely reconstructs optical flow information. Furthermore, the computation is far simpler than iterative methods, allowing for real-time processing. Another advantage of this computational simplicity is its ability to process motion information from multiple time windows simultaneously, thus overcoming the limitations of fixed distance and time windows in the Hassenstein-Reichardt and Barlow-Levick models, and further enhancing its ability to capture minute motions.
[0046] Optionally, a video saliency map is determined based on a static saliency map synthesized from a first saliency map and a dynamic saliency map synthesized from a second saliency map, including: Normalization is performed on each first and second saliency map to highlight the single maximum value, thus ensuring that the strongest global maximum value is highlighted.
[0047] The static saliency map is calculated by summing the first saliency map, and the dynamic saliency map is calculated by summing the second saliency map. The video saliency map is obtained by multiplying the static saliency map and the dynamic saliency map based on preset fitting parameters and adding the static saliency map and the dynamic saliency map together.
[0048] Specifically, based on preset fitting parameters, the video saliency map is obtained by multiplying the static saliency map and the dynamic saliency map, and by adding the static saliency map and the dynamic saliency map together. This includes: The video saliency map is obtained using the following method: Based on the static saliency map and dynamic saliency map The final generated video saliency map ,but: Where a and b are pre-computed empirical parameters that best perform on the public dataset DHF1K.
[0049] Based on the properties of the center-periphery operator and neurophysiological studies in the retina, primary visual cortex, and superior colliculus, it can be inferred that the fusion saliency map calculated by the model has a strong encoding ability for edges. If we continue to use the "winner-take-all" method to predict the saliency map, then our predicted bottom-up visual attention should be projected onto the edges rather than the center of the object. However, some researchers in the field believe that focusing on the center of the object is more biologically meaningful, on the one hand, focusing on the object itself is conducive to maximizing information; on the other hand, focusing on the object may be more meaningful in evolution. The significance of bottom-up visual saliency may lie in finding food, avoiding predators, and engaging in group activities, all of which require the perception of an object. To balance winner-take-all and information maximization, we introduce a greedy method of using multiple Gaussian kernels for fitting in the model.
[0050] After the saliency map is calculated, the proposed greedy multi-Gaussian peak fitting algorithm is used for further fitting. Each time, the position with the highest probability density in the saliency map is greedily selected, which is the "winner-takes-all" position. Establish a two-dimensional Gaussian distribution And based on this two-dimensional Gaussian distribution and the saliency map after fusion. By performing convolution, the convolution result can be calculated. The result is obtained by summing the convolution results pixel by pixel. ,by for Calculating the numerical partial derivatives separately yields the following formula (to prevent numerical issues from causing the variance to be too small, for...). Additional penalty terms can be calculated using empirical parameters. ).
[0051] After iterating a certain number of times as described above, assuming the Gaussian kernel obtained through iteration is close to optimal, the convolution result of the Gaussian kernel and the original image at the normalized optimal time is subtracted from the original image, and the remaining maximum value is calculated to see if it is lower than a certain threshold. If it is lower than the threshold, the detection stops; if it is still higher than the threshold, the point with the strongest intensity is greedily selected, and the iterative calculation described above is continued.
[0052] By iterating k times, a Gaussian distribution can be calculated. Calculate the sum of all Gaussian distributions (based on the maximum value of the original significance map at the start of the iteration). The final significance prediction result is calculated by normalizing the coefficients to the range of 0-255. .
[0053] Exponentially Weighted Moving Average (EWMA) is a simple and effective smoothing method. Each time, a new smoothing is calculated as follows: Time saliency plot results At that time, calculate For the final output, where ,in It is an exponentially smoothing parameter; for example, empirically, a value of 0.05 works well.
[0054] A schematic diagram of a video saliency generation device provided in an embodiment of the present invention is shown below. Figure 2 As shown, the video saliency map generation device includes: a Gaussian pyramid construction module 210, a first saliency map determination module 220, a simulated optical flow determination module 230, a second saliency map determination module 240, and a video saliency map generation module 250.
[0055] The Gaussian pyramid construction module 210 is used to construct a Gaussian pyramid containing multiple different scales for each preset feature channel based on multiple preset feature channel information of the video frame; wherein, the preset feature channels include brightness, color and edge; The first saliency map determination module 220 is used to determine the first saliency map of each preset feature channel based on the Gaussian pyramid of multiple preset feature channels. The simulated optical flow determination module 230 is used to calculate motion information in a preset dimension based on the brightness information of multiple video frames and the inter-frame difference after image displacement at different levels to obtain the simulated optical flow; wherein, the preset dimension includes multiple preset directions, multiple preset velocities and multiple preset scales; The second saliency map determination module 240 is used to extract a second saliency map with a preset direction and a preset velocity based on the center-edge difference method according to the optical flow-velocity information. The video saliency map generation module 250 is used to determine the video saliency map based on the static saliency map synthesized from the first saliency map and the dynamic saliency map synthesized from the second saliency map.
[0056] Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device includes a processor 310, a memory 320, an input device 330, and an output device 340; the number of processors 310 in the electronic device can be one or more. Figure 3 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, and output device 340 in the electronic device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.
[0057] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the video saliency generation method in this embodiment of the invention (e.g., the Gaussian pyramid construction module 210, the first saliency determination module 220, the analog optical flow determination module 230, the second saliency determination module 240, and the video saliency generation module 250 in the video saliency generation device). The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby implementing the aforementioned video saliency generation method.
[0058] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0059] Input device 330 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 340 may include display devices such as a display screen.
[0060] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a video saliency generation method, including: Based on multiple preset feature channel information of video frames, a Gaussian pyramid containing multiple different scales is constructed for each preset feature channel; wherein, the preset feature channels include brightness, color and edge; Based on the Gaussian pyramids of multiple preset feature channels, determine the first saliency map for each preset feature channel; Based on the brightness information of multiple video frames, motion information under a preset dimension is calculated by inter-frame difference after image displacement at different levels to obtain simulated optical flow; wherein, the preset dimension includes multiple preset directions, multiple preset velocities and multiple preset scales; Based on optical flow-velocity information, a second saliency map with a preset direction and preset velocity is extracted using the center-edge difference method; The video saliency map is determined based on the static saliency map synthesized from the first saliency map and the dynamic saliency map synthesized from the second saliency map.
[0061] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the video saliency generation method provided in any embodiment of the present invention.
[0062] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0063] It is worth noting that in the embodiments of the video salience generation device described above, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.
[0064] Although the present invention has been described in detail above with general description, specific embodiments, and experiments, some modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.
Claims
1. A method for generating video saliency maps, characterized in that, include: Based on multiple preset feature channel information of video frames, a Gaussian pyramid containing multiple different scales is constructed for each preset feature channel; wherein, the preset feature channels include brightness, color, and edge; Based on the Gaussian pyramids of the multiple preset feature channels, a first saliency map is determined for each preset feature channel; Based on the brightness information of multiple video frames, motion information in a preset dimension is calculated by inter-frame difference after image displacement at different levels to obtain simulated optical flow; wherein, the preset dimension includes multiple preset directions, multiple preset velocities and multiple preset scales; Based on optical flow-velocity information, a second saliency map of the preset direction and the preset velocity is extracted using the center-edge difference method; A video saliency map is determined based on the static saliency map synthesized from the first saliency map and the dynamic saliency map synthesized from the second saliency map.
2. The method according to claim 1, characterized in that, Before constructing a Gaussian pyramid containing multiple different scales for each of the preset feature channels based on the multiple preset feature channel information of the video frame, the method further includes: The video frame is acquired, and the feature channel information of the brightness, color, and edges of the video frame is extracted.
3. The method according to claim 1, characterized in that, The step of constructing a Gaussian pyramid containing multiple different scales for each preset feature channel based on multiple preset feature channel information of video frames includes: For the edge feature channel, edge information is extracted using a Gaussian pyramid constructed based on brightness information using a preset filter.
4. The method according to claim 3, characterized in that, The method of extracting edge information based on a Gaussian pyramid constructed from brightness information using a preset filter includes: The edge information is extracted based on the Gabor filter. The kernel decomposition algorithm is used to accelerate the Gabor operator to perform one-dimensional convolution on the brightness information of the video frame to obtain the edge information.
5. The method according to claim 1, characterized in that, The step of determining the first saliency map of each of the preset feature channels based on the Gaussian pyramids of the multiple preset feature channels includes: For each of the preset feature channels, the Gaussian pyramid is used to calculate the first saliency map corresponding to the preset feature channel based on the cross-scale center-edge difference algorithm.
6. The method according to claim 1, characterized in that, The step of calculating motion information in a preset dimension based on the brightness information of multiple video frames and through inter-frame difference calculation after image displacement at different levels to obtain simulated optical flow includes: Select adjacent video frames or select multiple video frames with an exponentially increasing interval, and shift the images of different levels of brightness information corresponding to the video frames in a preset direction to obtain a shifted image; Subtracting the displacement image from the hierarchical image determines the image similarity, resulting in simulated optical flow with different preset directions, speeds, and scales.
7. The method according to claim 1, characterized in that, Determining the video saliency map based on the static saliency map synthesized from the first saliency map and the dynamic saliency map synthesized from the second saliency map includes: For each of the first and second saliency maps, normalization is performed to highlight the single maximum value; The static saliency map is calculated by summing the first saliency map, and the dynamic saliency map is calculated by summing the second saliency map; The video saliency map is obtained by multiplying the static saliency map and the dynamic saliency map based on preset fitting parameters and adding the static saliency map and the dynamic saliency map together.
8. The method according to claim 7, characterized in that, The process of obtaining the video saliency map by multiplying the static saliency map and the dynamic saliency map based on preset fitting parameters and adding the static saliency map and the dynamic saliency map together includes: The video saliency map is obtained in the following manner: Let the static saliency map and the dynamic saliency map be denoted as follows: The final generated video saliency map is denoted as ,but: Where a and b are preset fitting parameters.
9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video saliency map generation method as described in any one of claims 1-8.
10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the video saliency generation method as described in any one of claims 1-8.