Video shadow detection method and device, equipment and storage medium

Abstract

The invention discloses a video shadow detection method, device and equipment and a storage medium, and the method comprises the steps: inputting a to-be-detected video sequence into a trained diffusion model, obtaining a video shadow detection result, and achieving the real-time performance of shadow detection, and a training process comprises the steps: carrying out the time sequence aggregation of the video sequence, obtaining an aggregation feature, and carrying out the detection of the video shadow. Extracting a corresponding shadow area prior according to the aggregation features; according to shadow area prior, boundary features are obtained through a self-attention mechanism; extracting an aggregation feature and a prior guide feature of the shadow region by using a guide encoder; then, splicing the boundary features and the guide features to obtain spliced features, inputting the spliced features into a pre-constructed initial diffusion model to obtain a trained diffusion model, and respectively performing representation learning on the shadow in time and content through guidance in a time domain and a shadow boundary mode to obtain a shadow feature; therefore, the diffusion model is guided to execute shadow detection on the video, so that the accuracy of video shadow detection is improved.

Description

Technical Field

[0001] The present application relates to the field of video shadow detection, and in particular to a video shadow detection method, device, equipment and storage medium. Background Art

[0002] Video shadow detection not only plays an important role in photography equipment, but also has wide demand in many practical application scenarios, such as intelligent monitoring, augmented reality, autonomous driving, etc. In these applications, accurate shadow detection is crucial for key steps such as target tracking, scene understanding, and image processing.

[0003] Existing video shadow detection methods mainly rely on the mainstream architecture of convolutional networks (CNN). However, in practical applications, CNN mainly processes based on single-frame image features, which makes it difficult to fully utilize the dynamic change information between frames in the video. In addition, due to the changeable shadow forms, a single CNN model is difficult to fully learn the features of different types of shadows; therefore, existing video shadow detection methods cannot fully utilize the dynamic change information between frames in the video, resulting in low accuracy of video shadow detection. Summary of the invention

[0004] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application proposes a video shadow detection method, device, equipment and storage medium, which can improve the accuracy of video shadow detection.

[0005] A first aspect of an embodiment of the present application provides a video shadow detection method, the method comprising:

[0006] Obtain a video sequence to be detected;

[0007] The video sequence to be detected is input into the trained diffusion model to obtain the video shadow detection result of the video sequence to be detected; wherein the training process of the diffusion model includes:

[0008] Obtain a video sequence training set;

[0009] Performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features;

[0010] Extracting a corresponding shadow region prior based on the aggregated features;

[0011] Based on the shadow area prior, boundary features are obtained through a self-attention mechanism;

[0012] Extracting the aggregated features and the shadow region prior guided features using a guided encoder;

[0013] Splicing the boundary feature and the guide feature to obtain a spliced ​​feature;

[0014] An initial diffusion model is constructed, and the concatenated features are input into the initial diffusion model to obtain the trained diffusion model.

[0015] The embodiment of the present application provides a video shadow detection method, which inputs a video sequence to be detected into a trained diffusion model to obtain a video shadow detection result, thereby realizing the real-time performance of shadow detection. The training process is to first perform time-series aggregation on the video sequence to obtain aggregated features, and extract the corresponding shadow area prior based on the aggregated features; based on the shadow area prior, obtain boundary features through a self-attention mechanism; use a guided encoder to extract the aggregated features and guided features of the shadow area prior; then splice the boundary features and the guided features to obtain spliced ​​features, and input the spliced ​​features into a pre-constructed initial diffusion model to obtain a trained diffusion model, and guide the shadows in time and content respectively by guiding them in the time domain and the shadow boundary, thereby guiding the diffusion model to perform shadow detection on the video, thereby improving the accuracy of video shadow detection.

[0016] In some embodiments of the present application, performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features includes:

[0017] Extracting deep learning features of video sequences in the video sequence training set through a backbone network encoder;

[0018] The deep learning features of the video sequences in the video sequence training set are input into a dual-scale feature module for temporal aggregation to obtain the aggregated features.

[0019] In some embodiments of the present application, the deep learning features of the video sequences in the video sequence training set are input into the dual-scale feature module for temporal aggregation to obtain the aggregated features, including:

[0020] Dividing the video sequence into a near-distance video frame and a far-distance video frame according to a current video frame, wherein the near-distance video frame includes two video frames adjacent to the current video frame, and the far-distance video frame includes the remaining video frames except the current video frame and the near-distance video frame;

[0021] Determine the deep learning features corresponding to the close-range video frame, and determine the deep learning features corresponding to the long-range video frame;

[0022] Calculate a first initial matrix according to the deep learning features corresponding to the close-range video frame;

[0023] Obtaining a close-range feature by multiplying the first initial matrix with the deep learning feature corresponding to the close-range video frame;

[0024] Calculating a residual matrix based on the deep learning features corresponding to the long-distance video frame;

[0025] Obtaining a long-range feature by multiplying the residual matrix with the deep learning feature corresponding to the long-range video frame;

[0026] A convolution operation is performed on the short-range feature and the long-range feature to obtain the aggregated feature.

[0027] In some embodiments of the present application, the formula for obtaining the boundary feature through the self-attention mechanism based on the shadow area prior includes:

[0028]

[0029]

[0030]

[0031]

[0032] Among them, q, k, v are the parameters in the self-attention mechanism, W q , W k , W v are learnable parameters, is the predicted mask, F' T is the aggregation feature, C is the bias parameter, SABB is the output result of the self-attention mechanism, Softmax is a mathematical function, and tr is the transposition symbol.

[0033] In some embodiments of the present application, the step of constructing an initial diffusion model and inputting the concatenated features into the initial diffusion model to obtain the trained diffusion model includes:

[0034] Inputting the spliced ​​features into the initial diffusion model for forward diffusion and reverse diffusion;

[0035] The diffused concatenated features are decoded using a decoder to obtain the trained diffusion model.

[0036] In some embodiments of the present application, the step of inputting the spliced ​​features into the initial diffusion model for forward diffusion and reverse diffusion includes:

[0037] The diffusion model implements the forward diffusion process of the spliced ​​features through the following formula:

[0038]

[0039] Where t is the time step, y 0 is the observed value of the initial state, y tis the observation value at time step t, ∈ is Gaussian noise, is the noise control function;

[0040] The reverse diffusion process of the spliced ​​features is achieved by the following formula:

[0041] p θ (y 0:t |g,x)=p(y t )∏p θ (y t-1 |y t ,g,x)

[0042] Among them, y 0:t is all the observations from step 0 to step t, x is the preset conditional variable, g is the concatenated feature, θ is the parameterized network, y t-1 is the observed value at time step t-1, and p() is the probability density function.

[0043] In some embodiments of the present application, the loss function of the diffusion model is:

[0044]

[0045] Among them, L bce represents the ambiguous cross entropy function, is the pseudo label mask, y is the real label, To predict the mask, is the boundary mask, and b is the true value of the boundary mask.

[0046] A second aspect of an embodiment of the present invention provides a video shadow detection device, the device comprising:

[0047] A data acquisition module, used to acquire a video sequence to be detected;

[0048] The shadow detection module inputs the video sequence to be detected into the trained diffusion model to obtain the video shadow detection result of the video sequence to be detected; wherein the training process of the diffusion model includes:

[0049] Obtain a video sequence training set;

[0050] Performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features;

[0051] Extracting a corresponding shadow region prior based on the aggregated features;

[0052] Based on the shadow area prior, boundary features are obtained through a self-attention mechanism;

[0053] Extracting the aggregated features and the shadow region prior guided features using a guided encoder;

[0054] Splicing the boundary feature and the guide feature to obtain a spliced ​​feature;

[0055] An initial diffusion model is constructed, and the concatenated features are input into the initial diffusion model to obtain the trained diffusion model.

[0056] A third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned video shadow detection method.

[0057] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned video shadow detection method.

[0058] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0060] Figure 1 It is a flowchart of a video shadow detection method provided in an embodiment of the present application;

[0061] Figure 2 It is a flowchart of a training process of a diffusion model of a video shadow detection method provided in an embodiment of the present application;

[0062] Figure 3 It is a schematic diagram of a process of performing time-series aggregation on a video sequence in a video shadow detection method provided in an embodiment of the present application;

[0063] Figure 4 It is a flow chart of a video shadow detection method provided in an embodiment of the present application, which is based on shadow area prior and obtains boundary features through a self-attention mechanism;

[0064] Figure 5 is a visual comparison diagram of a video shadow detection method provided by an embodiment of the present application and other detection methods;

[0065] Figure 6 is a structural schematic diagram of a video shadow detection device provided in an embodiment of the present application;

[0066] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0068] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0069] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0070] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0071] Video shadow detection not only plays an important role in photographic equipment, but also has wide demand in many practical application scenarios, such as intelligent monitoring, augmented reality, and autonomous driving. In these applications, accurate shadow detection is crucial for key steps such as target tracking, scene understanding, and image processing. At the same time, shadow detection in video scenes also faces more challenges, such as dynamic backgrounds and complex lighting conditions. Therefore, studying efficient and robust video shadow detection technology can not only improve the imaging quality of equipment, but also promote the further development of related applications, which is of great significance to the fields of computer vision and image processing.

[0072] Previous shadow detection methods were mostly based on images and relied on a large number of experimental prior settings and physical characteristics of shadows, such as manually setting thresholds for shadow area segmentation. However, considering the actual application scenarios and efficiency, most of these methods are not used for video shadow detection. In recent years, automated detection methods based on deep learning have gradually replaced manual feature extraction algorithms. Some work is still trained on single-frame images, which leads to the neglect of the temporal correlation between frames in the video and the inability to fully utilize the temporal information of the video. In the video field, existing methods still rely on two mainstream architectures: convolutional networks (CNNs) and Transformers (a deep learning model). However, in practical applications, CNNs are mainly based on single-frame image features for processing, and it is difficult to fully utilize the dynamic change information between frames in the video. In addition, due to the changeable shadow morphology, a single CNN model is difficult to fully learn the features of different types of shadows; while the Transformer model requires a lot of computing resources to process long sequences of video data, and there are challenges in real-time performance. Its core mechanism, self-attention, is better at modeling global dependencies, but its ability to model local visual features is limited.

[0073] In order to solve the above technical defects, refer to Figure 1 , the embodiment of the present application provides a video shadow detection method, comprising the following steps S101 to S102:

[0074] Step S101: Obtain a video sequence to be detected.

[0075] Step S102: input the video sequence to be detected into the trained diffusion model to obtain the video shadow detection result of the video sequence to be detected; wherein the training process of the diffusion model includes:

[0076] Obtain a video sequence training set;

[0077] Perform temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features;

[0078] Extract the corresponding shadow area prior based on the aggregated features;

[0079] Based on the shadow area prior, the boundary features are obtained through the self-attention mechanism;

[0080] The guided encoder is used to extract the aggregated features and the guided features of the shadow area prior;

[0081] The boundary feature and the guide feature are spliced ​​to obtain a spliced ​​feature;

[0082] Construct an initial diffusion model, input the concatenated features into the initial diffusion model, and obtain a trained diffusion model.

[0083] The following is an explanation of step S102:

[0084] like Figure 2 As shown, the video sequence in the video sequence training set in the embodiment of the present application is represented by {x T-i ,…,x T ,…,x T+i}, deep learning features are obtained through the shared encoder E, and the dual-scale aggregation module (DSA) is used to index and match the video features of the past frame-future frame to generate aggregate features, so that the model can learn the shadow consistency content in the entire time domain; based on the aggregate features, the corresponding shadow area prior is obtained through the multi-head decoder (Auxiliary Head), where the shadow area prior includes a predicted pseudo mask and a boundary mask, and the shadow boundary perception attention module (SBAA, Shadow Boundary-Aware Attention) is used to guide the boundary information and obtain the boundary features; finally, the diffusion process is executed to combine the aggregate features generated by the DSA mentioned above, generate guided features through the guided encoder encoding, and splice the guided features and boundary features, and use the spliced ​​features to guide the diffusion model to iteratively denoise and predict video shadows. It should be noted that Figure 2 Only the flowchart for predicting the Tth frame is shown. In the application, all video sequences can be predicted at the same time.

[0085] The diffusion model is a generative model that generates new samples by gradually adding noise to the data and learning how to reverse the process (i.e., denoising). The key feature of the diffusion model is that it can produce high-quality data samples and does not require complex likelihood estimation during training. It usually works by defining a forward diffusion and reverse diffusion. Forward diffusion refers to gradually replacing real data with noise. The reverse process is the learning process of the diffusion model, which aims to predict and remove the added noise at each time step in order to reconstruct the original data from the noise.

[0086] The self-attention mechanism was originally proposed in the field of natural language processing (NLP) for processing sequence data, such as sentences. In computer vision, the self-attention mechanism is used to capture long-distance dependencies in images and enhance the model's understanding of the global context; the self-attention module usually contains three parts: query, key, and value, which work together to calculate the attention weights, and then use these weights to perform a weighted summation on the input to obtain a more focused feature representation. The self-attention mechanism in the embodiment of the present application is a self-attention mechanism for boundary learning. By adding a decoder to learn the shadow boundary content, the model is guided to better distinguish between shadow and non-shadow areas to improve the accuracy of shadow detection.

[0087] The embodiment of the present application inputs the video sequence to be detected into a trained diffusion model to obtain a video shadow detection result, thereby realizing the real-time performance of shadow detection. The training process is to first perform time-series aggregation on the video sequence to obtain aggregated features, and extract the corresponding shadow area prior based on the aggregated features; based on the shadow area prior, obtain the boundary features through the self-attention mechanism; use the guided encoder to extract the aggregated features and the guided features of the shadow area prior; then splice the boundary features and the guided features to obtain the spliced ​​features, and input the spliced ​​features into the pre-constructed initial diffusion model to obtain the trained diffusion model, and guide the shadows in the time domain and the shadow boundary in two ways, respectively, so as to guide the diffusion model to perform shadow detection on the video, so as to improve the accuracy of video shadow detection.

[0088] In some embodiments, performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features includes the following steps S201 to S202:

[0089] Step S201, extracting deep learning features of video sequences in a video sequence training set through a backbone network encoder;

[0090] Step S202: input the deep learning features of the video sequences in the video sequence training set into the dual-scale feature module for temporal aggregation to obtain the aggregated features.

[0091] In this embodiment, for the video sequence {x T-i ,…,x T ,…,x T+i}, extract deep learning features of video sequences through backbone network editor E (such as ResNet, Vision Transformer) {F T-i ,…,F T ,…,F T+i}, where the backbone network is a general feature extractor.

[0092] The step S202 of inputting the deep learning features of the video sequence in the video sequence training set into the dual-scale feature module for temporal aggregation to obtain the aggregated features includes the following steps S301 to S307:

[0093] Step S301: dividing the video sequence into close-range video frames and long-range video frames according to the current video frame, wherein the close-range video frame includes two video frames adjacent to the current video frame, and the long-range video frame includes the remaining video frames except the current video frame and the close-range video frame;

[0094] Step S302: determining the deep learning features corresponding to the close-range video frame, and determining the deep learning features corresponding to the long-range video frame;

[0095] Step S303, calculating a first initial matrix according to the deep learning features corresponding to the close-range video frame;

[0096] Step S304, multiplying the first initial matrix by the deep learning feature corresponding to the close-range video frame to obtain a close-range feature;

[0097] Step S305, calculating a residual matrix according to the deep learning features corresponding to the long-distance video frame;

[0098] Step S306, obtaining a long-range feature by multiplying the residual matrix with the deep learning feature corresponding to the long-range video frame;

[0099] Step S307: performing a convolution operation on the short-range feature and the long-range feature to obtain the aggregated feature.

[0100] In this embodiment, if Figure 3 As shown, the deep learning features of different video frames obtained above {F T-i ,…,F T ,…,F T+i} is input into the DSA module (Dual Scale Aggregation) for temporal aggregation, and the video frames are pre-divided into close-range video frames and long-range video frames, where the close-range video frames are the two adjacent frames of the current frame. T-1 ,F T+1}, the distant frame is the video frame in the video sequence excluding the current frame and the near video frame {F T-i ,…,F T-2 ,F T+2 …,F T+i}, initial aggregation is used for close-range video frames, and residual aggregation is used for long-range video frames, specifically:

[0101] by Figure 3 As an example, the video sequence is encoded to obtain deep learning features {F T-2 ,F T-1 ,F T ,F T+1 ,F T+2};

[0102] For the close-range video frame, aggregate it with the current video frame and first calculate the first initial matrix W by formula (1): s , then through the first initial matrix W s The close-range feature is obtained by multiplying the corresponding deep learning feature in the close-range video frame.

[0103]

[0104] Among them, x represents the location coordinate index of the feature, and f is the L2 function.

[0105] For the long-distance video frame, the short-distance video frame in formula (1) is replaced by the long-distance video frame to obtain the second initial matrix W′, and the short-distance video frame in formula (1) is replaced by the current video frame to obtain the identity matrix W″, as shown in formula (2). The residual matrix is ​​calculated by the second initial matrix and the identity matrix, and the obtained residual matrix is ​​multiplied by the corresponding deep learning feature in the long-distance video frame to obtain the long-distance feature.

[0106] W l =||W″-W′|| (2)

[0107] All the obtained short-range features and long-range features are fused, and the fused features are input into the residual fusion module (i.e., the fused features are simply connected and convolved) to obtain the aggregated features. This is repeated, and all video frames are aggregated and the aggregated features are output.

[0108] In some embodiments, after the DSA module outputs the aggregated features, the aggregated features are input into the Auxiliary Head decoder for initial decoding to obtain a rough shadow area prior, wherein the shadow area prior includes a boundary mask and pseudo label mask

[0109] In some embodiments, Figure 4 As shown, the shadow area prior is input into SBAA (Shadow Boundary-Aware Attention) to perform the boundary-guided self-attention mechanism.

[0110] Based on the shadow area prior, the formula for obtaining boundary features through the self-attention mechanism includes:

[0111] q=F T 'W q (3)

[0112]

[0113]

[0114]

[0115] Among them, q, k, v are the parameters in the self-attention mechanism, W q , Wk , W v are learnable parameters, is the predicted mask, F' T is the aggregation feature, C is the bias parameter, which is set to 196 by default, SABB is the output result of the self-attention mechanism, Softmax is a mathematical function, and tr is the transposition symbol.

[0116] In this embodiment, the SBAA module is used to emphasize boundary learning and output boundary features to achieve the purpose of boundary guidance.

[0117] In some embodiments, all frames and pseudo-label masks in the time domain, i.e., past frames-future frames (aggregated features and shadow area priors), are input into the guidance encoder GE together to obtain guidance features, and the guidance features are spliced ​​with the boundary features output by the SBAA module to obtain spliced ​​features; wherein the guidance features are time domain features to achieve the purpose of time domain guidance.

[0118] In some embodiments, constructing an initial diffusion model, inputting the concatenated features into the initial diffusion model, and obtaining a trained diffusion model includes the following steps S401 to S402:

[0119] Step S401, input the spliced ​​features into the initial diffusion model for forward diffusion and reverse diffusion;

[0120] Step S402: Use a decoder to decode the diffused and spliced ​​features to obtain a trained diffusion model.

[0121] In this embodiment, when the spliced ​​features are input into the initial diffusion model, forward diffusion is performed first, as shown in formula (7), and then reverse diffusion is performed, as shown in formula (8). By performing the forward diffusion and reverse diffusion processes respectively, the trained diffusion model can give all shadow prediction results of the video frame by frame through the final decoder D.

[0122]

[0123] p θ (y 0:t |g,x)=p(y t )∏p θ (y t-1 |y t ,g,x) (8)

[0124] Where t is the time step, y 0 is the observed value of the initial state, y t is the observation value at time step t, ∈ is Gaussian noise, is the noise control function, y 0:tis all the observations from step 0 to step t, x is the preset conditional variable, g is the concatenated feature, θ is the parameterized network, y t-1 is the observed value at time step t-1, and p() is the probability density function.

[0125] In some embodiments, the training loss function of the diffusion model of the embodiment of the present application is shown in formula (9):

[0126]

[0127] Among them, L bce represents the ambiguous cross entropy function, is the pseudo label mask, y is the real label, To predict the mask, is the boundary mask, and b is the true value of the boundary mask.

[0128] As shown in Table 1, Table 1 is a quantitative comparison table of the video shadow detection method provided by the embodiment of the present application and the existing detection method. In Table 1, the mean error (MAE), F value (F β ), intersection over union (IoU), equalization error rate (BER), shadow equalization error rate (S-BER) and non-shadow equalization error rate (N-BER), where bold indicates the best result and underline indicates the suboptimal result. According to Table 1, the embodiment of the present application has currently achieved the best results on the public data set, exceeding the existing shadow detection method; under the same benchmark test and on the public data set ViSha, the video shadow detection method proposed in the embodiment of the present application exceeds the optimal algorithm by 2.27% in the segmentation detection effect indicator intersection over union (IoU). The embodiment of the present application can perform real-time shadow detection on long sequence videos: shadow detection can be performed under longer video settings (such as 100 frames), and it can flexibly cope with different video lengths. In addition, the video shadow detection method proposed in the embodiment of the present application has good mobility and adaptability, and can be applied to a variety of video surveillance scenarios, such as security monitoring, automatic driving, etc.

[0129]

[0130] Table 1

[0131] like Figure 5 As shown, Figure 5 From the visual comparison diagram of the video shadow detection method provided in the embodiment of the present application and other detection methods, it can be clearly seen that the video shadow detection method provided in the embodiment of the present application can better distinguish the boundary area of ​​the shadow, and the prediction result is basically similar to the actual annotation result (b).

[0132] like Figure 6As shown, some embodiments of the present application provide a video shadow detection device, the device includes a data acquisition module 1100 and a shadow detection module 1200, specifically:

[0133] The data acquisition unit 1100 is used to acquire a video sequence to be detected;

[0134] The shadow detection module 1200 is used to input the video sequence to be detected into the trained diffusion model to obtain the video shadow detection result of the video sequence to be detected; wherein the training process of the diffusion model includes:

[0135] Obtain a video sequence training set;

[0136] Performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features;

[0137] Extracting a corresponding shadow region prior based on the aggregated features;

[0138] Based on the shadow area prior, boundary features are obtained through a self-attention mechanism;

[0139] Extracting the aggregated features and the shadow region prior guided features using a guided encoder;

[0140] Splicing the boundary feature and the guide feature to obtain a spliced ​​feature;

[0141] An initial diffusion model is constructed, and the concatenated features are input into the initial diffusion model to obtain the trained diffusion model.

[0142] It should be noted that the video shadow detection device provided in this embodiment and the above-mentioned video shadow detection method are based on the same inventive concept, so the relevant content of the above-mentioned video shadow detection method is also applicable to the content of a video shadow detection device, and therefore, it will not be repeated here.

[0143] like Figure 7 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0144] at least one memory;

[0145] at least one processor;

[0146] at least one program;

[0147] The program is stored in the memory, and the processor executes at least one program to implement the video shadow detection method described above in the present disclosure.

[0148] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0149] The electronic device according to the embodiment of the present application is described in detail below.

[0150] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0151] The memory 1700 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the program code is stored in the memory 1700, and the processor 1600 calls and executes a video shadow detection method in an embodiment of the present invention.

[0152] Input / output interface 1800, used to implement information input and output;

[0153] The communication interface 1900 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0154] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0155] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0156] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned video shadow detection method.

[0157] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0159] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0160] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0162] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0163] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0164] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0168] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A video shadow detection method, characterized in that: The method comprises: Obtain a video sequence to be detected; The video sequence to be detected is input into the trained diffusion model to obtain the video shadow detection result of the video sequence to be detected; wherein the training process of the diffusion model includes: Obtain a video sequence training set; Performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features; Extracting a corresponding shadow region prior based on the aggregated features; Based on the shadow area prior, boundary features are obtained through a self-attention mechanism; Extracting the aggregated features and the shadow region prior guided features using a guided encoder; Splicing the boundary feature and the guide feature to obtain a spliced ​​feature; An initial diffusion model is constructed, and the concatenated features are input into the initial diffusion model to obtain the trained diffusion model.

2. The video shadow detection method according to claim 1, characterized in that: The step of performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features includes: Extracting deep learning features of video sequences in the video sequence training set through a backbone network encoder; The deep learning features of the video sequences in the video sequence training set are input into a dual-scale feature module for temporal aggregation to obtain the aggregated features.

3. The video shadow detection method according to claim 2, characterized in that: The step of inputting the deep learning features into a dual-scale feature module for time series aggregation to obtain the aggregated features includes: Dividing the video sequence into a near-distance video frame and a far-distance video frame according to a current video frame, wherein the near-distance video frame includes two video frames adjacent to the current video frame, and the far-distance video frame includes the remaining video frames except the current video frame and the near-distance video frame; Determine the deep learning features corresponding to the close-range video frame, and determine the deep learning features corresponding to the long-range video frame; Calculate a first initial matrix according to the deep learning features corresponding to the close-range video frame; Obtaining a close-range feature by multiplying the first initial matrix with the deep learning feature corresponding to the close-range video frame; Calculating a residual matrix based on the deep learning features corresponding to the long-distance video frame; Obtaining a long-range feature by multiplying the residual matrix with the deep learning feature corresponding to the long-range video frame; A convolution operation is performed on the short-range feature and the long-range feature to obtain the aggregated feature.

4. The video shadow detection method according to claim 1, characterized in that: The formula for obtaining the boundary feature based on the shadow area prior through the self-attention mechanism includes: q=F T ′W q Among them, q, k, v are the parameters in the self-attention mechanism, W q , W k , W v are learnable parameters, is the predicted mask, F' T is the aggregation feature, C is the bias parameter, SABB is the output result of the self-attention mechanism, Softmax is a mathematical function, and tr is the transposition symbol.

5. The video shadow detection method according to claim 1, characterized in that: The constructing of the initial diffusion model, inputting the spliced ​​features into the initial diffusion model, and obtaining the trained diffusion model, comprises: Inputting the spliced ​​features into the initial diffusion model for forward diffusion and reverse diffusion; The diffused concatenated features are decoded using a decoder to obtain the trained diffusion model.

6. The video shadow detection method according to claim 5, characterized in that: The step of inputting the spliced ​​features into the initial diffusion model for forward diffusion and reverse diffusion includes: The diffusion model implements the forward diffusion process of the spliced ​​features through the following formula: Where t is the time step, y 0 is the observed value of the initial state, y t is the observation value at time step t, ∈ is Gaussian noise, is the noise control function; The reverse diffusion process of the spliced ​​features is achieved by the following formula: p θ (and 0:t |g,x)=p(y t )∏p θ (and t-1 |and t ,g,x) Among them, y 0:t is all the observations from step 0 to step t, x is the preset conditional variable, g is the concatenated feature, θ is the parameterized network, and y t-1 is the observed value at time step t-1, and p() is the probability density function.

7. The video shadow detection method according to claim 1, characterized in that: The loss function of the diffusion model is: Among them, L bce represents the ambiguous cross entropy function, is the pseudo label mask, y is the real label, To predict the mask, is the boundary mask, and b is the true value of the boundary mask.

8. A video shadow detection device, characterized in that: The device comprises: A data acquisition module, used to acquire a video sequence to be detected; The shadow detection module is used to extract the deep learning features of the video sequence to be detected and input them into the trained diffusion model to obtain the video shadow detection result of the video sequence to be detected; wherein the training process of the diffusion model includes: Obtain a video sequence training set; Performing temporal aggregation on the video sequences in the video sequence training set to obtain aggregated features; Extracting a corresponding shadow region prior based on the aggregated features; Based on the shadow area prior, boundary features are obtained through a self-attention mechanism; Extracting the deep learning features and the shadow region prior guided features using a guided encoder; Splicing the boundary feature and the guide feature to obtain a spliced ​​feature; An initial diffusion model is constructed, and the concatenated features are input into the initial diffusion model to obtain the trained diffusion model.

9. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the video shadow detection method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the video shadow detection method according to any one of claims 1 to 7.

Images