Fan blade segmentation method and device based on video clip feature matching

Abstract

The present invention proposes a method and device for fan blade segmentation based on video clip feature matching. The method constructs a fan blade video target segmentation network; wherein the fan blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder; collects a historical data set of fan blade videos and trains the video target segmentation network; inputs the real-time fan blade video into the trained video target segmentation network, and the output result is the segmentation result of the fan blade region in the fan blade video. The present invention integrates an innovative dynamic memory mechanism to store the historical information of the video clip, enables the segmentation network to effectively process the video clip through feature matching, and generates segment-level prediction results through multi-frame input and multi-frame output, effectively improving the speed and accuracy of fan blade segmentation.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method, device, equipment, and storage medium for wind turbine blade segmentation based on video clip feature matching. Background Art

[0002] In order to help drones realize automatic detection of defects in wind turbine blades, drones need to identify the area of ​​the wind turbine in videos and images and obtain the accurate boundary of the wind turbine. At present, scientists use video target segmentation technology to extract wind turbine blade targets in videos to obtain the accurate boundary of the wind turbine. The purpose of video target segmentation is to predict foreground targets in videos. Broadly speaking, video target segmentation can be divided into two categories: unsupervised video target segmentation and semi-supervised video target segmentation. Unsupervised video target segmentation does not require manual labeling, and the algorithm automatically extracts foreground targets. Early methods explored two-stream architectures or variants of recurrent neural networks to capture more temporal information. Recent studies have solved comprehensive foreground reasoning from a global perspective through joint attention, anchor diffusion, and graph neural networks.

[0003] Semi-supervised video target segmentation uses limited manual annotations (usually the first frame) to obtain foreground targets in the video. In this case, semi-supervised video target segmentation is also called pixel-by-pixel tracking or mask propagation. A common approach is to learn a video-specific representation of the annotated object in the first frame, and then perform pixel-by-pixel detection in the remaining frames. Early semi-supervised video target segmentation methods were based on optical flow or recurrent connections. With the development of deep learning, more advanced methods are based on matching strategies, which usually train a prototype twin matching network to find the most matching pixel (or embedding in feature space) frame between the first frame (or segmented frame) and the query, and then implement label assignment accordingly. However, there is often a problem of low recognition accuracy. Summary of the Invention

[0004] The present invention provides a fan blade segmentation method, device, equipment and storage medium based on video clip feature matching, aiming to improve the accuracy of fan blade recognition in videos by utilizing time sequence information.

[0005] To this end, the first object of the present invention is to propose a wind turbine blade segmentation method based on video segment feature matching, comprising:

[0006] Construct a wind blade video target segmentation network; the wind blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder;

[0007] Collect historical video datasets of wind turbine blades and train the video object segmentation network;

[0008] The real-time video of the wind blades is input into the trained video target segmentation network, and the output result is the segmentation result of the wind blade area in the wind blade video.

[0009] The steps of collecting a historical video dataset of wind turbine blades and training a video object segmentation network include:

[0010] Obtain a wind blade video dataset, perform pixel-level annotation on the wind blade region in the wind blade video to generate corresponding annotation data, and divide the wind blade video historical dataset that generates the corresponding annotation data into a training set and a test set;

[0011] Input the training set into the video object segmentation model for training, determine the hyperparameters and loss function of the model training, and optimize the parameters of the video object segmentation network until the prediction accuracy of the video object segmentation network reaches a preset accuracy threshold;

[0012] The test set is input into the trained video object segmentation network to verify the accuracy of the video object segmentation network.

[0013] Before the step of training the video target segmentation network, the method further includes the step of preprocessing the blade video data in the training set; including:

[0014] The video data of any wind turbine blade in the training set is divided into several video segments according to the preset interval. The previous video segment and the corresponding annotation data are used as memory frames to predict the current video segment, and the current video segment to be predicted is used as the query frame.

[0015] Perform sample enhancement on each frame of the image, including random brightness, contrast, and saturation transformation.

[0016] Among them, the segmentation steps of the video object segmentation network include:

[0017] Both the memory frame and the query frame are input into the key encoder, which extracts the feature map corresponding to the spatiotemporal relationship;

[0018] Input the annotated data of the memory frame into the value encoder, and the value encoder extracts the annotated mask features of the memory frame;

[0019] The feature map corresponding to the spatiotemporal relationship extracted by the key encoder and the annotated mask features of the memory frame extracted by the value encoder are input into the feature matching module to mine the correlation information between the query frame feature map and the memory frame and the annotated mask feature map and perform matching;

[0020] The output of the feature matching module and the memory frame are input to the sequence refinement module, and the transformer structure is used to obtain the correlation within the video clip;

[0021] The decoder decodes the output of the sequence refinement module to obtain the prediction result of the query frame.

[0022] Among them, the key encoder uses the ResNet50 backbone network for feature extraction, and the query frame and the memory frame are input into the key encoder to obtain the feature value k Q 、k Q intra 、k M ;

[0023] The value encoder uses ResNet18 to extract the backbone network feature value, and the labeled data of the memory frame is input into the value encoder to obtain the feature value v m .

[0024] Among them, the feature matching module calculates the global similarity of the feature values ​​of the query frame and the memory frame. The similarity matrix A is calculated as follows:

[0025]

[0026] Among them, A i，j Indicates the similarity between the pixels in row i and column j;

[0027] After obtaining the global similarity matrix A between the query frame and the memory frame, the feature embedding v of the query frame retrieved from the memory frame and the annotated data is Q The calculation is as follows:

[0028] v Q =A(k Q , k M )v M

[0029] The sequence refinement module uses the multi-head attention mechanism of the Transformer network to refine the feature embedding of the query frame based on the spatiotemporal correlation between features in the video clip to enhance the ability to handle new targets, occlusions, or object deformations in the query frame.

[0030] The sequence refinement module includes: layer normalization, multi-head attention module, layer normalization and feedforward layer; to ensure the stability of training, v Q With k Q intra Normalization, after normalization, calculate v through the multi-head attention mechanism Q With k Q intra The similarity of ,feature extraction is completed by the feed-forward layer;

[0031] The feedforward layers are fully connected layer, GELU activation function, dropout, fully connected layer and dropout;

[0032] The entire calculation process is expressed by the formula:

[0033] v atten =A(f k (LN(k Q intra )), f k (LN(k Q intra )))f v (LN(v Q ))+LN(v Q )

[0034]

[0035] Where LN represents normalization, f k With f v Represents v Q With k Q intra Perform linear mapping, FF represents the feedforward layer; the output of the sequence refinement module is the feature embedding of the refined query frame

[0036] After inputting into the encoder, the final segmentation result is obtained;

[0037] The decoder upsamples the encoded features through bilinear interpolation and fuses the features with the encoder through skip-connection.

[0038] A second object of the present invention is to provide a wind turbine blade segmentation device based on video segment feature matching, comprising:

[0039] A model building module is used to build a wind blade video target segmentation network; wherein the wind blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder;

[0040] Model training module, used to collect historical video datasets of wind turbine blades and train the video object segmentation network;

[0041] The blade segmentation module is used to input the real-time video of the wind turbine blades into the trained video target segmentation network, and the output result is the segmentation result of the wind turbine blade area in the wind turbine blade video.

[0042] The third object of the present invention is to provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute each step in the method of the aforementioned technical solution.

[0043] A fourth object of the present invention is to provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute each step in the method according to the aforementioned technical solution.

[0044] Differentiating from existing technologies, the present invention provides a method for wind blade segmentation based on video clip feature matching. This method constructs a wind blade video target segmentation network, comprising a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder. A historical dataset of wind blade videos is collected to train the video target segmentation network. Real-time wind blade videos are fed into the trained video target segmentation network, and the output is the segmentation result of the wind blade region in the wind blade video. The present invention integrates an innovative dynamic memory mechanism to store historical video clip information. Feature matching enables the segmentation network to effectively process video clips. Through multi-frame input and output, segment-level prediction results are generated, effectively improving the speed and accuracy of wind blade segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention and / or additional aspects and advantages will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 It is a flow chart of a fan blade segmentation method based on video clip feature matching provided by the present invention.

[0047] Figure 2 It is a structural schematic diagram of a fan blade segmentation network in a fan blade segmentation method based on video clip feature matching provided by the present invention.

[0048] Figure 3 It is a logical schematic diagram of a sequence refinement module of a fan blade segmentation network in a fan blade segmentation method based on video clip feature matching provided by the present invention.

[0049] Figure 4 It is a structural schematic diagram of a fan blade segmentation device based on video clip feature matching provided by the present invention.

[0050] Figure 5 It is a structural schematic diagram of a non-transitory computer-readable storage medium storing computer instructions provided by the present invention. DETAILED DESCRIPTION

[0051] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present invention, but are not to be construed as limiting the present invention.

[0052] like Figure 1 As shown in FIG, a fan blade segmentation method based on video clip feature matching provided by an embodiment of the present invention specifically includes:

[0053] S110: Construct a wind turbine blade video target segmentation network; wherein the wind turbine blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder.

[0054] The constructed video target segmentation network structure is as follows Figure 2 As shown, it includes a key encoder, a value encoder, a feature matching module, a sequence refinement module, and a decoder. Both the memory frame and the query frame are input into the key encoder, which extracts the feature map corresponding to the spatiotemporal relationship. The annotation data of the memory frame is input into the value encoder, which extracts the annotation mask features of the memory frame. The feature maps extracted by the key encoder and the value encoder are input into the feature matching module, which is used to mine the correlation information between the query frame feature map and the memory frame and the annotation mask feature map and match them. The output of the feature matching module and the memory frame are input into the sequence refinement module, which uses the transformer structure to obtain the correlation within the video clip. Finally, the decoder decodes the output of the sequence refinement module to obtain the prediction result of the query frame.

[0055] The key encoder uses the ResNet50 backbone network to extract features, and the query frame and memory frame are input into the key encoder to obtain the feature value k Q 、k Q intra 、k M .

[0056] The value encoder uses ResNet18 to extract the backbone network feature value. The labeled data of the memory frame is input into the value encoder to obtain the feature value v m .

[0057] The feature matching module first calculates the global similarity of the feature values ​​of the query frame and the memory frame. The similarity matrix A is calculated as follows:

[0058]

[0059] Among them, A i，j Indicates the similarity of the pixels in the i-th row and j-th column.

[0060] After obtaining the global similarity matrix A between the query frame and the memory frame, the feature embedding v of the query frame retrieved from the memory frame and the annotated data is Q The calculation is as follows:

[0061] v Q =A(k Q , kM )v M

[0062] In order to enhance the model's ability to handle new targets, occlusions, or object deformations in the query frame, the sequence refinement module uses the multi-head attention mechanism of the Transformer network to refine the feature embedding of the query frame based on the spatiotemporal correlation between features in the video clip.

[0063] like Figure 3 As shown in Figure 2, the main structure of the sequence refinement module is: layer normalization, multi-head attention module, layer normalization, and feedforward layer. To ensure the stability of training, we first Q With k Q intra Normalization, after normalization, calculate v through the multi-head attention module Q With k Q intra The similarity of is calculated, and finally the feature extraction is completed by the feed-forward layer. The feed-forward layer is a fully connected layer, GELU activation function, dropout, fully connected layer, and dropout.

[0064] The entire calculation process is expressed by the formula:

[0065] v atten =A(f k (LN(k Q intra )), f k (LN(k Q intra )))f v (LN(v Q ))+LN(v Q )

[0066]

[0067] Where LN represents normalization, f k With f v Represents v Q With k Q intra Perform linear mapping, FF represents the feedforward layer. The output of the sequence refinement module is the feature embedding of the refined query frame

[0068] After inputting into the encoder, the final segmentation result is obtained.

[0069] The decoder upsamples the encoded features through bilinear interpolation and fuses the features with the encoder through skip-connection.

[0070] Before the step of training the video target segmentation network, the method further includes the step of preprocessing the blade video data in the training set; including:

[0071] The video data of any wind turbine blade in the training set is divided into several video segments according to the preset interval. The previous video segment and the corresponding annotation data are used as memory frames to predict the current video segment, and the current video segment to be predicted is used as the query frame.

[0072] Perform sample enhancement on each frame of the image, including random brightness, contrast, and saturation transformation.

[0073] S120: Collect a historical video dataset of wind turbine blades and train a video object segmentation network.

[0074] The training steps include:

[0075] Obtain a wind blade video dataset, perform pixel-level annotation on the wind blade region in the wind blade video to generate corresponding annotation data, and divide the wind blade video historical dataset that generates the corresponding annotation data into a training set and a test set;

[0076] Input the training set into the video object segmentation model for training, determine the hyperparameters and loss function of the model training, and optimize the parameters of the video object segmentation network until the prediction accuracy of the video object segmentation network reaches a preset accuracy threshold;

[0077] The test set is input into the trained video object segmentation network to verify the accuracy of the video object segmentation network.

[0078] The data is processed by the feature matching module and the sequence refinement module and input into the decoder. After decoding, the extraction results of the wind blades in the wind blade video that generates the labeled data are obtained. The network parameters are updated by the loss of the labeled data and the network output.

[0079] S130: Inputting the real-time video of the wind blade into the trained video object segmentation network, and outputting a segmentation result of the wind blade region in the wind blade video.

[0080] like Figure 4 As shown, the present invention provides a wind turbine blade segmentation device 300 based on video segment feature matching, comprising:

[0081] The model construction module 310 is used to construct a wind turbine blade video target segmentation network; wherein the wind turbine blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder;

[0082] Model training module 320, for collecting historical video datasets of wind turbine blades and training the video object segmentation network;

[0083] The blade segmentation module 330 is used to input the real-time video of the wind turbine blades into the trained video target segmentation network, and the output result is the segmentation result of the wind turbine blade area in the wind turbine blade video.

[0084] In order to implement the embodiment, the present invention also proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute each step of the wind blade segmentation method based on video clip feature matching in the aforementioned technical solution.

[0085] like Figure 5 As shown, a non-transitory computer-readable storage medium 800 includes an instruction memory 810 and an interface 830. The instructions can be executed by a wind turbine blade segmentation processor 820 based on video segment feature matching to complete the method. Alternatively, the storage medium can be a non-transitory computer-readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0086] To implement the embodiment, the present invention further proposes a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the wind turbine blade segmentation based on video clip feature matching as in the embodiment of the present invention is implemented.

[0087] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0088] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0089] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0090] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0091] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the embodiments described, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0092] Those skilled in the art will understand that all or part of the steps of the method for implementing the embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0093] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0094] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it is understood that the embodiments are exemplary and are not to be construed as limiting the present invention. Those skilled in the art may make changes, modifications, substitutions, and variations to the embodiments within the scope of the present invention.

Claims

1. A fan blade segmentation method based on video clip feature matching, characterized in that: include: Constructing a wind turbine blade video target segmentation network; wherein the wind turbine blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder; Collecting a historical dataset of wind turbine blade videos and training the video object segmentation network; The real-time video of the wind turbine blades is input into the trained video object segmentation network, and the output result is the segmentation result of the wind turbine blade area in the wind turbine blade video; The segmentation step of the video object segmentation network includes: Both the memory frame and the query frame are input into the key encoder, which extracts the feature map corresponding to the spatiotemporal relationship; Inputting the labeled data of the memory frame into a value encoder, and extracting the labeled mask features of the memory frame by the value encoder; The feature map corresponding to the spatiotemporal relationship extracted by the key encoder and the annotated mask features of the memory frame extracted by the value encoder are input into the feature matching module to mine the correlation information between the query frame feature map and the memory frame and the annotated mask feature map and perform matching; The output of the feature matching module and the memory frame are input to the sequence refinement module, and the transformer structure is used to obtain the correlation within the video segment; The decoder decodes the output of the sequence refinement module to obtain the prediction result of the query frame; The key encoder uses the ResNet50 backbone network to extract features, and the query frame and the memory frame are input into the key encoder to obtain feature values ​​respectively. 、 、 ; The value encoder uses ResNet18 to extract the backbone network feature value, and the labeled data of the memory frame is input into the value encoder to obtain the feature value ; The feature matching module calculates the global similarity of the feature values ​​of the query frame and the memory frame. The similarity matrix A is calculated as follows: in, Indicates the similarity between the pixels in row i and column j; After obtaining the global similarity matrix A between the query frame and the memory frame, the feature embedding of the query frame retrieved from the memory frame and the annotated data is The calculation is as follows: 。 2. The fan blade segmentation method based on video clip feature matching according to claim 1 is characterized in that: The steps of collecting a historical video dataset of wind turbine blades and training the video object segmentation network include: Obtain a wind blade video dataset, perform pixel-level annotation on the wind blade region in the wind blade video to generate corresponding annotation data, and divide the wind blade video historical dataset that generates the corresponding annotation data into a training set and a test set; Inputting the training set into the video object segmentation model for training, determining hyperparameters and loss functions for model training, and optimizing parameters of the video object segmentation network until the prediction result accuracy of the video object segmentation network reaches a preset accuracy threshold; The test set is input into the trained video object segmentation network to verify the accuracy of the video object segmentation network.

3. The fan blade segmentation method based on video segment feature matching according to claim 2, characterized in that: Before the step of training the video target segmentation network, the method further includes the step of preprocessing the blade video data in the training set; including: The video data of any wind turbine blade in the training set is divided into several video segments according to the preset interval. The previous video segment and the corresponding annotation data are used as memory frames to predict the current video segment, and the current video segment to be predicted is used as the query frame. Perform sample enhancement on each frame of the image, including random brightness, contrast, and saturation transformation.

4. The fan blade segmentation method based on video segment feature matching according to claim 1, characterized in that: The sequence refinement module uses the multi-head attention mechanism of the Transformer network to refine the feature embedding of the query frame based on the spatiotemporal correlation between features in the video clip to enhance the processing ability when new targets, occlusions or object deformation appear in the query frame. The sequence refinement module includes: layer normalization, multi-head attention module, layer normalization and feedforward layer; to ensure the stability of training, and Normalization, after normalization, calculation through the multi-head attention mechanism and The similarity of ,feature extraction is completed by the feed-forward layer; The feedforward layer is a fully connected layer, a GELU activation function, a dropout layer, a fully connected layer, and a dropout layer in sequence; The entire calculation process is expressed by the formula: Where LN represents normalization, and Respectively represent and Perform linear mapping, FF represents the feedforward layer; the output of the sequence refinement module is the feature embedding of the refined query frame ; After inputting into the encoder, the final segmentation result is obtained; The decoder upsamples the encoded features through bilinear interpolation and fuses the features with the encoder through skip-connection.

5. A fan blade segmentation device based on video segment feature matching, characterized in that: include: A model building module is used to build a wind turbine blade video target segmentation network; wherein the wind turbine blade target segmentation network includes a key encoder, a value encoder, a feature matching module, and a sequence refinement module decoder; A model training module is used to collect a historical data set of wind turbine blade videos and train the video object segmentation network; The blade segmentation module is used to input the real-time video of the wind turbine blades into the trained video target segmentation network, and the output result is the segmentation result of the wind turbine blade area in the wind turbine blade video; The segmentation step of the video object segmentation network includes: Both the memory frame and the query frame are input into the key encoder, which extracts the feature map corresponding to the spatiotemporal relationship; Inputting the labeled data of the memory frame into a value encoder, and extracting the labeled mask features of the memory frame by the value encoder; The feature map corresponding to the spatiotemporal relationship extracted by the key encoder and the annotated mask features of the memory frame extracted by the value encoder are input into the feature matching module to mine the correlation information between the query frame feature map and the memory frame and the annotated mask feature map and perform matching; The output of the feature matching module and the memory frame are input to the sequence refinement module, and the transformer structure is used to obtain the correlation within the video segment; The decoder decodes the output of the sequence refinement module to obtain the prediction result of the query frame; The key encoder uses the ResNet50 backbone network to extract features, and the query frame and the memory frame are input into the key encoder to obtain feature values ​​respectively. 、 、 ; The value encoder uses ResNet18 to extract the backbone network feature value, and the labeled data of the memory frame is input into the value encoder to obtain the feature value ; The feature matching module calculates the global similarity of the feature values ​​of the query frame and the memory frame. The similarity matrix A is calculated as follows: in, Indicates the similarity between the pixels in row i and column j; After obtaining the global similarity matrix A between the query frame and the memory frame, the feature embedding of the query frame retrieved from the memory frame and the annotated data is The calculation is as follows: 。 6. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform each step in the method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute each step of the method according to any one of claims 1 to 4.

Images