Sea surface key area extraction method, device and product based on visual computing
The deep convolution network and cross-correlation attention module process sea surface situation images and combine them with optical flow maps to perform significance detection, which solves the problem of low extraction accuracy of key sea surface areas in the existing technology, and achieves more efficient offshore target detection.
Patent Information
- Application Number
- CN202411329714.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-24
AI Technical Summary
The accuracy of extraction of key sea surface areas in the prior art is not high, mainly because the significance detection method does not consider the sea surface situation characteristics incompletely.
The sea surface situation image is processed through a deep convolutional network and a cross-correlation attention module, and the sea surface significance target and motion target are extracted, and the significance detection is performed in combination with the optical flow map to determine the key areas of the sea surface.
It improves the accuracy of extraction of key sea surface areas, can effectively locate dynamic targets, enhance the extraction ability of dynamic targets, comprehensively consider the significance of sea surface and moving targets, and improves the efficiency of maritime target detection.
Smart Images

Figure CN118840545B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a method, device and product for extracting key areas of a sea surface based on visual computing. Background Art
[0002] In the related art, the extraction of key areas of the sea surface mainly utilizes the saliency detection method. However, the saliency detection method does not take into account the characteristics of the sea surface situation comprehensively, which leads to low accuracy in extracting key areas of the sea surface using this method. Summary of the invention
[0003] Based on the above technical problems, the embodiments of the present invention provide a method, device and product for extracting key areas of the sea surface based on visual computing, which aims to extract key areas of the sea surface based on two aspects: significant targets and moving targets on the sea surface, so as to improve the accuracy of extracting key areas of the sea surface.
[0004] A first aspect of an embodiment of the present invention provides a method for extracting key areas of a sea surface based on visual computing, the method comprising:
[0005] Inputting a plurality of frames of sea surface situation images into an encoder, and obtaining a plurality of frame feature maps that aggregate global context information of the sea surface situation images through processing by a deep convolutional network in the encoder and a cross-correlation attention module connected in series with the deep convolutional network;
[0006] Inputting the multi-frame feature map into a decoder for decoding, and obtaining a multi-frame optical flow map output by the decoder;
[0007] The key sea surface areas in the multiple frames of sea surface situation images are determined according to the multiple frames of optical flow images and the detection results obtained by performing saliency detection on the multiple frames of sea surface situation images.
[0008] Optionally, the processing of the deep convolutional network in the encoder and the cross-correlation attention module connected in series with the deep convolutional network to obtain a multi-frame feature map that aggregates global context information of the sea surface situation image includes:
[0009] By using two convolution layers with 3×3 convolution kernels in the deep convolution network, image features of the sea surface situation image at 1 / 2 resolution and 1 / 4 resolution are extracted in sequence;
[0010] The image features at the 1 / 4 resolution are input into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module, and the feature map includes at least: local features of large-displacement motion targets and global features of small-displacement motion targets, the large-displacement motion targets are targets with motion displacement greater than a first threshold, and the small-displacement motion targets are targets with motion displacement less than a second threshold.
[0011] Optionally, the cross-correlation attention module includes a plurality of attention mechanism modules connected in series, and a convolution layer with a convolution kernel of 3×3 is provided between each attention mechanism module;
[0012] The step of inputting the image features at the 1 / 4 resolution into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module includes:
[0013] Input the image features at the 1 / 4 resolution into the first attention mechanism module until a multi-frame feature map with a resolution of 1 / 64 is output by the last attention mechanism module;
[0014] Except for the first attention mechanism module, the input of each attention mechanism module is the output of the previous attention mechanism module; the processing of each attention mechanism module is expressed by the following formula: ;
[0015] in, Represents the output of the previous attention mechanism module, The representative layer is normalized. represents the processing using a multilayer perceptron, CSwin_Attention represents the inverse operation of the decoding operation performed by the decoder, It is The intermediate output of the attention mechanism module, It is The output of an attention mechanism module.
[0016] Optionally, the multiple frames of sea surface situation images include at least three consecutive frames of sea surface situation images, and the multiple frames of feature graphs include at least a first frame feature graph, a second frame feature graph, and a third frame feature graph respectively corresponding to the three consecutive frames of sea surface situation images;
[0017] The step of inputting the multi-frame feature map into a decoder for decoding to obtain a multi-frame optical flow map output by the decoder includes:
[0018] The following decoding operations are performed by the decoder:
[0019] Calculate the correlation between the first frame feature map and the second frame feature map, and between the second frame feature map and the third frame feature map, respectively, to obtain a correlation result;
[0020] Performing occlusion decoding and optical flow decoding on the correlation result respectively to obtain an occlusion decoding result and an optical flow decoding result;
[0021] Based on the occlusion decoding result and the optical flow decoding result, the multi-frame optical flow map is obtained, and the multi-frame optical flow map at least includes: a first optical flow map corresponding to the first frame feature map and the second frame feature map, and a second optical flow map corresponding to the second frame feature map and the third frame feature map.
[0022] Optionally, the method further comprises:
[0023] Upsampling the second optical flow map obtained by the previous round of decoding operation to obtain a second sampled optical flow map;
[0024] Adding and fusing the second sampling optical flow map and the second frame feature map to obtain a second frame sampling feature map;
[0025] Adding and fusing the second sampling optical flow map and the third frame feature map to obtain a third frame sampling feature map;
[0026] Performing deformation processing on the second frame sampling feature map and the third frame sampling feature map respectively to obtain a second frame deformation feature map and a third frame deformation feature map;
[0027] The first frame feature map and the second frame feature map are respectively correlated with each other, and the second frame feature map and the third frame feature map are respectively correlated with each other to obtain a correlation result, including:
[0028] The first frame feature map and the second frame deformation feature map are correlated, and the second frame feature map and the third frame deformation feature map are correlated to obtain the correlation result.
[0029] Optionally, obtaining the multi-frame optical flow map based on the occlusion decoding result and the optical flow decoding result includes:
[0030] Processing the occlusion decoding result through an occlusion context network to obtain an optimized occlusion decoding graph;
[0031] Processing the optical flow decoding result through an optical flow context network to obtain an optimized optical flow decoding graph;
[0032] The optimized occlusion decoding map and the optimized optical flow decoding map are input into a bilateral refinement network to obtain the multi-frame optical flow map output by the bilateral refinement network.
[0033] Optionally, the method further comprises:
[0034] Upsampling the first optical flow map and the second optical flow map obtained by the previous round of decoding operation respectively to obtain a first sampled optical flow map and a second sampled optical flow map;
[0035] Adding and fusing the first sampled optical flow map and the correlation result to obtain a first sampled correlation result, and adding and fusing the second sampled optical flow map and the correlation result to obtain a second sampled correlation result;
[0036] The step of performing occlusion decoding and optical flow decoding on the correlation result to obtain an occlusion decoding result and an optical flow decoding result comprises:
[0037] Performing occlusion decoding on the first sampling correlation result to obtain a first sampling occlusion decoding result, and performing optical flow decoding on the second sampling correlation result to obtain a second sampling optical flow decoding result;
[0038] The occlusion decoding result is processed by the occlusion context network to obtain an optimized occlusion decoding graph, including:
[0039] Adding and fusing the first sampled optical flow map and the first sampled occlusion decoding result to obtain a fused sampled occlusion decoding result;
[0040] Processing the fused sampling occlusion decoding result through the occlusion context network to obtain an optimized fused sampling occlusion decoding graph;
[0041] The optical flow decoding result is processed by the optical flow context network to obtain an optimized optical flow decoding graph, including:
[0042] Adding and fusing the second sampled optical flow map and the second sampled optical flow decoding result to obtain a fused sampled optical flow decoding result;
[0043] The fused sampled optical flow decoding result is processed by the optical flow context network to obtain an optimized fused sampled optical flow decoding graph;
[0044] The step of inputting the optimized occlusion decoding graph and the optimized optical flow decoding graph into a bilateral refinement network to obtain the multi-frame optical flow graph output by the bilateral refinement network comprises:
[0045] The optimized fused sampled occlusion decoding map and the optimized fused sampled optical flow decoding map are input into the bilateral refinement network to obtain a multi-frame optimized optical flow map.
[0046] A second aspect of an embodiment of the present invention provides a device for extracting key sea surface areas based on visual computing, the device comprising:
[0047] An encoding module, used for inputting a plurality of frames of sea surface situation images into an encoder, and obtaining a plurality of frame feature maps that aggregate the global context information of the sea surface situation images through processing by a deep convolutional network in the encoder and a cross-correlation attention module connected in series with the deep convolutional network;
[0048] A decoding module, used for inputting the multi-frame feature map into a decoder for decoding, and obtaining a multi-frame optical flow map output by the decoder;
[0049] The key area generation module is used to determine the sea surface key area in the multi-frame sea surface situation image according to the multi-frame optical flow map and the detection results obtained by performing saliency detection on the multi-frame sea surface situation image.
[0050] Optionally, the encoding module includes:
[0051] A deep convolution module, used to extract image features of the sea surface situation image at 1 / 2 resolution and 1 / 4 resolution in sequence through two convolution layers with 3×3 convolution kernels in the deep convolution network;
[0052] A cross-correlation module is used to input the image features at the 1 / 4 resolution into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module, and the feature map at least includes: local features of large-displacement motion targets and global features of small-displacement motion targets, the large-displacement motion targets are targets with motion displacement greater than a first threshold, and the small-displacement motion targets are targets with motion displacement less than a second threshold.
[0053] Optionally, the cross-correlation attention module includes a plurality of attention mechanism modules connected in series, and a convolution layer with a convolution kernel of 3×3 is provided between each attention mechanism module;
[0054] The cross-correlation module comprises:
[0055] An attention module is used to input the image features at the 1 / 4 resolution into the first attention mechanism module until a multi-frame feature map with a resolution of 1 / 64 is output by the last attention mechanism module;
[0056] Except for the first attention mechanism module, the input of each attention mechanism module is the output of the previous attention mechanism module; the processing of each attention mechanism module is expressed by the following formula: ;
[0057] in, Represents the output of the previous attention mechanism module, The representative layer is normalized. represents the processing using a multilayer perceptron, CSwin_Attention represents the inverse operation of the decoding operation performed by the decoder, It is The intermediate output of the attention mechanism module, It is The output of an attention mechanism module.
[0058] Optionally, the multiple frames of sea surface situation images include at least three consecutive frames of sea surface situation images, and the multiple frames of feature graphs include at least a first frame feature graph, a second frame feature graph, and a third frame feature graph respectively corresponding to the three consecutive frames of sea surface situation images;
[0059] The decoding module is used to perform the following decoding operations through the decoder, and the decoding module includes:
[0060] A correlation module, used to calculate the correlation between the first frame feature map and the second frame feature map, and between the second frame feature map and the third frame feature map, to obtain a correlation result;
[0061] A first processing module, configured to perform occlusion decoding and optical flow decoding on the correlation result to obtain an occlusion decoding result and an optical flow decoding result;
[0062] The second processing module is used to obtain the multi-frame optical flow map based on the occlusion decoding result and the optical flow decoding result, and the multi-frame optical flow map at least includes: a first optical flow map corresponding to the first frame feature map and the second frame feature map, and a second optical flow map corresponding to the second frame feature map and the third frame feature map.
[0063] Optionally, the device further comprises:
[0064] A first sampling module, used for up-sampling the second optical flow map obtained by the previous round of decoding operation to obtain a second sampled optical flow map;
[0065] A first fusion module, used for adding and fusing the second sampled optical flow map and the second frame feature map to obtain a second frame sampled feature map;
[0066] A second fusion module is used to add and fuse the second sampled optical flow map and the third frame feature map to obtain a third frame sampled feature map;
[0067] A deformation module, used to perform deformation processing on the second frame sampling feature map and the third frame sampling feature map respectively to obtain a second frame deformation feature map and a third frame deformation feature map;
[0068] The correlation module comprises:
[0069] The third processing module is used to perform correlation calculation on the first frame feature map and the second frame deformation feature map, and perform correlation calculation on the second frame feature map and the third frame deformation feature map to obtain the correlation result.
[0070] Optionally, the second processing module includes:
[0071] An occlusion processing module, used to process the occlusion decoding result through an occlusion context network to obtain an optimized occlusion decoding graph;
[0072] An optical flow processing module, used for processing the optical flow decoding result through an optical flow context network to obtain an optimized optical flow decoding graph;
[0073] The optimization processing module is used to input the optimized occlusion decoding map and the optimized optical flow decoding map into a bilateral refinement network to obtain the multi-frame optical flow map output by the bilateral refinement network.
[0074] Optionally, the device further comprises:
[0075] A second sampling module is used to upsample the first optical flow map and the second optical flow map obtained by the previous round of decoding operation to obtain a first sampled optical flow map and a second sampled optical flow map;
[0076] A third fusion module is used to add and fuse the first sampled optical flow map and the correlation result to obtain a first sampled correlation result, and add and fuse the second sampled optical flow map and the correlation result to obtain a second sampled correlation result;
[0077] The first processing module comprises:
[0078] A first processing submodule, configured to perform occlusion decoding on the first sampling correlation result to obtain a first sampling occlusion decoding result, and perform optical flow decoding on the second sampling correlation result to obtain a second sampling optical flow decoding result;
[0079] The occlusion processing module comprises:
[0080] a fourth fusion module, configured to add and fuse the first sampled optical flow map and the first sampled occlusion decoding result to obtain a fused sampled occlusion decoding result;
[0081] An occlusion processing submodule, used to process the fused sampling occlusion decoding result through the occlusion context network to obtain an optimized fused sampling occlusion decoding graph;
[0082] The optical flow processing module comprises:
[0083] a fifth fusion module, configured to add and fuse the second sampled optical flow map and the second sampled optical flow decoding result to obtain a fused sampled optical flow decoding result;
[0084] An optical flow processing submodule, used for processing the fused sampled optical flow decoding result through the optical flow context network to obtain an optimized fused sampled optical flow decoding graph;
[0085] The optimization processing module comprises:
[0086] The optimization processing submodule is used to input the optimized fused sampling occlusion decoding map and the optimized fused sampling optical flow decoding map into the bilateral refinement network to obtain a multi-frame optimized optical flow map.
[0087] A third aspect of an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for extracting key sea surface areas based on visual computing as described in the first aspect of an embodiment of the present invention is implemented.
[0088] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for extracting key sea surface areas based on visual computing according to the first aspect of an embodiment of the present invention is implemented.
[0089] According to the method for extracting key sea surface areas based on visual computing of the embodiment of the present invention, first, a plurality of frames of sea surface situation images are input into an encoder, and a multi-frame feature map that aggregates the global context information of the sea surface situation images is obtained through processing by a deep convolutional network and a cross-correlation attention module in the encoder; then, the multi-frame feature map is input into a decoder for decoding to obtain a multi-frame optical flow map; finally, based on the multi-frame optical flow map and the detection results obtained by performing saliency detection on the multi-frame sea surface situation images, the key sea surface areas in the multi-frame sea surface situation images are extracted.
[0090] In this implementation, we no longer only consider the apparently significant feature of the sea surface saliency, but also consider the moving targets on the sea surface, and divide the extraction of key areas of the sea surface into two parts: extraction of significant targets on the sea surface and extraction of moving targets; for the extraction of moving targets, this embodiment designs an encoder through a cross-correlation attention module and a deep convolutional network to encode a multi-frame feature map that aggregates the global context information of the sea surface situation image, thereby decoding to obtain a multi-frame optical flow map to achieve effective positioning of dynamic targets. In this way, this embodiment determines the key areas of the sea surface based on the multi-frame optical flow map that represents the moving target and the detection results that represent the significant target, which can effectively improve the accuracy of the extraction of key areas of the sea surface. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0092] Figure 1 It is a schematic diagram of the relationship between situation awareness and attention mechanism in related technologies;
[0093] Figure 2 is a flow chart of a method for extracting key sea surface areas based on visual computing according to an embodiment of the present invention;
[0094] Figure 3 is a schematic diagram of extracting key sea surface areas based on saliency and dynamic targets according to an embodiment of the present invention;
[0095] Figure 4 It is a structural block diagram of a device for extracting key sea surface areas based on visual computing provided by an embodiment of the present invention;
[0096] Figure 5 It is a schematic diagram of an electronic device shown in an embodiment of the present invention. DETAILED DESCRIPTION
[0097] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0098] Attention is an important mechanism for people to observe and understand the world, and it is the ability to focus on key points. The study of attention mechanism originated from experimental psychology in the 19th century. Cognitive psychology and neurophysiology developed in the mid-20th century further promoted the study of attention mechanism. However, for a long time, attention has only been regarded as a physiological and psychological function of an individual in the process of perceiving and understanding external stimuli, with a focus on senses such as vision and hearing. Furthermore, the situational awareness theory was proposed to further expand the attention mechanism, from the individual's inner mental activities to the perception of the surrounding dynamic environment and the behavior and intentions of others, and integrated into the closed loop of cognition, decision-making and action. The relationship between attention mechanism and situational awareness is as follows Figure 1 As shown, Figure 1 This is a schematic diagram of the relationship between situation awareness and attention mechanism in related technologies. Figure 1In the process of situational awareness, attention is reflected in the awareness, interpretation, understanding, prediction, decision-making and action in working memory, while the relevant schemas and scripts in long-term memory can guide the interpretation, understanding, prediction, decision-making and action. Working memory can be stored in long-term memory through learning or memorization, and the response feedback is given to short-term sensory storage to affect working memory.
[0099] At present, the extraction of key areas mainly uses the saliency detection method, whose input is the sea surface situation image and the output is the salient area in the sea surface image. However, this method does not fully consider the characteristics of the sea surface situation, resulting in low accuracy in extracting key areas of the sea surface.
[0100] Therefore, in order to at least partially solve one or more of the above-mentioned problems and other potential problems, an embodiment of the present invention proposes a method for extracting key areas of the sea surface based on visual computing. In the method, the extraction of key areas of the sea surface is divided into two parts: extraction of significant targets on the sea surface and extraction of moving targets. For the extraction of moving targets, the present embodiment designs an encoder through a cross-correlation attention module and a deep convolutional network to encode a multi-frame feature map that aggregates the global context information of the sea surface situation image, thereby decoding to obtain a multi-frame optical flow map to achieve effective positioning of dynamic targets. In this way, the present embodiment comprehensively determines the key areas of the sea surface based on the multi-frame optical flow map representing the moving target and the detection results representing the significant target, thereby effectively improving the accuracy of the extraction of key areas of the sea surface.
[0101] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of a method for extracting key sea surface areas based on visual computing according to an embodiment of the present invention. Figure 2 As shown, the method may include steps S11 to S13:
[0102] Step S11: input multiple frames of sea surface situation images into the encoder, and obtain multiple frame feature maps that aggregate the global context information of the sea surface situation images through processing by the deep convolutional network in the encoder and the cross-correlation attention module connected in series with the deep convolutional network.
[0103] In this embodiment, for extracting the key sea area in the sea situation image, multiple frames of sea situation images can be input into an encoder. The encoder of this embodiment at least includes: a deep convolutional network, and a cross-correlation attention module connected in series with the deep convolutional network. After the multiple frames of sea situation images are processed by the cross-correlation attention module of the deep convolutional network in turn, a multi-frame feature map can be obtained, and the feature map aggregates the global context information of the sea situation image. Among them, each frame of the sea situation image in the multi-frame sea situation image and each frame of the feature map in the multi-frame feature map Figure 1 One to one correspondence.
[0104] In an optional implementation, the deep convolutional network can be replaced by a convolutional network with the latest performance, such as hyperparameter convolution, residual convolution, etc., which is not limited in this embodiment.
[0105] Preferably, the sea surface situation image in this embodiment may be hierarchically represented before being input into the encoder, that is, a plurality of frames of sea surface situation image that have been hierarchically represented may be input into the encoder for processing.
[0106] Step S12: input the multi-frame feature map into the decoder for decoding, and obtain the multi-frame optical flow map output by the decoder.
[0107] In this embodiment, after obtaining the multi-frame feature map output by the encoder, the multi-frame feature map can be input into the decoder corresponding to the encoder for decoding, and restored to the original size of the image for easy observation, and the output of the decoder is obtained: a multi-frame optical flow map. In this embodiment, the purpose of optical flow estimation (i.e., determining the optical flow map) is to find the corresponding matching relationship of the same pixel point in the process of changing two consecutive frames of image sequences, so as to estimate the movement size and direction of the pixel point.
[0108] Step S13: determining the key sea surface areas in the multi-frame sea surface situation images according to the multi-frame optical flow images and the detection results obtained by performing saliency detection on the multi-frame sea surface situation images.
[0109] In this embodiment, for multiple frames of sea surface situation images, on the one hand, the multiple frames of sea surface situation images are processed by the above-mentioned codec to obtain multiple frames of optical flow maps; on the other hand, saliency detection is performed on the multiple frames of sea surface situation images to obtain detection results; thus, after obtaining the multiple frames of optical flow maps and the detection results, the key sea surface areas in the multiple frames of sea surface situation images are determined based on the multiple frames of optical flow maps and the detection results obtained by performing saliency detection on the multiple frames of sea surface situation images.
[0110] Considering that the current research on attention mechanism is mainly focused on target positioning and recognition, there is no situational awareness perspective to study the attention mechanism. In fact, there is a tendency to focus on key sea targets, key sea areas, etc., and accordingly give more emphasis on decision-making and action. Therefore, this embodiment expands the connotation and scope of the attention mechanism. Based on the research on the attention mechanism in psychology and situational awareness theory, from the perspective of sea situation awareness, the attention mechanism is defined as: a situational awareness mechanism that focuses on key sea situation elements. The focus will change dynamically, which serves to form a high-level cognitive result of the decision maker on the sea situation and lays the foundation for subsequent decision-making and action.
[0111] Based on this, this embodiment mainly considers bottom-up situation data drive, and pays more attention to the key points highlighted in the situation information (including sudden unexpected situations on the sea surface, etc.) according to the current perceived situation (general situation map).
[0112] Specifically, this embodiment takes into account that people's attention is more easily attracted by objects, especially moving objects or moving parts of objects, and moving targets in the real sea surface are more worthy of attention. The extraction of key areas of the sea surface is divided into two parts: sea surface salient targets and moving target extraction: For the extraction of sea surface salient targets, this embodiment obtains the detection results by performing saliency detection on multiple frames of sea surface situation images; for the extraction of moving targets, this embodiment uses a cross-correlation attention module and a deep convolutional network to calculate optical flow (a type of motion estimation) to effectively locate dynamic targets, so as to enhance the extraction of dynamic targets.
[0113] In a preferred embodiment, based on the hierarchical representation of multiple frames of sea surface situation images, a deep neural network based on visual computing is further used to predict key areas of the sea surface, imitating brain-like observation to alleviate information overload and greatly reduce the uncertainty of the sea surface observation space.
[0114] In this embodiment, instead of only considering the apparently significant feature of the sea surface saliency, the moving targets on the sea surface are also considered, and the extraction of the key areas of the sea surface is divided into two parts: extraction of significant targets on the sea surface and extraction of moving targets; for the extraction of moving targets, an encoder is designed through a cross-correlation attention module and a deep convolutional network to encode a multi-frame feature map that aggregates the global context information of the sea surface situation image, thereby decoding to obtain a multi-frame optical flow map to achieve effective positioning of dynamic targets. In this way, based on the multi-frame optical flow map representing the moving target and the detection results representing the significant target, the key areas of the sea surface are comprehensively determined, which can effectively improve the accuracy of the extraction of the key areas of the sea surface. The key areas of the sea surface extracted in this embodiment can be used for marine target detection. Based on more accurate sea surface key areas for marine target detection, the analysis scope of marine target detection can be narrowed and the detection efficiency of marine target detection can be improved.
[0115] In combination with the above embodiments, in one implementation, the present invention further provides a method for extracting key areas of a sea surface based on visual computing, in which the "obtaining a multi-frame feature map of aggregated global context information of a sea surface situation image by processing a deep convolutional network in the encoder and a cross-correlation attention module connected in series with the deep convolutional network" in the above step S11 may specifically include steps S21 to S22:
[0116] Step S21: extracting image features of the sea surface situation image at 1 / 2 resolution and 1 / 4 resolution in turn through two convolution layers with 3×3 convolution kernels in the deep convolution network.
[0117] In this embodiment, the deep convolution network in the encoder includes two convolution layers with a convolution kernel of 3×3, which are used to extract features of images at 1 / 2 resolution and 1 / 4 resolution. In this way, multiple frames of sea surface situation images are input into the encoder, and features are extracted through the deep convolution network. Specifically, the image features of the sea surface situation images at 1 / 2 resolution and 1 / 4 resolution are extracted in sequence through two convolution layers with a convolution kernel of 3×3 in the deep convolution network.
[0118] Step S22: Input the image features at the 1 / 4 resolution into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module.
[0119] In this embodiment, after obtaining the image features of the sea surface situation image output by the deep convolutional network at 1 / 4 resolution, the image features at 1 / 4 resolution are input into the cross-correlation attention module connected in series with the deep convolutional network, and a multi-frame feature map output by the cross-correlation attention module is obtained in order to extract the optical flow features. The feature map output by the cross-correlation attention module in this embodiment includes at least: local features of large displacement moving targets and global features of small displacement moving targets; wherein, the large displacement moving target is a target whose motion displacement is greater than the first threshold, and the small displacement moving target is a target whose motion displacement is less than the second threshold, and the first threshold and the second threshold can be arbitrarily set according to actual needs, and the first threshold can be the same as or different from the second threshold, and this embodiment does not limit this.
[0120] In combination with the above embodiments, in one implementation, the present invention further provides a sea surface key area extraction method based on visual computing, in which the cross-correlation attention module includes a plurality of attention mechanism modules connected in series, and a convolution layer with a convolution kernel of 3×3 is provided between each attention mechanism module to change the number of channels, and the above step S22 may specifically include step S31:
[0121] Step S31: Input the image features at the 1 / 4 resolution into the first attention mechanism module until a multi-frame feature map with a resolution of 1 / 64 is output by the last attention mechanism module.
[0122] In this embodiment, after obtaining the image features at 1 / 4 resolution output by the deep convolutional network, the image features at 1 / 4 resolution are input into the first attention mechanism module in the cross-correlation attention module to obtain the output of the first attention mechanism module, and then the output of the first attention mechanism module is input into the next attention mechanism module for processing until a multi-frame feature map with a resolution of 1 / 64 output by the last attention mechanism module is obtained.
[0123] That is to say, in the cross-correlation attention module of this embodiment, except for the first attention mechanism module, the input of each attention mechanism module is the output of the previous attention mechanism module, and the input of the first attention mechanism module is the output of the deep convolutional network.
[0124] Among them, the processing of each attention mechanism module can be expressed by the following formula:
[0125] ;
[0126] in, Represents the output of the previous attention mechanism module, The representative layer is normalized. represents the processing using a multi-layer perceptron, CSwin_Attention represents the inverse operation of the decoding operation performed by the decoder, It is The intermediate output of the attention mechanism module, It is The output of an attention mechanism module.
[0127] In real sea scenes, some objects are far away and some are near. Small objects with large displacement are common in situation scene graphs. In order to improve the attention of such objects, this embodiment designs a cross-correlation attention module. By stacking the attention mechanism in layers, a feature map of the global context information of the aggregated image is finally obtained, which is conducive to generating large displacement motion clues, thereby restoring the diluted large displacement motion local features and small target motion features.
[0128] Preferably, after obtaining the image features at 1 / 4 resolution output by the deep convolutional network, the output image features at 1 / 4 resolution can be passed through a standard convolutional layer and used as the input of the first attention mechanism module and input into the cross-correlation attention module.
[0129] In one embodiment, the final output of the encoder for extracting optical flow features based on the cross-correlation attention module can be expressed as:
[0130] ;
[0131] in, is the input of the cross-correlation attention module, , Represents three convolution layers with 3X3 convolution kernels, CS represents the operation of the cross-correlation attention module. When the cross-correlation attention module includes four attention mechanism modules, CS represents the operation of four attention mechanism modules.
[0132] In this embodiment, the cross-correlation attention module can aggregate global long-distance contextual information from the input information. Benefiting from the attention mechanism's ability to focus on global information and capture long-range dependencies, the network can improve its ability to extract features in large displacement areas while maintaining the original detection accuracy, thereby improving the performance of optical flow estimation in large displacement areas, which helps to improve the situation observation of small targets and large displacement dynamic targets.
[0133] In combination with the above embodiments, the present invention further provides a method for extracting sea surface key areas based on visual computing, in which the multi-frame sea surface situation image includes at least three consecutive frames of sea surface situation images, the multi-frame feature map includes at least a first frame feature map, a second frame feature map and a third frame feature map respectively corresponding to the three consecutive frames of sea surface situation images, and the above step S12 may be a decoding operation performed by a decoder, and the decoding operation may specifically include steps S41 to S43:
[0134] Step S41: performing correlation calculations on the first frame feature map and the second frame feature map, and on the second frame feature map and the third frame feature map, respectively, to obtain correlation results.
[0135] In this embodiment, after obtaining the first frame feature map, the second frame feature map and the third frame feature map corresponding to three consecutive frames of sea surface situation images output by the encoder, a correlation calculation can be performed between two adjacent feature maps, that is, the correlation calculation is performed between the first frame feature map and the second frame feature map, and the correlation calculation is performed between the second frame feature map and the third frame feature map to obtain a correlation result.
[0136] Step S42: performing occlusion decoding and optical flow decoding on the correlation result respectively to obtain an occlusion decoding result and an optical flow decoding result.
[0137] In this embodiment, after the correlation result is obtained, the correlation result can be occluded and decoded by an occlusion decoder in the decoder to obtain an occlusion decoding result; and the correlation result can be optically decoded by an optical flow decoder in the decoder to obtain an optical flow decoding result. Among them, the occlusion decoding of this embodiment is to use the relationship between the previous and next frames to determine the decoding of the occluded position; and the optical flow decoding is to reversely calculate the speed of each pixel value from the extracted features.
[0138] Step S43: obtaining the multi-frame optical flow map based on the occlusion decoding result and the optical flow decoding result.
[0139] In this embodiment, after obtaining the occlusion decoding result and the optical flow decoding result, processing can be performed based on the occlusion decoding result and the optical flow decoding result to obtain a multi-frame optical flow map. The multi-frame optical flow map of this embodiment includes at least: a first optical flow map corresponding to the first frame feature map and the second frame feature map, and a second optical flow map corresponding to the second frame feature map and the third frame feature map.
[0140] In combination with the above embodiments, in one implementation, the present invention further provides a method for extracting key sea surface areas based on visual computing. In the method, the above step S43 may specifically include steps S51 to S53:
[0141] Step S51: Processing the occlusion decoding result through an occlusion context network to obtain an optimized occlusion decoding graph.
[0142] In this embodiment, the decoder further includes: an occlusion context network, an optical flow context network and a bilateral refinement network. After obtaining the occlusion decoding result, the occlusion decoding result can be processed by the occlusion context network to obtain an optimized occlusion decoding graph.
[0143] Step S52: Processing the optical flow decoding result through the optical flow context network to obtain an optimized optical flow decoding graph.
[0144] In this embodiment, after the optical flow decoding result is obtained, the optical flow decoding result can be processed by the optical flow context network to obtain an optimized optical flow decoding graph.
[0145] Step S53: inputting the optimized occlusion decoding map and the optimized optical flow decoding map into a bilateral refinement network to obtain the multi-frame optical flow map output by the bilateral refinement network.
[0146] In this embodiment, after obtaining the optimized occlusion decoding map and the optimized optical flow decoding map, the optimized occlusion decoding map and the optimized optical flow decoding map can be input into the bilateral refinement network to optimize the optical flow and obtain the optimized multi-frame optical flow map output by the bilateral refinement network.
[0147] In combination with the above embodiments, in one implementation, the present invention further provides a sea surface key area extraction method based on visual computing. In the method, in addition to the above steps, steps S61 and S62 may be further included. The above step S42 may specifically include step S63. The above step S51 may specifically include step S64 and step S65. The above step S52 may specifically include step S66 and step S67. Furthermore, the above step S53 may specifically include step S68:
[0148] Step S61: up-sampling the first optical flow map and the second optical flow map obtained by the previous round of decoding operation respectively to obtain a first sampled optical flow map and a second sampled optical flow map.
[0149] In this embodiment, the decoder can perform multiple rounds of decoding operations based on the input multi-frame feature maps, wherein each round performs decoding operations on the first frame feature map, the second frame feature map, and the third frame feature map corresponding to three consecutive frames of sea surface situation images.
[0150] After the decoder outputs the first optical flow map and the second optical flow map through the current round of decoding operation, the first optical flow map and the second optical flow map will be upsampled to act on the next round of decoding operation. That is to say, before the next round of decoding operation starts, the first optical flow map and the second optical flow map obtained by the previous round of decoding operation will be upsampled to obtain the first sampled optical flow map and the second sampled optical flow map.
[0151] Step S62: adding and fusing the first sampled optical flow map and the correlation result to obtain a first sampled correlation result, and adding and fusing the second sampled optical flow map and the correlation result to obtain a second sampled correlation result.
[0152] In this embodiment, after obtaining the first sampling optical flow map, the first sampling optical flow map and the correlation result obtained in the current round can be added and fused to obtain the first sampling correlation result; and after obtaining the second sampling optical flow map, the second sampling optical flow map and the correlation result obtained in the current round can be added and fused to obtain the second sampling correlation result.
[0153] Step S63: performing occlusion decoding on the first sampling correlation result to obtain a first sampling occlusion decoding result, and performing optical flow decoding on the second sampling correlation result to obtain a second sampling optical flow decoding result.
[0154] In this embodiment, in the current round of decoding operation, the first sampling correlation result is occluded and decoded by the occlusion decoder to obtain the first sampling occlusion decoding result; and the second sampling correlation result is optical flow decoded by the optical flow decoder to obtain the second sampling optical flow decoding result.
[0155] Step S64: adding and fusing the first sampled optical flow map and the first sampled occlusion decoding result to obtain a fused sampled occlusion decoding result.
[0156] In this embodiment, after obtaining the first sampling occlusion decoding result, it is necessary to first add and fuse the first sampling optical flow map and the first sampling occlusion decoding result to obtain a fused sampling occlusion decoding result.
[0157] Step S65: Processing the fused sampling occlusion decoding result through the occlusion context network to obtain an optimized fused sampling occlusion decoding graph.
[0158] In this embodiment, the fused sampling occlusion decoding result is input into the occlusion context network, and the fused sampling occlusion decoding result is processed by the occlusion context network to obtain an optimized fused sampling occlusion decoding graph.
[0159] Step S66: adding and fusing the second sampled optical flow map and the second sampled optical flow decoding result to obtain a fused sampled optical flow decoding result.
[0160] In this embodiment, after obtaining the second sampled optical flow decoding result, it is necessary to first add and fuse the second sampled optical flow map and the second sampled optical flow decoding result to obtain a fused sampled optical flow decoding result.
[0161] Step S67: Processing the fused sampled optical flow decoding result through the optical flow context network to obtain an optimized fused sampled optical flow decoding graph.
[0162] In this embodiment, the fused sampled optical flow decoding result is input into the optical flow context network, and the fused sampled optical flow decoding result is processed by the optical flow context network to obtain an optimized fused sampled optical flow decoding graph.
[0163] Step S68: inputting the optimized fused sampled occlusion decoding map and the optimized fused sampled optical flow decoding map into the bilateral refinement network to obtain a multi-frame optimized optical flow map.
[0164] In this embodiment, the optimized fused sampled occlusion decoding map and the optimized fused sampled optical flow decoding map are input into the bilateral thinning network to obtain a multi-frame optimized optical flow map.
[0165] In this embodiment, the optical flow map outputted from the previous round of decoding operation is used for the next round of decoding operation, and a multi-frame optical flow map with higher accuracy can be outputted, thereby further improving the accuracy of the extraction of key areas of the sea surface.
[0166] In combination with the above embodiments, in one implementation, the present invention further provides a method for extracting key sea surface areas based on visual computing. In this method, in addition to the above steps, steps S71 to S74 may also be included, and the above step S41 may specifically include step S75:
[0167] Step S71: up-sample the second optical flow map obtained by the previous round of decoding operation to obtain a second sampled optical flow map.
[0168] In this embodiment, the decoder can perform multiple rounds of decoding operations based on the input multi-frame feature maps, wherein each round performs decoding operations on the first frame feature map, the second frame feature map, and the third frame feature map corresponding to three consecutive frames of sea surface situation images.
[0169] After the decoder outputs the first optical flow map and the second optical flow map through the current round of decoding operation, the first optical flow map and the second optical flow map are also upsampled to act on the next round of decoding operation. In this embodiment, before the next round of decoding operation begins, the second optical flow map obtained by the previous round of decoding operation is also upsampled to obtain a second sampled optical flow map.
[0170] Step S72: adding and fusing the second sampling optical flow map and the second frame feature map to obtain a second frame sampling feature map.
[0171] Step S73: adding and fusing the second sampling optical flow map and the third frame feature map to obtain a third frame sampling feature map.
[0172] In this embodiment, after obtaining the second sampled optical flow map, the second sampled optical flow map and the second frame feature map can be added and fused to obtain the second frame sampling feature map after addition and fusion; and the second sampled optical flow map and the third frame feature map can be added and fused to obtain the third frame sampling feature map after addition and fusion.
[0173] Step S74: deforming the second frame sampling feature map and the third frame sampling feature map respectively to obtain a second frame deformation feature map and a third frame deformation feature map.
[0174] In this embodiment, between the feature maps, one of the maps is deformed to perform correlation calculation with the other map. After obtaining the second frame sampling feature map and the third frame sampling feature map, the second frame sampling feature map and the third frame sampling feature map can be deformed respectively to obtain the second frame deformation feature map after the second frame sampling feature map is deformed, and to obtain the third frame deformation feature map after the third frame sampling feature map is deformed.
[0175] Step S75: performing correlation calculation on the first frame feature map and the second frame deformation feature map, and performing correlation calculation on the second frame feature map and the third frame deformation feature map to obtain the correlation result.
[0176] In this embodiment, when performing correlation calculation of feature maps, the correlation calculation is performed between the feature map of the first frame and the deformed feature map of the second frame after deformation; and the correlation calculation is performed between the feature map of the second frame and the deformed feature map of the third frame after deformation to obtain a correlation result.
[0177] In one embodiment, if Figure 3 As shown, Figure 3 FIG. 1 is a schematic diagram of extracting key sea surface areas based on saliency and dynamic targets according to an embodiment of the present invention. Figure 3 In the figure, the upper part is for the extraction of salient targets on the sea surface: multiple frames of sea surface situation images are subjected to saliency detection to obtain detection result ①; the lower part is for the extraction of moving targets: multiple frames of sea surface situation images are input into the encoder to obtain multiple frames of feature maps, which are then input into the decoder for processing to obtain multiple frames of optical flow maps ②. Finally, a comprehensive situation judgment and decision is made based on the detection result ① and the multiple frames of optical flow maps ② to obtain the key areas of the sea surface, so as to detect sea targets based on the key areas of the sea surface, thereby narrowing the analysis scope of sea target detection and improving the detection efficiency of sea target detection.
[0178] Among them, the encoder includes a deep convolutional network and a cross-correlation attention module connected in series with the deep convolutional network. The cross-correlation attention module includes 4 attention mechanism modules connected in series, and there is a convolution layer with a convolution kernel of 3×3 between each attention mechanism module.
[0179] The decoder includes at least an occlusion decoder, an optical flow decoder, an occlusion context network, an optical flow context network and a bilateral refinement network. After multiple frames of feature maps are input into the decoder, when performing correlation calculation, one of the feature maps is deformed to perform correlation calculation with another feature map to obtain a correlation result; then, the correlation result is respectively input into the occlusion decoder and the optical flow decoder, and occlusion decoding and optical flow decoding are performed respectively to obtain an occlusion decoding result and an optical flow decoding result; then, the occlusion decoding result is processed by the occlusion context network to obtain an optimized occlusion decoding map; the optical flow decoding result is processed by the optical flow context network to obtain an optimized optical flow decoding map; finally, the optimized occlusion decoding map and the optimized optical flow decoding map are input into the bilateral refinement network to obtain a multi-frame optical flow map output by the bilateral refinement network.
[0180] And, after obtaining the multi-frame optical flow map output by the current round decoding operation, the multi-frame optical flow map can be used for the next round decoding operation, specifically: the optical flow map output by the current round is upsampled to obtain a sampled optical flow map, and then before the feature map of the next round is deformed, the sampled optical flow map is added and fused with the feature map and then deformed to perform correlation calculation. And, the optical flow map output by the current round is upsampled to obtain a sampled optical flow map, which is added and fused with the correlation result and then respectively input into the occlusion decoder and the optical flow decoder for occlusion decoding and optical flow decoding. And, the optical flow map output by the current round is upsampled to obtain a sampled optical flow map, which is added and fused with the occlusion decoding result and the optical flow decoding result, and then respectively input into the occlusion context network and the optical flow context network for processing to improve the extraction accuracy of the optical flow map of the next round.
[0181] In this embodiment, we explore learning from humans and use brain-like attention observation mechanisms to conduct research on sea surface situation perception mechanisms. In the sea surface, key areas not only look significant, but moving targets are more important than static targets. Therefore, this embodiment introduces visual computing to determine and locate the key areas of greatest concern, so as to extract key areas of the sea surface based on both significant targets and moving targets, thereby improving the accuracy of sea surface key area extraction.
[0182] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0183] Based on the same inventive concept, an embodiment of the present invention provides a device 400 for extracting key sea surface areas based on visual computing. Figure 4 , Figure 4 FIG. 1 is a structural block diagram of a device for extracting key sea surface areas based on visual computing provided by an embodiment of the present invention. Figure 4 As shown, the device 400 includes:
[0184] The encoding module 401 is used to input multiple frames of sea surface situation images into an encoder, and obtain multiple frame feature maps that aggregate global context information of the sea surface situation images through processing by a deep convolutional network in the encoder and a cross-correlation attention module connected in series with the deep convolutional network;
[0185] A decoding module 402 is used to input the multi-frame feature map into a decoder for decoding, and obtain a multi-frame optical flow map output by the decoder;
[0186] The key area generating module 403 is used to determine the sea surface key area in the multiple frames of sea surface situation images according to the multiple frames of optical flow images and the detection results obtained by performing saliency detection on the multiple frames of sea surface situation images.
[0187] Optionally, the encoding module 401 includes:
[0188] A deep convolution module, used to extract image features of the sea surface situation image at 1 / 2 resolution and 1 / 4 resolution in sequence through two convolution layers with 3×3 convolution kernels in the deep convolution network;
[0189] A cross-correlation module is used to input the image features at the 1 / 4 resolution into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module, and the feature map at least includes: local features of large-displacement motion targets and global features of small-displacement motion targets, the large-displacement motion targets are targets with motion displacement greater than a first threshold, and the small-displacement motion targets are targets with motion displacement less than a second threshold.
[0190] Optionally, the cross-correlation attention module includes a plurality of attention mechanism modules connected in series, and a convolution layer with a convolution kernel of 3×3 is provided between each attention mechanism module;
[0191] The cross-correlation module comprises:
[0192] An attention module is used to input the image features at the 1 / 4 resolution into the first attention mechanism module until a multi-frame feature map with a resolution of 1 / 64 is output by the last attention mechanism module;
[0193] Except for the first attention mechanism module, the input of each attention mechanism module is the output of the previous attention mechanism module; the processing of each attention mechanism module is expressed by the following formula: ;
[0194] in, Represents the output of the previous attention mechanism module, The representative layer is normalized. represents the processing using a multilayer perceptron, CSwin_Attention represents the inverse operation of the decoding operation performed by the decoder, It is The intermediate output of the attention mechanism module, It is The output of an attention mechanism module.
[0195] Optionally, the multiple frames of sea surface situation images include at least three consecutive frames of sea surface situation images, and the multiple frames of feature graphs include at least a first frame feature graph, a second frame feature graph, and a third frame feature graph respectively corresponding to the three consecutive frames of sea surface situation images;
[0196] The decoding module 402 is used to perform the following decoding operations through the decoder, and the decoding module 402 includes:
[0197] A correlation module, used to calculate the correlation between the first frame feature map and the second frame feature map, and between the second frame feature map and the third frame feature map, to obtain a correlation result;
[0198] A first processing module, configured to perform occlusion decoding and optical flow decoding on the correlation result to obtain an occlusion decoding result and an optical flow decoding result;
[0199] The second processing module is used to obtain the multi-frame optical flow map based on the occlusion decoding result and the optical flow decoding result, and the multi-frame optical flow map at least includes: a first optical flow map corresponding to the first frame feature map and the second frame feature map, and a second optical flow map corresponding to the second frame feature map and the third frame feature map.
[0200] Optionally, the device 400 further includes:
[0201] A first sampling module, used for up-sampling the second optical flow map obtained by the previous round of decoding operation to obtain a second sampled optical flow map;
[0202] A first fusion module, used for adding and fusing the second sampled optical flow map and the second frame feature map to obtain a second frame sampled feature map;
[0203] A second fusion module is used to add and fuse the second sampled optical flow map and the third frame feature map to obtain a third frame sampled feature map;
[0204] A deformation module, used to perform deformation processing on the second frame sampling feature map and the third frame sampling feature map respectively to obtain a second frame deformation feature map and a third frame deformation feature map;
[0205] The correlation module comprises:
[0206] The third processing module is used to perform correlation calculation on the first frame feature map and the second frame deformation feature map, and perform correlation calculation on the second frame feature map and the third frame deformation feature map to obtain the correlation result.
[0207] Optionally, the second processing module includes:
[0208] An occlusion processing module, used to process the occlusion decoding result through an occlusion context network to obtain an optimized occlusion decoding graph;
[0209] An optical flow processing module, used for processing the optical flow decoding result through an optical flow context network to obtain an optimized optical flow decoding graph;
[0210] The optimization processing module is used to input the optimized occlusion decoding map and the optimized optical flow decoding map into a bilateral refinement network to obtain the multi-frame optical flow map output by the bilateral refinement network.
[0211] Optionally, the device 400 further includes:
[0212] A second sampling module is used to upsample the first optical flow map and the second optical flow map obtained by the previous round of decoding operation to obtain a first sampled optical flow map and a second sampled optical flow map;
[0213] A third fusion module is used to add and fuse the first sampled optical flow map and the correlation result to obtain a first sampled correlation result, and add and fuse the second sampled optical flow map and the correlation result to obtain a second sampled correlation result;
[0214] The first processing module comprises:
[0215] A first processing submodule, configured to perform occlusion decoding on the first sampling correlation result to obtain a first sampling occlusion decoding result, and perform optical flow decoding on the second sampling correlation result to obtain a second sampling optical flow decoding result;
[0216] The occlusion processing module comprises:
[0217] a fourth fusion module, configured to add and fuse the first sampled optical flow map and the first sampled occlusion decoding result to obtain a fused sampled occlusion decoding result;
[0218] An occlusion processing submodule, used to process the fused sampling occlusion decoding result through the occlusion context network to obtain an optimized fused sampling occlusion decoding graph;
[0219] The optical flow processing module comprises:
[0220] a fifth fusion module, configured to add and fuse the second sampled optical flow map and the second sampled optical flow decoding result to obtain a fused sampled optical flow decoding result;
[0221] An optical flow processing submodule, used for processing the fused sampled optical flow decoding result through the optical flow context network to obtain an optimized fused sampled optical flow decoding graph;
[0222] The optimization processing module comprises:
[0223] The optimization processing submodule is used to input the optimized fused sampling occlusion decoding map and the optimized fused sampling optical flow decoding map into the bilateral refinement network to obtain a multi-frame optimized optical flow map.
[0224] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the method for extracting key sea surface areas based on visual computing as described in any of the above embodiments of the present invention are implemented.
[0225] Based on the same inventive concept, another embodiment of the present invention provides an electronic device 500, such as Figure 5 shown. Figure 5 1 is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 502, a processor 501, and a computer program stored in the memory and executable on the processor, and when executed by the processor, the steps of the method for extracting key sea surface areas based on visual computing described in any of the above embodiments of the present invention are implemented.
[0226] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0227] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0228] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, embodiments of the present invention may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware. Furthermore, embodiments of the present invention may take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0229] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0230] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0231] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0232] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0233] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.
[0234] The above is a detailed introduction to the sea surface key area extraction method, device and product based on visual computing provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A method for extracting key sea surface areas based on visual computing, characterized in that: The method comprises: Inputting a plurality of frames of sea surface situation images into an encoder, and obtaining a plurality of frame feature maps that aggregate global context information of the sea surface situation images through processing by a deep convolutional network in the encoder and a cross-correlation attention module connected in series with the deep convolutional network; Inputting the multi-frame feature map into a decoder for decoding, and obtaining a multi-frame optical flow map output by the decoder; Determining a key sea surface area in the multi-frame sea surface situation image according to the multi-frame optical flow map and a detection result obtained by performing saliency detection on the multi-frame sea surface situation image; The multiple frames of sea surface situation images at least include: three consecutive frames of sea surface situation images, and the multiple frames of feature graphs at least include: a first frame feature graph, a second frame feature graph, and a third frame feature graph respectively corresponding to the three consecutive frames of sea surface situation images; The step of inputting the multi-frame feature map into a decoder for decoding to obtain a multi-frame optical flow map output by the decoder includes: The following decoding operations are performed by the decoder: Calculate the correlation between the first frame feature map and the second frame feature map, and between the second frame feature map and the third frame feature map, respectively, to obtain a correlation result; Performing occlusion decoding and optical flow decoding on the correlation result respectively to obtain an occlusion decoding result and an optical flow decoding result; Based on the occlusion decoding result and the optical flow decoding result, the multi-frame optical flow map is obtained, and the multi-frame optical flow map at least includes: a first optical flow map corresponding to the first frame feature map and the second frame feature map, and a second optical flow map corresponding to the second frame feature map and the third frame feature map.
2. The method for extracting key sea surface areas based on visual computing according to claim 1, characterized in that: The method obtains a multi-frame feature map of the global context information of the sea surface situation image by processing the deep convolutional network in the encoder and the cross-correlation attention module connected in series with the deep convolutional network, including: By using two convolution layers with 3×3 convolution kernels in the deep convolution network, image features of the sea surface situation image at 1 / 2 resolution and 1 / 4 resolution are extracted in sequence; The image features at the 1 / 4 resolution are input into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module, and the feature map includes at least: local features of large-displacement motion targets and global features of small-displacement motion targets, the large-displacement motion targets are targets with motion displacement greater than a first threshold, and the small-displacement motion targets are targets with motion displacement less than a second threshold.
3. The method for extracting key sea surface areas based on visual computing according to claim 2 is characterized in that: The cross-correlation attention module includes a plurality of attention mechanism modules connected in series, and a convolution layer with a convolution kernel of 3×3 is provided between each attention mechanism module; The step of inputting the image features at the 1 / 4 resolution into the cross-correlation attention module to obtain the multi-frame feature map output by the cross-correlation attention module includes: Input the image features at the 1 / 4 resolution into the first attention mechanism module until a multi-frame feature map with a resolution of 1 / 64 is output by the last attention mechanism module; Except for the first attention mechanism module, the input of each attention mechanism module is the output of the previous attention mechanism module; the processing of each attention mechanism module is expressed by the following formula: ; in, Represents the output of the previous attention mechanism module, The representative layer is normalized. represents the processing using a multilayer perceptron, CSwin_Attention represents the inverse operation of the decoding operation performed by the decoder, It is The intermediate output of the attention mechanism module, It is The output of an attention mechanism module.
4. The method for extracting key sea surface areas based on visual computing according to claim 1, characterized in that: The method further comprises: Upsampling the second optical flow map obtained by the previous round of decoding operation to obtain a second sampled optical flow map; Adding and fusing the second sampling optical flow map and the second frame feature map to obtain a second frame sampling feature map; Adding and fusing the second sampling optical flow map and the third frame feature map to obtain a third frame sampling feature map; Performing deformation processing on the second frame sampling feature map and the third frame sampling feature map respectively to obtain a second frame deformation feature map and a third frame deformation feature map; The first frame feature map and the second frame feature map are respectively correlated with each other, and the second frame feature map and the third frame feature map are respectively correlated with each other to obtain a correlation result, including: The first frame feature map and the second frame deformation feature map are correlated, and the second frame feature map and the third frame deformation feature map are correlated to obtain the correlation result.
5. The method for extracting key sea surface areas based on visual computing according to claim 1, characterized in that: The obtaining the multi-frame optical flow map based on the occlusion decoding result and the optical flow decoding result includes: Processing the occlusion decoding result through an occlusion context network to obtain an optimized occlusion decoding graph; Processing the optical flow decoding result through an optical flow context network to obtain an optimized optical flow decoding graph; The optimized occlusion decoding map and the optimized optical flow decoding map are input into a bilateral refinement network to obtain the multi-frame optical flow map output by the bilateral refinement network.
6. The method for extracting key sea surface areas based on visual computing according to claim 5, characterized in that: The method further comprises: Upsampling the first optical flow map and the second optical flow map obtained by the previous round of decoding operation respectively to obtain a first sampled optical flow map and a second sampled optical flow map; Adding and fusing the first sampled optical flow map and the correlation result to obtain a first sampled correlation result, and adding and fusing the second sampled optical flow map and the correlation result to obtain a second sampled correlation result; The step of performing occlusion decoding and optical flow decoding on the correlation result to obtain an occlusion decoding result and an optical flow decoding result comprises: Performing occlusion decoding on the first sampling correlation result to obtain a first sampling occlusion decoding result, and performing optical flow decoding on the second sampling correlation result to obtain a second sampling optical flow decoding result; The occlusion decoding result is processed by the occlusion context network to obtain an optimized occlusion decoding graph, including: Adding and fusing the first sampled optical flow map and the first sampled occlusion decoding result to obtain a fused sampled occlusion decoding result; Processing the fused sampling occlusion decoding result through the occlusion context network to obtain an optimized fused sampling occlusion decoding graph; The optical flow decoding result is processed by the optical flow context network to obtain an optimized optical flow decoding graph, including: Adding and fusing the second sampled optical flow map and the second sampled optical flow decoding result to obtain a fused sampled optical flow decoding result; The fused sampled optical flow decoding result is processed by the optical flow context network to obtain an optimized fused sampled optical flow decoding graph; The step of inputting the optimized occlusion decoding graph and the optimized optical flow decoding graph into a bilateral refinement network to obtain the multi-frame optical flow graph output by the bilateral refinement network comprises: The optimized fused sampled occlusion decoding map and the optimized fused sampled optical flow decoding map are input into the bilateral refinement network to obtain a multi-frame optimized optical flow map.
7. A device for extracting key sea surface areas based on visual computing, characterized in that: The device comprises: An encoding module, used for inputting a plurality of frames of sea surface situation images into an encoder, and obtaining a plurality of frame feature maps that aggregate the global context information of the sea surface situation images through processing by a deep convolutional network in the encoder and a cross-correlation attention module connected in series with the deep convolutional network; A decoding module, used for inputting the multi-frame feature map into a decoder for decoding, and obtaining a multi-frame optical flow map output by the decoder; the multi-frame sea surface situation image includes at least: three consecutive frames of sea surface situation images, and the multi-frame feature map includes at least: a first frame feature map, a second frame feature map and a third frame feature map respectively corresponding to the three consecutive frames of sea surface situation images; the multi-frame feature map is input into the decoder for decoding, and the multi-frame optical flow map output by the decoder is obtained, including: performing the following decoding operations by the decoder: respectively performing correlation calculation on the first frame feature map and the second frame feature map, and the second frame feature map and the third frame feature map, to obtain a correlation result; performing occlusion decoding and optical flow decoding on the correlation result, respectively, to obtain an occlusion decoding result and an optical flow decoding result; based on the occlusion decoding result and the optical flow decoding result, the multi-frame optical flow map is obtained, and the multi-frame optical flow map includes at least: a first optical flow map corresponding to the first frame feature map and the second frame feature map, and a second optical flow map corresponding to the second frame feature map and the third frame feature map; The key area generation module is used to determine the sea surface key area in the multi-frame sea surface situation image according to the multi-frame optical flow map and the detection results obtained by performing saliency detection on the multi-frame sea surface situation image.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the method for extracting key sea surface areas based on visual computing as described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting key sea surface areas based on visual computing as described in any one of claims 1 to 6 is implemented.
Citation Information
Patent Citations
Continuous frame RD image target identification method based on deep optical flow network
CN110348356A
Video anomaly detection method based on double-cross attention auto-encoder
CN116704408A
Multi-modal visual target tracking method based on self-distillation symmetric adapter
CN117710414A
