Video segmentation method, device, equipment and medium based on fusion coding network

Abstract

The present invention relates to the technical field of video segmentation, and discloses a video segmentation method, device, equipment and medium based on a fusion coding network. The method comprises: performing a series of processing on the reference frame image and the reference segmentation image based on the fusion coding network to obtain a reference convolution feature map and a reference fusion feature map; obtaining the current frame image in the video to be segmented, and inputting the current frame image into the fusion coding network to obtain the current resolution feature map; performing a series of processing on the current resolution feature map, the reference convolution feature map and the reference fusion feature map to obtain an output segmentation map; if the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed. The embodiment of the present invention can improve the efficiency and segmentation accuracy of video segmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of video segmentation, and in particular to a video segmentation method, device, equipment and medium based on a fusion coding network. Background Art

[0002] Video segmentation is to segment certain specific categories of objects in the input video frame image to obtain its segmentation mask image. Existing video segmentation includes unsupervised video segmentation and semi-supervised video segmentation, and semi-supervised video segmentation refers to the segmentation result of a given frame, which is used as a reference for the propagation of subsequent frames. However, the existing supervised video segmentation methods such as MATNet, FSNet, 3DC-Seg and RTNet have low video segmentation accuracy, and the existing semi-supervised video segmentation methods such as FEELVOS, STM, FRTM-VOS, GraphMemVOS, etc. have complex model architecture and slow model operation speed, which leads to low video segmentation efficiency. Summary of the invention

[0003] The embodiments of the present invention provide a video segmentation method, apparatus, device and medium based on a fusion coding network, aiming to solve the problem that the existing video segmentation efficiency and segmentation accuracy are both low.

[0004] In a first aspect, an embodiment of the present invention provides a video segmentation method based on a fusion coding network, which includes:

[0005] Acquire a reference frame image specified by a user and a reference segmented image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented;

[0006] Inputting the reference frame image into a fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map;

[0007] Inputting the reference resolution feature map and the reference segmented image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlap expansion to obtain a reference fusion feature map;

[0008] Acquire a current frame image in the video to be segmented, input the current frame image into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map, and perform convolution processing on the third current resolution feature map to obtain a current convolution feature map;

[0009] Performing attention weight calculation on the current convolution feature map, the reference convolution feature map, and the reference fusion feature map to obtain a weighted feature map;

[0010] Decoding the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map;

[0011] If the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed.

[0012] In a second aspect, an embodiment of the present invention further provides a video segmentation device based on a fusion coding network, comprising:

[0013] An acquisition unit, used to acquire a reference frame image specified by a user and a reference segmented image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented;

[0014] A first coding convolution unit, used for inputting the reference frame image into a fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map;

[0015] A cascade fusion unit, used for inputting the reference resolution feature map and the reference segmented image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlap expansion to obtain a reference fusion feature map;

[0016] A second encoding convolution unit is used to obtain a current frame image in the video to be segmented, and input the current frame image into the fusion encoding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map, and perform convolution processing on the third current resolution feature map to obtain a current convolution feature map;

[0017] An attention unit, used to calculate the attention weights of the current convolution feature map, the reference convolution feature map and the reference fusion feature map to obtain a weighted feature map;

[0018] A decoding unit, configured to decode the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map;

[0019] An updating unit is used to update the reference convolution feature map and the reference fusion feature map if the feature map update condition is met until the segmentation of the video to be segmented is completed.

[0020] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above method when executing the computer program.

[0021] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program can implement the above method when executed by a processor.

[0022] The embodiments of the present invention provide a method, apparatus, device and medium for video segmentation based on a fusion coding network. The method includes: obtaining a reference frame image specified by a user and a reference segmentation image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented; inputting the reference frame image into the fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map; inputting the reference resolution feature map and the reference segmentation image into the fusion coding network to obtain a reference coding feature map, and performing cascading and fusion processing on the reference coding feature map and the reference resolution feature map after overlapping and expansion to obtain a reference fusion feature map; obtaining the current frame image in the video to be segmented, and The current frame image is input into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map and a third current resolution feature map, and the third current resolution feature map is convolved to obtain a current convolution feature map; the current convolution feature map, the reference convolution feature map and the reference fusion feature map are subjected to attention weight calculation to obtain a weight feature map; the weight feature map, the first current resolution feature map and the second current resolution feature map are decoded to obtain an output segmentation map; if the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed. The technical solution of the embodiment of the present invention can improve the efficiency of video segmentation to a certain extent by processing the reference frame image, the reference segmentation image and the current frame image through the fusion coding network; in the process of generating the output segmentation map, the accuracy of video segmentation can be improved by updating the reference convolution feature map and the reference fusion feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0024] Figure 1A schematic diagram of a flow chart of a video segmentation method based on a fusion coding network provided by an embodiment of the present invention;

[0025] Figure 2 A schematic diagram of a sub-process of a video segmentation method based on a fusion coding network provided by an embodiment of the present invention;

[0026] Figure 3 A schematic diagram of a sub-process of a video segmentation method based on a fusion coding network provided by an embodiment of the present invention;

[0027] Figure 4 A schematic diagram of a sub-process of a video segmentation method based on a fusion coding network provided by an embodiment of the present invention;

[0028] Figure 5 A schematic block diagram of a video segmentation device based on a fusion coding network provided by an embodiment of the present invention; and

[0029] Figure 6 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0032] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0033] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0035] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a video segmentation method based on a fusion coding network provided by an embodiment of the present invention. The video segmentation method based on a fusion coding network is described in detail below. Figure 1 As shown, the method includes the following steps S100-S160.

[0036] S100, obtaining a reference frame image specified by a user and a reference segmented image corresponding to the reference frame image, wherein the reference frame image is any frame image in a video to be segmented.

[0037] In an embodiment of the present invention, a user uploads a video to be segmented in a video segmentation software, and specifies any frame image in the video to be segmented as a reference frame image, and processes the reference frame image by a preset processing method to obtain a reference segmentation image, wherein the preset processing method is to process the reference frame image by one of a smearing method, a hand-drawing method, or a click-to-select target method to obtain a reference segmentation image. It should be noted that in an embodiment of the present invention, the reference frame image is an image after certain specific categories of objects have been segmented. It should also be noted that the video segmentation software in an embodiment of the present invention is software written based on the STCN network architecture, wherein STCN is composed of three modules: a fusion encoding (Encoder) network, a decoding (decoder) network, and an associative memory network. Among them, the fusion encoding network is divided into three submodules, corresponding to: KeyEncoder network, ValueEncoderSO network and ValueEncoder network; the decoding network is a simple CNN network superimposed based on the output of three different resolutions for feature fusion; the associative memory network is used to solve the approximate feature map of the fusion feature.

[0038] S110, inputting the reference frame image into a fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map.

[0039] In an embodiment of the present invention, after obtaining the reference frame image and the reference segmented image, the reference frame image is input into a fusion coding network. Since there is only one input, the reference frame image, the first coding network in the fusion coding network is selected to perform feature extraction to obtain a reference resolution feature map, wherein the first coding network is a KeyEncoder network; the reference resolution feature map includes a first reference resolution feature map, a second reference resolution feature map, and a third reference resolution feature map; the third reference resolution feature map is convolved to obtain a reference convolution feature map. For ease of understanding, an example is given below, assuming that the reference frame image is 1╳3╳320╳432, the reference frame image 1╳3╳320╳432 is input into the KeyEncoder network to perform feature extraction to obtain a 1 / 4 reference resolution feature map. Figure 1 ╳24╳80╳108, 1 / 8 reference resolution characteristics Figure 1 ╳32╳40╳54 and 1 / 16 reference resolution features Figure 1 ╳96╳20╳27, and 1 / 16 resolution feature Figure 1 ╳96╳20╳27 performs convolution operation to obtain reference convolution features Figure 1 ╳64╳20╳27. It should be noted that, in the present invention, the ValueEncoderSO network and the ValueEncoder network are the second encoding network.

[0040] S120, inputting the reference resolution feature map and the reference segmented image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlapping and expansion to obtain a reference fusion feature map.

[0041] In the embodiment of the present invention, assuming that the reference segmented image is K╳1╳320╳432, after obtaining the reference convolution feature map, the third reference resolution feature map is Figure 1 ╳96╳20╳27 and the reference segmentation image K╳1╳320╳432 are input into the second encoding network in the fusion encoding network to obtain the reference encoding feature map K╳96╳20╳27. It should be noted that in practical applications, if K=1 is single target tracking, the ValueEncoderSO network is used, and when K>1 is multi-target tracking, the ValueEncoder network is used. Figure 1╳96╳20╳27 is expanded by a repeat function to obtain a K╳96╳20╳27 feature map, and then the K╳96╳20╳27 feature map is cascaded with the reference segmentation image K╳1╳320╳432 to obtain a cascade feature map K╳192╳20╳27, and finally the cascade feature map K╳192╳20╳27 is residually fused to obtain a reference fused feature map K╳96╳20╳27.

[0042] S130, obtaining a current frame image in the video to be segmented, and inputting the current frame image into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map, and performing convolution processing on the third current resolution feature map to obtain a current convolution feature map.

[0043] In an embodiment of the present invention, the reference convolution feature is obtained Figure 1 ╳64╳20╳27 and the reference fusion feature map K╳96╳20×27, the current frame image 1╳3╳320╳432 in the video to be segmented is obtained. It can be understood that the current frame image is any frame image in the remaining frames except the reference frame image. The current frame image 1╳3╳320╳432 is input into the first coding network in the fusion coding network to perform layer-by-layer feature extraction to obtain the first current resolution feature map. Figure 1 ╳24╳80╳108, Second Current Resolution Feature Figure 1 ╳32╳40╳54 and the third current resolution feature Figure 1 ╳96╳20╳27, the third current resolution feature map is convolved to obtain a current convolution feature map, wherein the convolution process includes a first convolution process and a second convolution process, and correspondingly, the current convolution feature map includes a first current convolution feature map and a second current convolution feature map. It should be noted that in an embodiment of the present invention, the first convolution process and the second convolution process are implemented in the KeyProj module and the KeyComp module, respectively, and the KeyProj module and the KeyComp module are convolution operation modules, and the first current convolution feature map and the second current convolution feature map are 1╳64╳20╳27 and 1╳96╳20╳27, respectively.

[0044] S140, performing attention weight calculation on the current convolution feature map, the reference convolution feature map and the reference fusion feature map to obtain a weighted feature map.

[0045] In this embodiment of the present invention, the first current convolution feature is Figure 1 ╳64╳20╳27, the second current convolution feature Figure 1 ╳96╳20╳27, the reference convolution feature Figure 1 ╳64╳20╳27 and the reference fusion feature map K╳96╳20╳27 are used to calculate the attention weight to obtain the weight feature map. It should be noted that in the embodiment of the present invention, the accuracy of video segmentation can be improved by using the associative memory network.

[0046] See also Figure 2 In one embodiment, for example, in an embodiment of the present invention, the step S140 includes the following steps S141-S142.

[0047] S141, performing dimensionality reduction, multiplication, subtraction, and normalization operations on the first current convolution feature map and the reference convolution feature map to obtain an attention feature map;

[0048] S142. Perform attention weight calculation on the attention feature map, the second current convolution feature map and the reference fusion feature map to obtain a weighted feature map.

[0049] In this embodiment of the present invention, the first current convolution feature Figure 1 ╳64╳20╳27 and the reference convolution feature Figure 1 ╳64╳20╳27 performs dimensionality reduction, multiplication, subtraction and normalization operations in sequence to obtain attention features Figure 1 x5400x540, wherein the normalization operation is implemented through the Softmax layer. Get the attention feature Figure 1 After ╳5400╳540, the second current convolution feature Figure 1 ╳96╳20╳27 is traversed to obtain multiple traversal feature maps, that is, multi-target tracking is converted into single target tracking; dimensionality reduction processing is performed on each of the multiple traversal feature maps to obtain a reduced dimension feature map; the reduced dimension feature map and the attention feature map are matrix multiplied, dimensionally increased and cascaded to obtain a target attention feature map K╳96╳20╳27; the reference fusion feature map K╳96╳20╳27 after overlapping and expanding K times is cascaded with the target attention feature map K╳96╳20╳27 to obtain a weighted feature map K╳192╳20╳27.

[0050] S150, decoding the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map.

[0051] In an embodiment of the present invention, the weight feature map, the first current resolution feature map and the second current resolution feature map are decoded by a decoding network to obtain an output segmentation map, wherein the decoding process includes a residual operation, a sampling operation and a convolution operation. It should be noted that in an embodiment of the present invention, the residual operation is specifically composed of two layers of convolution. Assuming that its input is a feature map of 1╳in_channels╳H╳W, then the number of input kernels of the convolution layer is in_channels, and the number of output kernels is out_channels. Then the number of input kernels of the first layer of convolution is in_channels, the number of output kernels is out_channels, the kernel size is 3, the step size is 1, and the padding is 1. The number of input kernels of the second layer of convolution is out_channels, the number of output kernels is out_channels, the kernel size is 3, the step size is 1, and the padding is 1. After two layers of convolution, a 1×out_channels×H×W feature map is finally obtained.

[0052] See also Figure 3 In one embodiment, for example, in an embodiment of the present invention, the step S140 includes the following steps S151-S153.

[0053] S151, performing residual processing on the weight feature map to obtain a residual feature map;

[0054] S152, performing an upsampling operation on the residual feature map, the first current resolution feature map, and the second current resolution feature map to obtain a sampled feature map;

[0055] S153, performing convolution and upsampling operations on the sampling feature map to obtain an output segmentation map.

[0056] In the embodiment of the present invention, the weight feature map K╳192╳20╳27 is subjected to residual processing to obtain a residual feature map K×96╳20╳27, and the residual feature map K╳96╳20╳27 is compared with the second current resolution feature map. Figure 1 ╳32╳40╳54 is upsampled to obtain a K╳48╳40╳54 feature map, and then the K╳48╳40╳54 feature map is combined with the first current resolution feature map Figure 1╳24╳80╳108 is upsampled to obtain a sampling feature map K╳48╳80╳108, and finally the sampling feature map K╳48╳80╳108 is convolved. It should be noted that the number of input kernels of the last convolutional layer is 48, the number of output kernels is 1, the kernel size is 3, the step size is 1, and the padding is 1. Finally, the sampling feature map K╳1╳80╳108 is directly upsampled by 4 times to obtain the output segmentation map K╳1╳320╳432.

[0057] S160: If the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed.

[0058] In an embodiment of the present invention, in order to make the video segmentation more accurate, the reference convolution feature map and the reference fusion feature map are associated through an associative memory network during the segmentation process. Therefore, it is necessary to detect the feature map update condition. When the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated according to the output segmentation map until the segmentation of the video to be segmented is completed. It should be noted that in an embodiment of the present invention, the feature map update condition is that the frame count is an integer multiple of the preset frame number, wherein the preset frame number is 5. It can be understood that when the frame count is 5, 10, 15, etc., the reference convolution feature map and the reference fusion feature map are updated.

[0059] See also Figure 4 In one embodiment, for example, in an embodiment of the present invention, the step S140 includes the following steps S161-S163.

[0060] S161, detecting whether the frame count is an integer multiple of a preset frame number;

[0061] S162: If the frame count is an integer multiple of the preset frame number, inputting the output segmentation map, the current frame image, and the third current resolution feature map into the second encoding network for target tracking to obtain an updated feature map;

[0062] S163. Perform convolution processing on the updated feature map to obtain a first updated convolution feature map and a second updated convolution feature map, and use the first updated convolution feature map and the second updated convolution feature map as the reference convolution feature map and the reference fusion feature map.

[0063] In the embodiment of the present invention, it is detected whether the frame count is an integer multiple of the preset frame count. If the frame count is an integer multiple of the preset frame count, the output segmentation map K╳1╳320╳432, the current frame image 1╳3╳320╳432 and the third current resolution feature Figure 1 ╳96╳20╳27 is input into the second coding network for single target or multi-target tracking to obtain an updated feature map K╳96╳20╳27; the updated feature map K╳96╳20╳27 is subjected to a first convolution process and a second convolution process to obtain a first updated convolution feature map K╳96╳N╳20╳27 and a second updated convolution feature map Figure 1 ╳64╳N╳20╳27, and use the first updated convolution feature map and the second updated convolution feature map as the reference convolution feature map and the reference fusion feature map, wherein N is the number of stored information, which can be set to 10. It is understandable that if the frame count is not an integer multiple of the preset number of frames, for example, the frame count is 2, 3, 4, etc., the reference convolution feature map and the reference fusion feature map are not updated. It should be noted that in an embodiment of the present invention, the video segmentation model of the present invention is obtained by continuous fitting and training through loss functions such as BCE / F1 / SSIM / Laplacian pyramid. Compared with other video segmentation models, the video segmentation model of the present invention is more accurate in video segmentation and has higher segmentation efficiency.

[0064] Figure 5 FIG. 2 is a schematic block diagram of a video segmentation device 200 based on a fusion coding network provided by an embodiment of the present invention. Figure 5 As shown, corresponding to the above-mentioned video segmentation method based on the fusion coding network, the present invention also provides a video segmentation device 200 based on the fusion coding network. The video segmentation device 200 based on the fusion coding network includes a unit for executing the above-mentioned video segmentation method based on the fusion coding network, and the device can be configured in a terminal. Specifically, please refer to Figure 5 The video segmentation device 200 based on the fusion coding network includes an acquisition unit 201, a first coding convolution unit 202, a cascade fusion unit 203, a second coding convolution unit 204, an attention unit 205, a decoding unit 206 and an updating unit 207.

[0065] The acquisition unit 201 is used to acquire a reference frame image specified by a user and a reference segmentation image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented; the first coding convolution unit 202 is used to input the reference frame image into a fusion coding network to obtain a reference resolution feature map, and perform convolution processing on the reference resolution feature map to obtain a reference convolution feature map; the cascade fusion unit 203 is used to input the reference resolution feature map and the reference segmentation image into the fusion coding network to obtain a reference coding feature map, and perform cascade and fusion processing on the reference coding feature map and the reference resolution feature map after overlapping and expansion to obtain a reference fusion feature map; the second coding convolution unit 204 is used to acquire the current image in the video to be segmented. Frame image, and input the current frame image into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map and a third current resolution feature map, and convolve the third current resolution feature map to obtain a current convolution feature map; the attention unit 205 is used to calculate the attention weight of the current convolution feature map, the reference convolution feature map and the reference fusion feature map to obtain a weight feature map; the decoding unit 206 is used to decode the weight feature map, the first current resolution feature map and the second current resolution feature map to obtain an output segmentation map; the updating unit 207 is used to update the reference convolution feature map and the reference fusion feature map if the feature map update condition is met, until the segmentation of the video to be segmented is completed.

[0066] In some embodiments, such as the present embodiment, the second encoding convolution unit 204 includes an extraction unit.

[0067] Among them, the extraction unit is used to input the current frame image into the first coding network in the fusion coding network to perform layer-by-layer feature extraction to obtain a first current resolution feature map, a second current resolution feature map and a third current resolution feature map.

[0068] In some embodiments, such as this embodiment, the cascade fusion unit 203 includes an input unit, a first cascade unit, and a fusion unit.

[0069] Among them, the input unit is used to input the reference resolution feature map and the reference segmentation image into the second coding network in the fusion coding network for target tracking to obtain a reference coding feature map; the first cascade unit is used to overlap and expand the reference resolution, and cascade the reference resolution feature map after the overlap expansion with the reference coding feature map to obtain a reference cascade feature map; the fusion unit is used to perform residual fusion processing on the reference cascade feature map to obtain a reference fusion feature map.

[0070] In some embodiments, such as the present embodiment, the attention unit 205 includes a dimensionality reduction normalization unit and a weight calculation unit.

[0071] Among them, the dimensionality reduction and normalization unit is used to perform dimensionality reduction, multiplication, subtraction and normalization operations on the first current convolution feature map and the reference convolution feature map in sequence to obtain an attention feature map; the weight calculation unit is used to perform attention weight calculation on the attention feature map, the second current convolution feature map and the reference fusion feature map to obtain a weighted feature map.

[0072] In some embodiments, such as the present embodiment, the weight calculation unit includes a traversal unit, a dimension reduction unit, a calculation unit, and a second cascade unit.

[0073] Among them, the traversal unit is used to traverse the second current convolution feature map to obtain multiple traversal feature maps; the dimensionality reduction unit is used to perform dimensionality reduction processing on each of the multiple traversal feature maps to obtain a reduced dimensionality feature map; the calculation unit is used to multiply, increase the dimension and cascade the reduced dimensionality feature map with the attention feature map to obtain a target attention feature map; the second cascade unit is used to cascade the reference fusion feature map after overlapping expansion with the target attention feature map to obtain a weighted feature map.

[0074] In some embodiments, such as this embodiment, the decoding unit 206 includes a residual processing unit, a sampling unit, and an output unit.

[0075] Among them, the residual processing unit is used to perform residual processing on the weight feature map to obtain a residual feature map; the sampling unit is used to upsample the residual feature map, the first current resolution feature map and the second current resolution feature map to obtain a sampling feature map; the output unit is used to perform convolution and upsampling operations on the sampling feature map to obtain an output segmentation map.

[0076] In some embodiments, such as this embodiment, the updating unit 207 includes a detection unit, an input unit, and an updating sub-unit.

[0077] Among them, the detection unit is used to detect whether the frame count is an integer multiple of a preset frame number; the input unit is used to input the output segmentation map, the current frame image and the third current resolution feature map into the second encoding network for target tracking to obtain an updated feature map if the frame count is an integer multiple of the preset frame number; the updating subunit is used to perform convolution processing on the updated feature map to obtain a first updated convolution feature map and a second updated convolution feature map, and use the first updated convolution feature map and the second updated convolution feature map as the reference convolution feature map and the reference fusion feature map.

[0078] The above-mentioned video segmentation device based on fusion coding network can be implemented in the form of a computer program. Figure 6 Run on the terminal shown.

[0079] See also Figure 6 , Figure 6 300 is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device 300 is a display device with a video segmentation function based on a fusion coding network.

[0080] See also Figure 6 The computer device 300 includes a processor 302 , a memory and a network interface 305 connected via a system bus 301 , wherein the memory may include a non-volatile storage medium 303 and an internal memory 304 .

[0081] The non-volatile storage medium 303 may store an operating system 3031 and a computer program 3032. When the computer program 3032 is executed, the processor 302 may execute a video segmentation method based on a fusion coding network.

[0082] The processor 302 is used to provide computing and control capabilities to support the operation of the entire computer device 300 .

[0083] The internal memory 304 provides an environment for the operation of the computer program 3032 in the non-volatile storage medium 303. When the computer program 3032 is executed by the processor 302, the processor 302 can execute a video segmentation method based on a fusion coding network.

[0084] The network interface 305 is used to communicate with other devices over the network. Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 300 to which the solution of the present invention is applied. The specific computer device 300 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0085] The processor 302 is used to run a computer program 3032 stored in a memory to implement the following steps: obtaining a reference frame image specified by a user and a reference segmentation image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented; inputting the reference frame image into a fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map; inputting the reference resolution feature map and the reference segmentation image into the fusion coding network to obtain a reference coding feature map, and performing cascading and fusion processing on the reference coding feature map and the reference resolution feature map after overlapping and expansion to obtain a reference fusion feature map; obtaining the The current frame image in the video to be segmented is input into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map and a third current resolution feature map, and the third current resolution feature map is convolved to obtain a current convolution feature map; the current convolution feature map, the reference convolution feature map and the reference fusion feature map are subjected to attention weight calculation to obtain a weight feature map; the weight feature map, the first current resolution feature map and the second current resolution feature map are decoded to obtain an output segmentation map; if the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed.

[0086] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of inputting the current frame image into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map, the processor 302 specifically implements the following steps: inputting the current frame image into the first coding network in the fusion coding network to perform layer-by-layer feature extraction to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map.

[0087] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of inputting the reference resolution feature map and the reference segmentation image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlapping expansion to obtain a reference fused feature map, the processor 302 specifically implements the following steps: inputting the reference resolution feature map and the reference segmentation image into the second coding network in the fusion coding network to perform target tracking to obtain a reference coding feature map; overlappingly expanding the reference resolution, and cascading the reference resolution feature map after overlapping expansion with the reference coding feature map to obtain a reference cascade feature map; and performing residual fusion processing on the reference cascade feature map to obtain a reference fused feature map.

[0088] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of calculating the attention weights of the current convolution feature map, the reference convolution feature map, and the reference fusion feature map to obtain a weighted feature map, the processor 302 specifically implements the following steps: performing dimensionality reduction, multiplication, subtraction, and normalization operations on the first current convolution feature map and the reference convolution feature map in turn to obtain an attention feature map; performing attention weight calculations on the attention feature map, the second current convolution feature map, and the reference fusion feature map to obtain a weighted feature map.

[0089] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of calculating the attention weights of the attention feature map, the second current convolution feature map, and the reference fusion feature map to obtain a weighted feature map, the processor 302 specifically implements the following steps: traversing the second current convolution feature map to obtain a plurality of traversed feature maps; performing dimensionality reduction processing on each of the plurality of traversed feature maps to obtain a reduced dimensionality feature map; multiplying, increasing dimensionality, and cascading the reduced dimensionality feature map with the attention feature map to obtain a target attention feature map; and cascading the reference fusion feature map after overlapping and expansion with the target attention feature map to obtain a weighted feature map.

[0090] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of decoding the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map, the processor 302 specifically implements the following steps: performing residual processing on the weight feature map to obtain a residual feature map; performing upsampling operations on the residual feature map, the first current resolution feature map, and the second current resolution feature map to obtain a sampling feature map; and performing convolution and upsampling operations on the sampling feature map to obtain an output segmentation map.

[0091] In certain embodiments, such as the present embodiment, when the processor 302 implements the step of updating the reference convolution feature map and the reference fusion feature map if the feature map update condition is met until the segmentation of the video to be segmented is completed, the processor 302 specifically implements the following steps: detecting whether the frame count is an integer multiple of a preset number of frames; if the frame count is an integer multiple of the preset number of frames, inputting the output segmentation map, the current frame image, and the third current resolution feature map into the second encoding network for target tracking to obtain an updated feature map; performing convolution processing on the updated feature map to obtain a first updated convolution feature map and a second updated convolution feature map, and using the first updated convolution feature map and the second updated convolution feature map as the reference convolution feature map and the reference fusion feature map.

[0092] It should be understood that in the embodiment of the present invention, the processor 302 may be a central processing unit (CPU), and the processor 302 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0094] Therefore, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program. When the computer program is executed by a processor, the processor executes any embodiment of the above-mentioned video segmentation method based on a fusion coding network.

[0095] The storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk, etc., which are computer-readable storage media that can store program codes.

[0096] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0097] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0098] The steps in the method of the embodiment of the present invention can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present invention can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.

[0100] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0102] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A video segmentation method based on a fusion coding network, characterized in that: include: Acquire a reference frame image specified by a user and a reference segmented image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented; Inputting the reference frame image into a fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map; Inputting the reference resolution feature map and the reference segmented image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlap expansion to obtain a reference fusion feature map; Acquire a current frame image in the video to be segmented, input the current frame image into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map, and perform convolution processing on the third current resolution feature map to obtain a current convolution feature map; Performing attention weight calculation on the current convolution feature map, the reference convolution feature map, and the reference fusion feature map to obtain a weighted feature map; Decoding the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map; If the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed.

2. The method according to claim 1, characterized in that The step of inputting the current frame image into the fusion coding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map comprises: The current frame image is input into the first coding network in the fusion coding network to perform layer-by-layer feature extraction to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map.

3. The method according to claim 2, characterized in that The step of inputting the reference resolution feature map and the reference segmented image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlap expansion to obtain a reference fusion feature map comprises: Inputting the reference resolution feature map and the reference segmentation image into a second coding network in the fusion coding network to perform target tracking to obtain a reference coding feature map; Overlapping and extending the reference resolution feature map, and concatenating the overlapped and extended reference resolution feature map with the reference coding feature map to obtain a reference concatenated feature map; The reference cascade feature map is subjected to residual fusion processing to obtain a reference fused feature map.

4. The method according to claim 1, characterized in that The current convolution feature map includes a first current convolution feature map and a second current convolution feature map, and the step of performing attention weight calculation on the current convolution feature map, the reference convolution feature map, and the reference fusion feature map to obtain a weighted feature map includes: Performing dimension reduction, multiplication, subtraction, and normalization operations on the first current convolution feature map and the reference convolution feature map in sequence to obtain an attention feature map; Attention weights are calculated on the attention feature map, the second current convolution feature map, and the reference fusion feature map to obtain a weighted feature map.

5. The method according to claim 4, characterized in that The step of performing attention weight calculation on the attention feature map, the second current convolution feature map and the reference fusion feature map to obtain a weighted feature map comprises: Traversing the second current convolution feature map to obtain a plurality of traversal feature maps; Performing dimensionality reduction processing on each of the multiple traversal feature graphs to obtain a reduced dimensionality feature graph; The reduced-dimensional feature map and the attention feature map are multiplied, dimensionally upgraded, and cascaded to obtain a target attention feature map; The reference fusion feature map after overlapping expansion is cascaded with the target attention feature map to obtain a weighted feature map.

6. The method according to claim 1, characterized in that The step of decoding the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map comprises: Performing residual processing on the weight feature map to obtain a residual feature map; Performing an upsampling operation on the residual feature map, the first current resolution feature map, and the second current resolution feature map to obtain a sampled feature map; Convolution and upsampling operations are performed on the sampled feature map to obtain an output segmentation map.

7. The method according to claim 3, characterized in that If the feature map update condition is met, the reference convolution feature map and the reference fusion feature map are updated until the segmentation of the video to be segmented is completed, comprising: Check whether the frame count is an integer multiple of the preset frame count; If the frame count is an integer multiple of the preset frame number, inputting the output segmentation map, the current frame image and the third current resolution feature map into the second encoding network for target tracking to obtain an updated feature map; The updated feature map is convolved to obtain a first updated convolution feature map and a second updated convolution feature map, and the first updated convolution feature map and the second updated convolution feature map are used as the reference convolution feature map and the reference fusion feature map.

8. A video segmentation device based on a fusion coding network, characterized in that: include: An acquisition unit, used to acquire a reference frame image specified by a user and a reference segmented image corresponding to the reference frame image, wherein the reference frame image is any frame image in the video to be segmented; A first coding convolution unit, used for inputting the reference frame image into a fusion coding network to obtain a reference resolution feature map, and performing convolution processing on the reference resolution feature map to obtain a reference convolution feature map; A cascade fusion unit, used for inputting the reference resolution feature map and the reference segmented image into the fusion coding network to obtain a reference coding feature map, and cascading and fusing the reference coding feature map and the reference resolution feature map after overlap expansion to obtain a reference fusion feature map; A second encoding convolution unit is used to obtain a current frame image in the video to be segmented, and input the current frame image into the fusion encoding network to obtain a first current resolution feature map, a second current resolution feature map, and a third current resolution feature map, and perform convolution processing on the third current resolution feature map to obtain a current convolution feature map; An attention unit, used to calculate the attention weights of the current convolution feature map, the reference convolution feature map and the reference fusion feature map to obtain a weighted feature map; A decoding unit, configured to decode the weight feature map, the first current resolution feature map, and the second current resolution feature map to obtain an output segmentation map; An updating unit is used to update the reference convolution feature map and the reference fusion feature map if the feature map update condition is met until the segmentation of the video to be segmented is completed.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images