A panoramic segmentation method and device combining frequency domain attention and multi-scale fusion

Abstract

The present invention relates to a panoramic segmentation method and device combining frequency domain attention and multi-scale fusion, belonging to the field of computer vision. The method comprises: extracting features of the received original image through the backbone network, and outputting pyramid features with different scales; the pyramid features include pyramid features of the minimum scale and pyramid features of other scales; performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain semantic segmentation results; performing instance segmentation on the pyramid features of the minimum scale in the pyramid features through the instance segmentation head to obtain instance segmentation results; and fusing the semantic segmentation results and the instance segmentation results through the panoramic fusion head to obtain a panoramic segmentation result. Through this method, the segmentation effect of panoramic segmentation can be improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision, and specifically relates to a panoramic segmentation method and device combining frequency domain attention and multi-scale fusion. Background Art

[0002] In the field of computer vision, panoramic segmentation is a novel and important research topic. It is the cornerstone of emerging cutting-edge technologies such as scene perception, autonomous driving, and human-computer interaction, and has very important research significance.

[0003] Panoramic segmentation is a unification of semantic segmentation and instance segmentation. Its task requires not only distinguishing the background semantic categories and foreground semantic categories in the input image, but also segmenting different instances in the same category of foreground semantics. Therefore, the panoramic segmentation task is more difficult than simple semantic segmentation or instance segmentation tasks.

[0004] In panoramic segmentation, for countable targets called things (pedestrians, vehicles), not only do we need to segment each pixel where the target is located, but for uncountable areas called stuff (such as trees, sky, buildings), we only need to distinguish their categories.

[0005] Panoptic segmentation tasks are currently mainly implemented using deep learning networks, such as Figure 1 A typical top-down panoramic segmentation network, namely UPSNet, is shown. The network structure consists of four parts: Backbone, Semantic Head, Instance Head and Panoptic Head.

[0006] However, when the original image is processed based on the panoramic segmentation network, the segmentation effect obtained is poor. Summary of the invention

[0007] In view of this, the purpose of the present application is to provide a panoramic segmentation method and device combining frequency domain attention and multi-scale fusion to improve the segmentation effect of panoramic segmentation.

[0008] The embodiment of the present application is implemented as follows:

[0009] In a first aspect, an embodiment of the present application provides a panoramic segmentation method combining frequency domain attention and multi-scale fusion, which is applied to a panoramic segmentation network, wherein the panoramic segmentation network includes a backbone network, a semantic segmentation head, an instance segmentation head and a panoramic fusion head; a frequency domain attention mechanism module is introduced into the backbone network, and a cascaded multi-scale fusion module is introduced into the semantic segmentation head. The method includes: extracting features of the received original image through the backbone network, and outputting pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales; performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain a semantic segmentation result; performing instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain an instance segmentation result; and fusing the semantic segmentation result and the instance segmentation result through the panoramic fusion head to obtain a panoramic segmentation result.

[0010] In combination with the embodiment of the first aspect, in a possible implementation, the backbone network includes a residual network and a feature pyramid network, and the frequency domain attention mechanism module is added at the end of each residual block included in the residual network. The frequency domain attention mechanism module enhances the feature extraction performance of the residual network by learning feature information of multiple frequency domain channels.

[0011] In combination with the first aspect embodiment, in a possible implementation, the frequency domain attention mechanism module is used to divide the acquired input feature X into n partial features along the channel dimension, obtain the corresponding frequency domain information from each partial feature, and after the corresponding frequency domain information is applied to the corresponding partial feature, the frequency domain tensors of each partial feature are obtained, the frequency domain tensors of each partial feature are connected as the final attention tensor, and the final attention tensor is applied to the input feature X to obtain the feature tensor Y containing multiple frequency domain information.

[0012] In combination with the first aspect embodiment, in a possible implementation manner, the frequency domain attention mechanism module is based on the formula Y=σ(cat([Freq 0 ,Freq 1 ,L,Freq n-1 ]))*X, and

[0013] The feature tensor Y is calculated; where X is the input feature, Y is the feature tensor, σ is the sigmoid activation function, cat is the connection operation, * is the tensor multiplication, Freq i is the frequency domain tensor corresponding to different partial features, h and w are the height and width of the local image corresponding to the input feature, H and W are the height and width of the original image,

[0014] In combination with the embodiment of the first aspect, in a possible implementation, a cascaded multi-scale fusion module is introduced into the semantic segmentation head, that is, an inter-layer cascade structure is introduced into the original multi-scale fusion module, so that the semantic segmentation head learns the optimal combination of features at each scale, thereby improving the performance of semantic segmentation.

[0015] In combination with the embodiment of the first aspect, in a possible implementation manner, the cascaded multi-scale fusion module obtains a set of new scale feature tensors with richer information by learning the features of the scales of adjacent layers between layers; specifically: F2 downsampling and F3 fusion obtain a new feature tensor F3' at the scale, F3' downsampling and F4 fusion obtain a new feature tensor F4' at the scale, F4' downsampling and F5 fusion obtain a new feature tensor F5' at the scale, F2 remains unchanged, recorded as F2'.

[0016] In combination with the embodiment of the first aspect, in a possible implementation manner, the inter-layer cascade structure acts on the original feature tensors of adjacent scales respectively through the structure to obtain new feature tensors at four scales, namely F2', F3', F4', and F5', wherein F3', F4', and F5' are calculated according to the following formula:

[0017] X′ (r＝1) =S (r＝0.5) *α (r＝1) +X (r＝1) *(1-α (r＝1) );

[0018] Where X and S are adjacent scale features, the scale of S is twice that of X, r is the compression factor, r = 1 means no scaling, r = 0.5 means 2x downsampling, α is the obtained attention tensor, and X' is the new feature tensor at this scale obtained after inter-layer fusion;

[0019] The formula of attention tensor α is: α = σ([MaxPool(x), AvgPool(x)]);

[0020] Where σ is the sigmoid activation function, MaxPool and AvgPool are maximum pooling and average pooling respectively, and x is the feature tensor of the original scale feature after 3x3 convolution.

[0021] In the second aspect, an embodiment of the present application provides a panoramic segmentation device combining frequency domain attention and multi-scale fusion, which is applied to a panoramic segmentation network, wherein the panoramic segmentation network includes a backbone network, a semantic segmentation head, an instance segmentation head and a panoramic fusion head; a frequency domain attention mechanism module is introduced into the backbone network, and a multi-scale fusion module is introduced into the semantic segmentation head. The device includes: a feature extraction module, a semantic segmentation module, an instance segmentation module and a fusion module.

[0022] A feature extraction module, used to extract features from the received original image through the backbone network, and output pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales;

[0023] A semantic segmentation module, used for performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain a semantic segmentation result;

[0024] An instance segmentation module, used for performing instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain an instance segmentation result;

[0025] The fusion module is used to fuse the semantic segmentation result and the instance segmentation result through the panoramic fusion head to obtain a panoramic segmentation result.

[0026] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a memory and a processor, wherein the memory and the processor are connected; the memory is used to store programs; and the processor calls the program stored in the memory to execute the method provided in the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the first aspect embodiment.

[0027] In a fourth aspect, an embodiment of the present application further provides a non-volatile computer-readable storage medium (hereinafter referred to as storage medium) on which a computer program is stored, and when the computer program is run by a computer, the method provided by the above-mentioned first aspect embodiment and / or any possible implementation method in combination with the first aspect embodiment is executed.

[0028] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by implementing the embodiments of the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings without creative work. As shown in the drawings, the above and other purposes, features and advantages of the present application will be clearer. The same reference numerals indicate the same parts in all the drawings. The drawings are not deliberately scaled to the actual size, and the focus is on showing the main purpose of the present application.

[0030] Figure 1 A schematic diagram of an existing panoptic segmentation network is shown.

[0031] Figure 2 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown.

[0032] Figure 3 A flowchart of a panoramic segmentation method combining frequency domain attention and multi-scale fusion provided in an embodiment of the present application is shown.

[0033] Figure 4 A schematic diagram of a panoptic segmentation network provided in an embodiment of the present application is shown.

[0034] Figure 5 A schematic diagram of a frequency domain attention mechanism module provided in an embodiment of the present application is shown.

[0035] Figure 6 A schematic diagram of an equally weighted multi-scale fusion module provided in an embodiment of the present application is shown.

[0036] Figure 7 A schematic diagram of a multi-scale fusion module provided in an embodiment of the present application is shown.

[0037] Figure 8 A schematic diagram of a hierarchical attention mechanism module provided in an embodiment of the present application is shown.

[0038] Fig. 9 A structural block diagram of a panoramic segmentation device combining frequency domain attention and multi-scale fusion provided in an embodiment of the present application is shown.

[0039] Figure numerals: 100 - electronic device; 110 - processor; 120 - memory; 400 - panoramic segmentation device combining frequency domain attention and multi-scale fusion; 410 - feature extraction module; 420 - semantic segmentation module; 430 - instance segmentation module; 440 - fusion module. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0041] It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in the subsequent figures. At the same time, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the statement "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.

[0042] Furthermore, the term "and / or" in this application is merely a term used to describe the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0043] In addition, the defects existing in the panoramic segmentation scheme in the prior art are the results obtained by the applicant after practice and careful research. Therefore, the process of discovering the above defects and the solutions proposed in the embodiments of this application for the above defects below should be the contributions made by the applicant to this application during the application process.

[0044] In order to solve the above problems, the embodiments of the present application provide a panoramic segmentation method and device combining frequency domain attention and multi-scale fusion to improve the segmentation effect of panoramic segmentation.

[0045] The technology can be implemented by using corresponding software, hardware or a combination of software and hardware. The following is a detailed introduction to the embodiments of the present application.

[0046] First, refer to Figure 2 To describe an electronic device 100 for implementing the panoramic segmentation method and device combining frequency domain attention and multi-scale fusion in an embodiment of the present application.

[0047] The electronic device 100 can provide services for the user (vocabulary learner / browser), for example, displaying the panoramic segmented image to the user.

[0048] Optionally, the electronic device 100 may be, but not limited to, a personal computer (PC), a smart phone, a tablet computer, a mobile Internet device (MID), a personal digital assistant, a server, etc. The server may be, but not limited to, a network server, a database server, a cloud server, etc.

[0049] The electronic device 100 may include: a processor 110 and a memory 120 .

[0050] It should be noted that Figure 2 The components and structures of the electronic device 100 shown are only exemplary and non-limiting. The electronic device 100 may also have other components and structures as required. For example, in some cases, the electronic device 100 may also include a display.

[0051] The processor 110, the memory 120, and other components that may appear in the electronic device 100 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, the processor 110, the memory 120, and other components that may appear may be electrically connected to each other via one or more communication buses or signal lines.

[0052] The memory 120 is used to store programs, for example, a program corresponding to a panoramic segmentation method combining frequency domain attention and multi-scale fusion described later or a panoramic segmentation device combining frequency domain attention and multi-scale fusion described later. Optionally, when a panoramic segmentation device combining frequency domain attention and multi-scale fusion is stored in the memory 120, the panoramic segmentation device combining frequency domain attention and multi-scale fusion includes at least one software function module that can be stored in the memory 120 in the form of software or firmware.

[0053] Optionally, the software function modules included in the panoramic segmentation device combining frequency domain attention and multi-scale fusion can also be solidified in the operating system (OS) of the electronic device 100.

[0054] The processor 110 is used to execute executable modules stored in the memory 120, such as software function modules or computer programs included in the panoramic segmentation device combining frequency domain attention and multi-scale fusion. After receiving the execution instruction, the processor 110 can execute the computer program, for example, executing: extracting features of the received original image through the backbone network, and outputting pyramid features with different scales; the pyramid features include pyramid features of the minimum scale and pyramid features of other scales; performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain semantic segmentation results; performing instance segmentation on the pyramid features of the minimum scale in the pyramid features through the instance segmentation head to obtain instance segmentation results; and fusing the semantic segmentation results and the instance segmentation results through the panoramic fusion head to obtain a panoramic segmentation result.

[0055] Of course, the method disclosed in any embodiment of the present application can be applied to the processor 110 or implemented by the processor 110.

[0056] The following will introduce the panoramic segmentation method combining frequency domain attention and multi-scale fusion provided in this application.

[0057] See also Figure 3 , the embodiment of the present application provides a panoramic segmentation method combining frequency domain attention and multi-scale fusion applied to the above electronic device 100. A panoramic segmentation network is pre-stored in the electronic device, and the schematic diagram of the panoramic segmentation network is as shown in Figure 4 As shown, it can include a backbone network Backbone, a semantic segmentation head Semantic Head, an instance segmentation head Instance Head and a panoramic fusion head Panoptic Head.

[0058] Among them, a frequency domain attention mechanism module AttentionMechanism is introduced into the backbone network Backbone, and a cascaded multi-scale fusion module MultiscaleFusion is introduced into the semantic segmentation head Semantic Head.

[0059] The following will be combined Figure 3 Describe the steps involved.

[0060] Step S110: extracting features from the received original image through the backbone network, and outputting pyramid features with different scales.

[0061] Among them, the backbone network Backbone includes a residual network (ResNet-50 or ResNet-101) and a feature pyramid network (FPN), which is responsible for extracting features from the original image and then inputting the extracted features into the semantic segmentation head and the instance segmentation head for processing.

[0062] Specifically, each feature extraction block included in the feature pyramid network outputs a pyramid feature of a scale, so that the feature pyramid network outputs multiple pyramid features with different scales, for example, outputs pyramid features of scales from 1 / 4 to 1 / 64 (ie, F2 to F6).

[0063] For these pyramid features, the pyramid feature with the smallest scale, i.e., F6, is used as the input of the instance segmentation head InstanceHead, and the pyramid features with other scales, i.e., F2 to F5, are used as the input of the semantic segmentation head Semantic Head.

[0064] In the prior art, no attention mechanism is used in the backbone network.

[0065] In an embodiment of the present application, a frequency domain attention mechanism module Attention Mechanism is introduced in the backbone network Backbone. The attention mechanism focuses on certain specific foreground information in the image and ignores the rest of the information, which is conducive to improving the perceived information while retaining rich contextual information. In an embodiment of the present application, the feature extraction performance of the backbone network Backbone is improved by learning feature information in other frequency domains through the frequency domain attention mechanism module Attention Mechanism, thereby improving the overall performance of the panoramic segmentation network.

[0066] Specifically, as mentioned above, the residual network ResNet included in the backbone network Backbone includes multiple residual blocks. In an embodiment of the present application, a frequency domain attention mechanism module Attention Mechanism is added at the end of each residual block included in the residual network ResNet to enhance the feature extraction performance of the residual network by learning more feature information of the frequency domain channels.

[0067] The frequency domain attention mechanism is introduced below.

[0068] First, introduce the two-dimensional discrete cosine transform formula:

[0069]

[0070] sth∈{0,1,L,H-1},w∈{0,1,L,W-1}, (1)

[0071] where f 2d is the frequency domain of the two-dimensional space, h and w are the corresponding height and width respectively.

[0072] According to formula (1), the inverse operation of the two-dimensional discrete cosine transform formula can be obtained:

[0073]

[0074] sth∈{0,1,L,H-1},w∈{0,1,L,W-1}, (2)

[0075] make:

[0076]

[0077] Then formula (2) can be simplified as:

[0078]

[0079] sth∈{0,1,L,H-1},w∈{0,1,L,W-1}, (4)

[0080] It can be seen from formula (4) that image features can be expressed as a combination of different frequency components; in this way, more frequency domain information can be introduced to solve the problem of insufficient channel attention feature information.

[0081] Based on the above description, a frequency domain attention mechanism module can be set to divide the acquired input feature X into n partial features along the channel dimension, obtain the corresponding frequency domain information from each partial feature, and after the corresponding frequency domain information is applied to the corresponding partial feature, the frequency domain tensor of each partial feature is obtained, the frequency domain tensors of each partial feature are connected as the final attention tensor, and the final attention tensor is applied to the input feature X to obtain the feature tensor Y containing multiple frequency domain information.

[0082] Specifically, the frequency domain attention mechanism module can be based on the formula

[0083] Y=σ(cat([Freq 0 ,Freq 1 ,L,Freq n-1 ]))*X, and

[0084] Calculate and obtain the feature tensor Y;

[0085] Where X is the input feature, Y is the feature tensor, σ is the sigmoid activation function, cat is the concatenation operation, * is the tensor multiplication, Freq i is the frequency domain tensor corresponding to different partial features, h and w are the height and width of the local image corresponding to the input feature, H and W are the height and width of the original image,

[0086] In order to further reduce the amount of calculation, through multiple experiments, the embodiment of the present application selects 16 frequency domain channels with the richest feature information and adds them to the frequency domain attention mechanism module.

[0087] Among them, the schematic diagram of the frequency domain attention mechanism module can be referred to Figure 5 .

[0088] Step S120: performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain a semantic segmentation result.

[0089] The semantic segmentation head follows the FCN network. For the feature tensors fpn_p2, fpn_p3, fpn_p4, and fpn_p5 from FPN, they are first independently subjected to deformable convolution, and then the branches are connected by upsampling, the features of each scale are cascaded, and finally softmax is used for 1×1 convolution to predict the semantic class.

[0090] In the prior art, the multi-scale method used in the semantic segmentation head is to equally weight the features of each scale, then perform upsampling connections, and finally generate the final feature tensor through 1×1 convolution for prediction, such as Figure 6 However, since each scale is weighted equally, the importance of feature information at different scales is ignored, resulting in suboptimal results.

[0091] In an embodiment of the present application, a cascaded multi-scale fusion module Multiscale Fusion is introduced into the semantic segmentation head, so that the semantic segmentation head network can learn a better combination of features of each scale during training, thereby improving the performance of the panoramic segmentation network.

[0092] Specifically, a cascaded multi-scale fusion module Multiscale Fusion is introduced in the semantic segmentation head. By adaptively learning the importance of features at different scales, it can better combine features at different levels and thus enhance the performance of panoramic segmentation.

[0093] The multiscale fusion module Multiscale Fusion proposed in the embodiment of this application is shown in Figure 7As shown. This module learns the features of adjacent layer scales through training, and obtains a set of scale feature tensors (F2', F3', F4', F5') with richer information. The process is: F2 is downsampled and fused with F3 to obtain a new feature tensor F3' at this scale, F3' is downsampled and fused with F4 to obtain a new feature tensor F4' at this scale, F4' is downsampled and fused with F5 to obtain a new feature tensor F5' at this scale, and F2 remains unchanged, recorded as F2'.

[0094] During network training, the inter-layer fusion structure can learn feature information of adjacent scales. Figure 8 As shown, Figure 8 The Conv in refers to the 3×3 standard convolutional layer and the batch normalization layer.

[0095] The new feature tensor obtained by the inter-layer fusion structure is as follows:

[0096] X′ (r＝1) =S (r＝0.5) *α (r＝1) +X (r＝1) *(1-α (r＝1) )

[0097] Where X and S are adjacent scale features, the scale of S is twice that of X, r is the compression factor, r = 1 means no scaling, r = 0.5 means 2x downsampling. α is the obtained attention tensor, and X′ is the new feature tensor obtained after the attention operation. The attention tensor α is shown below.

[0098] α=σ([MaxPool(x),AvgPool(x)]).

[0099] Where σ is the sigmoid activation function, MaxPool and AvgPool are maximum pooling and average pooling respectively, and x is the feature tensor of the original scale feature after 3x3 convolution.

[0100] Step S130: performing instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain an instance segmentation result; and fusing the semantic segmentation result and the instance segmentation result through the panoramic fusion head to obtain a panoramic segmentation result.

[0101] The instance segmentation head follows the Mask R-CNN network, and the output includes bounding box regression, classification, and segmentation mask. Since this part is a relatively mature existing technology, it will not be repeated here.

[0102] Step S140: The semantic segmentation result and the instance segmentation result are fused through the panoramic fusion head to obtain a panoramic segmentation result.

[0103] The Panoptic Head takes the logical mask of each pixel before each branch classification layer as input, and the logical mask from the semantic segmentation head is expressed as Where N stuff ,N things Represents the number of stuff and things classes respectively, H and W are the height and width of the image; for the logical mask from the instance segmentation head, they all have a mask Y of the instance segmentation head of size 28×28; the panoramic fusion head Panoptic Head fuses X and Y to generate a panoramic segmentation result that accurately segments the stuff and things class information.

[0104] Since this part is a relatively mature existing technology, it will not be described here in detail.

[0105] A panoramic segmentation method combining frequency domain attention and multi-scale fusion provided in an embodiment of the present application is applied to a panoramic segmentation network, wherein the panoramic segmentation network includes a backbone network, a semantic segmentation head, an instance segmentation head and a panoramic fusion head; a frequency domain attention mechanism module is introduced into the backbone network, and a cascaded multi-scale fusion module is introduced into the semantic segmentation head. The method includes: extracting features of a received original image through the backbone network, and outputting pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales; performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain a semantic segmentation result; performing instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain an instance segmentation result; and fusing the semantic segmentation result and the instance segmentation result through the panoramic fusion head to obtain a panoramic segmentation result. Since the frequency domain attention mechanism module is introduced in the backbone network, the feature extraction performance of the network backbone can be improved by learning feature information in other frequency domains, thereby improving the performance of the panoramic segmentation network. Since the cascaded multi-scale fusion module is introduced in the semantic segmentation head, the semantic segmentation head can learn a better combination of features at each scale during training to improve the performance of the panoramic segmentation network. Therefore, after performing panoramic segmentation based on the panoramic segmentation network, the panoramic segmentation effect can be improved.

[0106] like Fig. 9As shown, an embodiment of the present application also provides a panoramic segmentation device 400 combining frequency domain attention and multi-scale fusion, which is applied to a panoramic segmentation network, wherein the panoramic segmentation network includes a backbone network, a semantic segmentation head, an instance segmentation head and a panoramic fusion head; a frequency domain attention mechanism module is introduced into the backbone network, and a cascaded multi-scale fusion module is introduced into the semantic segmentation head. The panoramic segmentation device 400 combining frequency domain attention and multi-scale fusion may include: a feature extraction module 410, a semantic segmentation module 420, an instance segmentation module 430 and a fusion module 440.

[0107] The feature extraction module 410 is used to extract features from the received original image through the backbone network, and output pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales;

[0108] A semantic segmentation module 420, configured to perform semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain a semantic segmentation result;

[0109] An instance segmentation module 430 is used to perform instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain an instance segmentation result;

[0110] The fusion module 440 is used to fuse the semantic segmentation result and the instance segmentation result through the panoramic fusion head to obtain a panoramic segmentation result.

[0111] In one possible implementation, the backbone network includes a residual network and a feature pyramid network. The frequency domain attention mechanism module is added at the end of each residual block included in the residual network. The frequency domain attention mechanism module enhances the feature extraction performance of the residual network by learning feature information of multiple frequency domain channels.

[0112] In a possible implementation, the frequency domain attention mechanism module is used to divide the acquired input feature X into n partial features along the channel dimension, obtain the corresponding frequency domain information from each partial feature, and after applying the corresponding frequency domain information to the corresponding partial feature, obtain the frequency domain tensor of each partial feature, connect the frequency domain tensors of each partial feature as a final attention tensor, and apply the final attention tensor to the input feature X to obtain the feature tensor Y containing multiple frequency domain information.

[0113] In one possible implementation, the frequency domain attention mechanism module is based on the formula

[0114] Y=σ(cat([Freq 0 ,Freq 1,L,Freq n-1 ]))*X, and

[0115] Calculate and obtain the feature tensor Y;

[0116] Where X is the input feature, Y is the feature tensor, σ is the sigmoid activation function, cat is the concatenation operation, * is the tensor multiplication, Freq i is the frequency domain tensor corresponding to different partial features, h and w are the height and width of the local image corresponding to the input feature, H and W are the height and width of the original image,

[0117] In a possible implementation, a cascaded multi-scale fusion module is introduced into the semantic segmentation head, that is, an inter-layer cascade structure is introduced into the original multi-scale fusion module, so that the semantic segmentation head learns the best combination of features at each scale, thereby improving the performance of semantic segmentation.

[0118] In a possible implementation, the cascaded multi-scale fusion module obtains a set of new scale feature tensors with richer information by learning the features of the scales of adjacent layers between layers. Specifically, F2 is downsampled and fused with F3 to obtain a new feature tensor F3' at the scale, F3' is downsampled and fused with F4 to obtain a new feature tensor F4' at the scale, F4' is downsampled and fused with F5 to obtain a new feature tensor F5' at the scale, and F2 remains unchanged, recorded as F2'.

[0119] In a possible implementation manner, the inter-layer cascade structure finally outputs new feature tensors at four scales, namely F2', F3', F4', and F5', wherein F3', F4', and F5' are calculated according to the following formulas.

[0120] X′ (r＝1) =S (r＝0.5) *α (r＝1) +X (r＝1) *(1-α (r＝1) )

[0121] Where X and S are adjacent scale features, the scale of S is twice that of X, r is the compression factor, r = 1 means no scaling, r = 0.5 means 2x downsampling. α is the obtained attention tensor, and X' is the new feature tensor at this scale obtained after inter-layer fusion. The attention tensor α is shown in the following formula.

[0122] α=σ([MaxPool(x),AvgPool(x)])

[0123] Where σ is the sigmoid activation function, MaxPool and AvgPool are maximum pooling and average pooling respectively, and x is the feature tensor of the original scale feature after 3x3 convolution.

[0124] The panoramic segmentation device 400 combining frequency domain attention and multi-scale fusion provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0125] In addition, an embodiment of the present application also provides a storage medium, on which a computer program is stored. When the computer program is run by a computer, the steps included in the panoramic segmentation method combining frequency domain attention and multi-scale fusion as described above are executed.

[0126] In summary, an embodiment of the present invention proposes a panoramic segmentation method and device combining frequency domain attention and multi-scale fusion, which are applied to a panoramic segmentation network. The panoramic segmentation network includes a backbone network, a semantic segmentation head, an instance segmentation head and a panoramic fusion head; a frequency domain attention mechanism module is introduced into the backbone network, and a cascaded multi-scale fusion module is introduced into the semantic segmentation head. The method includes: extracting features of the received original image through the backbone network, and outputting pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales; semantically predicting the pyramid features of other scales through the semantic segmentation head to obtain semantic segmentation results; performing instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain instance segmentation results; fusing the semantic segmentation results and the instance segmentation results through the panoramic fusion head to obtain a panoramic segmentation result. Since the frequency domain attention mechanism module is introduced in the backbone network, the feature extraction performance of the network backbone can be improved by learning feature information in other frequency domains, thereby improving the performance of the panoramic segmentation network. Since the cascaded multi-scale fusion module is introduced in the semantic segmentation head, the semantic segmentation head can learn a better combination of features at each scale during training to improve the performance of the panoramic segmentation network. Therefore, after performing panoramic segmentation based on the panoramic segmentation network, the panoramic segmentation effect can be improved.

[0127] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0128] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0130] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a laptop, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0131] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to a panoptic segmentation network, the panoptic segmentation network comprising a backbone network, a semantic segmentation head, an instance segmentation head, and a panoptic fusion head; A frequency domain attention mechanism module is introduced into the backbone network, and a cascaded multi-scale fusion module is introduced into the semantic segmentation head. The method includes: Performing feature extraction on the received original image through the backbone network, and outputting pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales; Performing semantic prediction on the pyramid features of other scales by the semantic segmentation head to obtain a semantic segmentation result; Performing instance segmentation on the pyramid features of the smallest scale in the pyramid features by using the instance segmentation head to obtain an instance segmentation result; The semantic segmentation result and the instance segmentation result are fused by the panoramic fusion head to obtain a panoramic segmentation result; The backbone network includes a residual network and a feature pyramid network, and the frequency domain attention mechanism module is added at the end of each residual block included in the residual network, and the frequency domain attention mechanism module enhances the feature extraction performance of the residual network by learning feature information of multiple frequency domain channels; The frequency domain attention mechanism module is used to divide the acquired input feature X into n partial features along the channel dimension, obtain the corresponding frequency domain information from each partial feature, and after the corresponding frequency domain information is applied to the corresponding partial feature, obtain the frequency domain tensor of each partial feature, connect the frequency domain tensors of each partial feature as the final attention tensor, and apply the final attention tensor to the input feature X to obtain the feature tensor Y containing multiple frequency domain information; The frequency domain attention mechanism module is based on the formula Y = σ(cat([Freq 0 , Freq 1 ,L,Freq n-1 ]))*X and Calculate and obtain the feature tensor Y; Wherein, X is the input feature, Y is the feature tensor, σ is the sigmoid activation function, cat is the connection operation, * is the tensor multiplication, Freqi is the frequency domain tensor corresponding to different partial features, h and w are the height and width of the local image corresponding to the input feature, respectively, H and W are the height and width of the original image, respectively, The cascaded multi-scale fusion module introduces an inter-layer cascade structure in the multi-scale fusion module, so that the semantic segmentation head learns the best combination of features at each scale, thereby improving the performance of the semantic segmentation head; The cascaded multi-scale fusion module obtains a set of new scale feature tensors by learning the features of the scales of adjacent layers between layers; specifically: F2 downsampling and F3 fusion obtains a new feature tensor F3' at the scale, F3' downsampling and F4 fusion obtains a new feature tensor F4' at the scale, F4' downsampling and F5 fusion obtains a new feature tensor F5' at the scale, and F2 remains unchanged, recorded as F2'; The inter-layer cascade structure obtains new feature tensors at four scales, namely F2', F3', F4', and F5', wherein F3', F4', and F5' are calculated according to the following formula: X' (r＝1) =S (r＝0.5) *α (r＝1) +X (r＝1) *(1-α (r＝1) ); Where X and S are adjacent scale features, the scale of S is twice that of X, r is the compression factor, r = 1 means no scaling, r = 0.5 means 2x downsampling, α is the obtained attention tensor, and X' is the new feature tensor at this scale obtained after inter-layer fusion; The formula of attention tensor α is: α = σ([MaxPool(x), AvgPool(x)]); Where σ is the sigmoid activation function, MaxPool and AvgPool are maximum pooling and average pooling respectively, and x is the feature tensor of the original scale feature after 3x3 convolution.

2. An image processing device, characterized in that: Applied to a panoptic segmentation network, the panoptic segmentation network comprising a backbone network, a semantic segmentation head, an instance segmentation head, and a panoptic fusion head; A frequency domain attention mechanism module is introduced into the backbone network, and a cascaded multi-scale fusion module is introduced into the semantic segmentation head. The device includes: A feature extraction module, used to extract features from the received original image through the backbone network, and output pyramid features with different scales; the pyramid features include pyramid features of the smallest scale and pyramid features of other scales; A semantic segmentation module, used for performing semantic prediction on the pyramid features of other scales through the semantic segmentation head to obtain a semantic segmentation result; An instance segmentation module, used for performing instance segmentation on the pyramid features of the smallest scale in the pyramid features through the instance segmentation head to obtain an instance segmentation result; A fusion module, used for fusing the semantic segmentation result and the instance segmentation result through the panoramic fusion head to obtain a panoramic segmentation result; The backbone network includes a residual network and a feature pyramid network, and the frequency domain attention mechanism module is added at the end of each residual block included in the residual network, and the frequency domain attention mechanism module enhances the feature extraction performance of the residual network by learning feature information of multiple frequency domain channels; The frequency domain attention mechanism module is used to divide the acquired input feature X into n partial features along the channel dimension, obtain the corresponding frequency domain information from each partial feature, and after the corresponding frequency domain information is applied to the corresponding partial feature, obtain the frequency domain tensor of each partial feature, connect the frequency domain tensors of each partial feature as the final attention tensor, and apply the final attention tensor to the input feature X to obtain the feature tensor Y containing multiple frequency domain information; The frequency domain attention mechanism module is based on the formula Y = σ(cat([Freq 0 , Freq 1 ,L,Freq n-1 ]))*X and Calculate and obtain the feature tensor Y; Wherein, X is the input feature, Y is the feature tensor, σ is the sigmoid activation function, cat is the connection operation, * is the tensor multiplication, Freqi is the frequency domain tensor corresponding to different partial features, h and w are the height and width of the local image corresponding to the input feature, respectively, H and W are the height and width of the original image, respectively, The cascaded multi-scale fusion module introduces an inter-layer cascade structure in the multi-scale fusion module, so that the semantic segmentation head learns the best combination of features at each scale, thereby improving the performance of the semantic segmentation head; The cascaded multi-scale fusion module obtains a set of new scale feature tensors by learning the features of the scales of adjacent layers between layers; specifically: F2 downsampling and F3 fusion obtains a new feature tensor F3' at the scale, F3' downsampling and F4 fusion obtains a new feature tensor F4' at the scale, F4' downsampling and F5 fusion obtains a new feature tensor F5' at the scale, and F2 remains unchanged, recorded as F2'; The inter-layer cascade structure obtains new feature tensors at four scales, namely F2', F3', F4', and F5', wherein F3', F4', and F5' are calculated according to the following formula: X' (r＝1) =S (r＝0.5) *α (r＝1) +X (r＝1) *(1-α (r＝1) ); Where X and S are adjacent scale features, the scale of S is twice that of X, r is the compression factor, r = 1 means no scaling, r = 0.5 means 2x downsampling, α is the obtained attention tensor, and X' is the new feature tensor at this scale obtained after inter-layer fusion; The formula of attention tensor α is: α = σ([MaxPool(x), AvgPool(x)]); Where σ is the sigmoid activation function, MaxPool and AvgPool are maximum pooling and average pooling respectively, and x is the feature tensor of the original scale feature after 3x3 convolution.

3. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are connected; The memory is used to store programs; The processor calls the program stored in the memory to execute the method as claimed in claim 1.

4. A storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the method as claimed in claim 1 is executed.

Images