Feature fusion network interactive image segmentation method, electronic device and storage medium

Abstract

This application discloses a feature fusion network interactive image segmentation method, electronic device, and storage medium. This method addresses the problem of poor fusion feature specificity caused by insufficient feature interaction in existing IIS tasks. It proposes a click and image fusion framework based on CNNs and an attention mechanism, which can generate visual feature maps with stronger feature specificity while making clicks and images interact more fully. To address the pixel imbalance problem that is prevalent in IIS tasks, this problem is defined and a loss called balanced normalized focal loss is proposed based on existing loss theory and the actual needs of IIS tasks. By adding a balancing weight, the gradient centroid area ratio of positive click pixels and negative click pixels is effectively controlled to adjust the pixel imbalance problem during model training.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an attention-based feature fusion network interactive image segmentation method, electronic device and storage medium. Background Art

[0002] Interactive image segmentation (IIS), as an important branch of image processing, has been widely studied and applied in medical and other industrial fields in recent years due to its sufficient interactivity and practical value. At the same time, it has also shown potential applications in areas such as dataset production and visual privacy protection.

[0003] Interactive image segmentation involves the user providing a few interactive operations, such as clicks or line strokes, to locate and segment specific objects in an image. Existing interactive image segmentation methods are mainly divided into traditional methods and deep learning methods. Traditional methods classify pixels into "target," "background," and target edge regions based on user labels, providing hard and soft constraints for the model, respectively, to segment the target image. However, this segmentation approach can only be applied when the foreground and background of an image are clearly distinct. When it comes to objects with similar differences, it requires a large number of interactive operations and the segmentation effect is poor. Based on deep learning methods, researchers have extensively explored feature fusion and loss functions. In terms of feature fusion, existing research uses direct fusion methods such as adding or concatentating click and image features. In terms of loss functions, some studies have explored backpropagation rules in interactive segmentation tasks, introducing loss functions used in other fields, such as Soft_Iou, BCE, FL, and NFL, to interactive segmentation tasks. These deep learning methods have achieved better target segmentation results without changing user input. However, in terms of feature fusion, this direct fusion method has the problem of insufficient interaction, which may lead to low accuracy of model fusion results and insensitivity of the model to click features; in terms of loss function, existing methods all introduce loss functions that have been effective in other fields, but they cannot solve the pixel imbalance problem in the interactive segmentation field (the pixel imbalance problem consists of two parts: pixel difficulty imbalance and pixel gradient imbalance. Pixel difficulty imbalance refers to a large number of easy-to-classify samples distributed far away from the decision boundary, while a small number of difficult-to-classify samples are distributed between the foreground and the background; pixel gradient imbalance refers to the distribution imbalance between the target area to be segmented and other pixels in the image). Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide an attention-based feature fusion network interactive image segmentation method and device to solve the problem of low efficiency in daily affairs processing due to the need to prepare to-do items in advance.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides an attention-based feature fusion network interactive image segmentation method, comprising the following steps:

[0007] Step S1: Data preparation, providing image data, the actual segmentation mask of the original image, and click data, converting the image data into RGB form, and encoding the click data into spatial form in the form of coordinates;

[0008] Step S2: Data fusion: input the image data and click data into the click-aware pixel attention fusion module in the model to obtain fusion features;

[0009] Step S3: Mask segmentation: Input the obtained fusion features into the segmentation network, first use the HRNet backbone network in the segmentation network for deep encoding, then use the OCR network and linear processing layer in the segmentation network to decode the deep-encoded fusion features, and finally the segmentation network outputs the final mask of the target;

[0010] Step S4: Model training. The final mask obtained in step S3 and the actual segmentation mask of the original image are compared and optimized through Balanced Normalized Focal Loss to complete the image segmentation.

[0011] In a second aspect, an electronic device includes:

[0012] at least one processor;

[0013] At least one memory for storing at least one program;

[0014] When the at least one program is executed by the at least one processor, the at least one processor implements the steps of the method described in the first aspect.

[0015] In a third aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0017] The beneficial effects of this application are as follows:

[0018] 1. This application proposes an attention-based click and image fusion module for interactive segmentation tasks. This module, called the Click-aware pixel attention fusion module, effectively enhances the interactivity between clicks and images while significantly improving the data specificity of fused features in both channel and spatial dimensions.

[0019] 2. This application proves that interactive segmentation tasks often suffer from the pixel imbalance problem and defines this problem for the first time. It proposes a novel loss, Balanced normalized focal loss, which introduces a balancing weight to adjust the gradient centroid weights of positive click pixels and negative click pixels. It can significantly improve the performance of the model under pixel imbalance conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0021] Figure 1 This is a flowchart of an attention-based feature fusion network interactive image segmentation method provided in an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of the process structure of an attention-based feature fusion network interactive image segmentation method provided in an embodiment of the present application;

[0023] Figure 3 is a structural diagram of a click-aware pixel attention fusion module provided in an embodiment of the present application;

[0024] Figure 4 This is one of the hardware structure diagrams of the electronic device provided in the embodiment of the present application;

[0025] Figure 5 This is the second hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0027] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0028] The following describes in detail the attention-based feature fusion network interactive image segmentation method provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0029] See Figure 1 , is an attention-based feature fusion network interactive image segmentation method provided in an embodiment of the present application, comprising the following steps:

[0030] Step S1: Data preparation, providing image data, the actual segmentation mask of the original image, and click data, converting the image data into RGB form, and encoding the click data into spatial form in the form of coordinates;

[0031] Step S2: Data fusion: input the image data and click data into the click-aware pixel attention fusion module in the model to obtain fusion features;

[0032] Step S3: Mask segmentation: Input the obtained fusion features into the segmentation network, first use the HRNet backbone network in the segmentation network for deep encoding, then use the OCR network and linear processing layer in the segmentation network to decode the deep-encoded fusion features, and finally the segmentation network outputs the final mask of the target;

[0033] Step S4: Model training. The final mask obtained in step S3 and the actual segmentation mask of the original image are compared and optimized through Balanced Normalized Focal Loss to complete the image segmentation.

[0034] In step S1, this application follows the strategy of the RITM model to process click data and image data, and all settings remain consistent with RITM.

[0035] In step S2, the present application considers the fusion problem of clicks and images from another perspective. Most existing methods use addition to fuse clicks and images, because the use of addition can directly add the information in the click to the corresponding dimension of the image, will not change the number of channels of the image, and can also make the feature information more direct and sufficient, which is convenient for subsequent feature extraction. Although this method can bring the click information into the image, simple addition will weaken the model's interaction ability, which in turn leads to insufficient specificity of the fusion result. Fortunately, studies have shown that the appropriate use of CNNs and attention mechanisms in an organic combination can effectively enhance the model's ability to represent features and improve the specificity of the model's fusion data. Based on this, the present application designs a Click-aware pixel attention fusion module based on CNNs and attention mechanisms, which can improve the problem of insufficient interaction in the process of click and image fusion, improve the specificity of the fusion results, and significantly enhance the model's feature fusion and extraction capabilities.

[0036] Combine Figure 2 and 3 As shown, specifically including:

[0037] Step S21: extract features from the click data and image data respectively to obtain deep features;

[0038] Step S22: directly fuse the acquired deep features to obtain coarse fusion features, i.e., the first feature interaction;

[0039] Step S23: Use the channel attention model to perform noise filtering and click enhancement on the coarse fusion features, and then perform a secondary fusion of the clicks and the coarse fusion features, i.e., the second feature interaction;

[0040] Step S24: Utilize the spatial attention model and image features to filter heterogeneous information in the spatial dimension of the obtained coarse fusion features, i.e., the third feature interaction;

[0041] Step S25: Characterize the fused features using a deep feature encoder.

[0042] Step S22 specifically includes:

[0043] Step S221: Given a set {C, I}, where click C = {C pos , C Neg , C pre}, C pos , C Neg and C pre Represents positive click, negative click and previous click in click, image I∈R H×W×C , H, W and C represent the length, width and number of channels of the image respectively;

[0044] Step S222: Design a feature encoding module ω t Used to encode the features of click C, specifically:

[0045] C t =ω t (C)

[0046] Among them, click C t is the feature after clicking C encoding, ω t It is a convolution mapping function based on disks with a fixed radius;

[0047] Step S223: Design an image encoder ω f Used to get and click C t Corresponding image feature I t , specifically:

[0048] I t =ω f (I)

[0049] Among them, ω t is the convolution mapping function;

[0050] Step S224: Given a combination of click and image features {C t , I t}, first click C t With image feature I t The features of t , specifically:

[0051] F t =[C t , I t ]

[0052] Among them, [*] is the addition operation.

[0053] It should be noted that in the coarse fusion features obtained using [*], clicks and image pixel information are treated equally. This weakens clicks in the fused features, making the model insensitive to clicks. To address this issue, this application uses a lightweight attention model to enhance the click features that are weakened in this process, thereby increasing the model's sensitivity to clicks.

[0054] Step S23 specifically includes:

[0055] Channel Attention Model (A C ) mainly enhances the weakened click feature at the channel level, which is expressed by the following formula:

[0056]

[0057] To avoid feature F t After the channel self-attention model loses a lot of content information, the click C is directly added. t With feature F A c Features are directly fused to enhance click information, specifically:

[0058] F A c =[C t , F A c ]

[0059] In summary, the channel attention model can suppress the channel heterogeneous information in the coarse fusion features, making the model pay more attention to the click information in the channel.

[0060] It is further explained that the channel attention model is divided into a channel attention branch and a spatial attention branch.

[0061] Among them, channel attention branch: (A C ) mainly enhances the weakened click features at the channel level;

[0062] Spatial attention branch: To suppress F A c In terms of heterogeneous information in the spatial dimension, the study adopted a spatial attention module (A S ) to achieve filtering of heterogeneous information in the spatial dimension.

[0063] Step S24 specifically includes:

[0064] is the suppression feature F A c In terms of heterogeneous information in the spatial dimension, the spatial attention module (A S ) to filter heterogeneous information in the spatial dimension, specifically:

[0065]

[0066] After the above feature filtering, feature F A c The other information in the image is weakened, and the click feature is enhanced again. However, considering the image content loss in the process of click enhancement, the original image feature I t Add enhanced feature F A s middle:

[0067] A t =[I t , F As ]

[0068] Through the above feature layer, feature F t The click features in the fusion are enhanced, the specificity of the fusion results is improved, and the sensitivity of the model to clicks is enhanced.

[0069] In step S25, to better characterize A t , specifically including:

[0070] Using a deep feature encoder ω j To characterize the fused features, specifically:

[0071] F C =ω j (A t )

[0072] Among them, F C It is the fusion feature obtained after processing that can be directly used in the segmentation part.

[0073] In summary, the CPAFM module for click and image fusion proposed in this application organically uses a combination of CNNs and attention mechanisms, which can enable clicks and images to interact more fully, effectively improving the full interaction between clicks and images while significantly improving the data specificity of the fusion features.

[0074] In step S3, like RITM, this application uses HRNet as the backbone to perform deep encoding on the fused features, and then uses the OCR network and linear processing layer to decode the encoded features. Finally, the model outputs the final mask of the target (P n ).

[0075] In step S4, existing models are mostly trained using the BCE or FL loss series, which have achieved state-of-the-art results in previous work. However, studies have shown that the BCE loss treats all samples equally. This slows down training in the later stages of training because the difference between gradients from correctly segmented regions and those from incorrect regions is minimal. Therefore, the FL loss series was introduced to address this issue. However, this paper argues that the reason the FL loss series effectively improves the accuracy of the IIS model is not because it effectively distinguishes gradients from correctly segmented regions from those from incorrect regions, but rather because it indirectly alleviates the pixel-difficulty imbalance problem defined in this paper.

[0076] Therefore, unlike previous studies, this application considers the loss problem in existing work from the perspective of the pixel imbalance problem that is prevalent in IIS segmentation tasks. The pixel imbalance problem is divided into sample difficulty imbalance and pixel gradient imbalance. Figure 3 shown.

[0077] This application first proposed the BNFL loss and mathematically proved the correctness and rationality of the function. A large number of experiments have shown that BNFL can significantly improve the performance of the IIS model under pixel imbalance.

[0078] It is worth mentioning that the model proposed in this paper achieved SOTA performance on the public beta dataset without using any mask enhancement module, which demonstrates the superiority and effectiveness of the model proposed in this paper.

[0079] Step S4 specifically includes:

[0080] The final mask P obtained in step S3 n And the actual mask Ground Truth (y corresponding to the original image t ), according to the Focal loss commonly used in interactive segmentation tasks, it is defined as:

[0081]

[0082] right Perform the following Taylor transform:

[0083]

[0084] Taking the derivative we get:

[0085]

[0086] From the above formula, we can get: The gradient value of is mainly adjusted by γ. Studies have shown that it is because The adjustment factor of the series function adjusts the constant term 1 in the original BCE loss gradient to (1+γ)(1-P n ) γ and (1+γ)(P n ) γ ,so It can solve the problem of sample difficulty in the image while preventing the model from overfitting. The preceding coefficients of the above formula are discussed as follows. If you encounter difficult classification samples, The precondition coefficient satisfies On the contrary, if you encounter an easy sample Due to this characteristic, FL loss has the problem of gradient decay over time, which increases the probability of misjudging boundary pixels. Indeed, compared with BCE loss, Although it can prevent model overfitting to a certain extent, the benefits it brings to IIS tasks are limited.

[0087] In addition, it can be seen from the above formula that, except for the first term, the gradient values ​​of the remaining terms will shrink exponentially, which indicates that the first polynomial of the gradient in the above formula is In this study, this application defines it as The centroid of the gradient plays a leading role in the back propagation of the gradient. Therefore, the core idea of ​​this application is to accurately guide the change of the gradient value during the training process by adjusting the size of the centroid.

[0088] In the IIS task, the pixel (y t =1) is much smaller than the negative click pixel (y t = others). This will cause the gradient changes of the model in the later stages of training to mainly come from negative click pixels, which will lead to slow gradient updates and reduced convergence speed in the later stages of training, severely limiting the robustness and accuracy of the model. In order to effectively solve or alleviate this pixel imbalance problem in the IIS task, Based on this, a balance weight ε= Acting on the above formula to control the gradient centroid direction of positive click pixels and negative click pixels, so as to achieve the purpose of alleviating pixel gradient imbalance, It is expressed by the following formula:

[0089]

[0090] From this we can get The loss is expressed as follows:

[0091]

[0092] According to the above formula, it can be easily obtained. When targeting a positive click pixel, the gradient centroid term becomes [(1+ε)(1+γ)(1-P n ) γ ], and the gradient centroid term of the negative click pixel becomes [(1+(1-ε))(1+γ)(P n ) γ ]. Therefore, when a pixel is positively clicked, the gradient centroid term of the positive click is more enhanced, while the gradient centroid term of the negative click pixel is relatively weakened, thereby alleviating the pixel gradient imbalance problem.

[0093] In order to eliminate or alleviate the common problem of sample difficulty imbalance in IIS tasks, this application introduces the normalized term in the normalized focal loss and adds it to the proposed China overcomes primitive The total gradient loss problem caused by the gradient in the gradient decaying over time. The loss can also inherit the ability of NFL loss to classify pixel difficulty imbalance. While solving the pixel difficulty imbalance, it emphasizes the balance between positive and negative pixel gradients, thereby solving the pixel imbalance problem.

[0094] In summary, the application The final form of the loss can be expressed as:

[0095]

[0096] Where, and y t When taking different values The total gradient sum of the corresponding network output, and It is not included in the back propagation calculation process and is only used to normalize the FL loss value to solve the problem of gradient decay over time.

[0097] The attention-based feature fusion network interactive image segmentation method provided by this application is described in detail below with specific embodiment 1.

[0098] Example 1

[0099] In this experiment, the batch size is set to 32, and the original image is fixedly cropped to an image of size 320*480 and input into the network. For all experiments in this application, the Adam optimizer is used to optimize the training of the model in this article, and the parameter settings are specifically β1=0.9, β2=0.999. The pre-trained HRNet weights are used as the feature extraction module. The above parameter settings all follow the principle of fairness and are consistent with the experimental parameter settings of the existing methods. All model training and testing are performed on two NAVIDIATITAN RTX 6000. The model of this application is trained on the SBD dataset and the COCO+LVIS dataset, and the performance of the model of this application is verified on the GarbCut, SBD, DAVIS, and Berkeley datasets.

[0100] Table 1 shows the accuracy comparison of the proposed model and several advanced methods on the smaller datasets of Augmented VOC and SBD. Compared with traditional methods such as GraphCut and Geodesic star convexity, the proposed model is able to obtain object masks with high IoU with fewer clicks, thus achieving better accuracy in a more convenient manner. Even in comparison with all state-of-the-art methods, the proposed model achieves the best performance in all metrics, resulting in significant accuracy improvements. This significant improvement is due to the advanced components proposed in this model, such as BNFL, which can enhance model training, and CPAFM, which provides highly specific fused visual features.

[0101] Table 1

[0102]

[0103] Table 2 further illustrates the comparison results of our model with existing state-of-the-art models on the large dataset COCO-LVIS. Compared to other comparable models that use HRNet as a backbone, such as EdgeFlow, f-BRS-B, RITM, and Pseudoclick, our proposed model demonstrates a significant performance improvement.

[0104] Table 2

[0105]

[0106] At the same time, in order to objectively evaluate the performance of the BNFL loss proposed in this paper in the interactive segmentation task, this paper trains the model on the COCO+LVIS dataset and verifies the effectiveness of the BNFL loss proposed in this paper on four test datasets. It is compared with the existing loss functions commonly used in interactive segmentation tasks. The experimental results are shown in Table 3.

[0107] Table 3

[0108]

[0109] Theoretically, comparing the results of FL and BCE loss, we can see that the introduction of FL can improve the performance of the model. However, due to the problem of gradient decay over time in FL loss, this once again verifies what we mentioned in the theoretical analysis, that FL does not significantly improve the model in interactive segmentation tasks. The significant improvement of NFL in this regard verifies that the theoretical analysis of NFL above improves the gradient decay problem of FL in interactive segmentation tasks. Compared with NFL, the improvement brought by BNFL proposed in this invention is more significant. BNFL proposed in this invention can more effectively address the pixel imbalance problem of the model in interactive segmentation tasks, thereby significantly improving model performance.

[0110] like Figure 4 As shown, an embodiment of the present application also provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned embodiment of the attention-based feature fusion network interactive image segmentation method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0111] It should be noted that the first electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device mentioned above.

[0112] Figure 5 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0113] The electronic device 700 includes but is not limited to components such as a radio frequency unit 701 , a network module 702 , an audio output unit 703 , an input unit 704 , a sensor 705 , a display unit 706 , a user input unit 707 , an interface unit 708 , a memory 709 , and a processor 710 .

[0114] Those skilled in the art will understand that the electronic device 700 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0115] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The graphics processor 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, power keys, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0116] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned embodiment of the attention-based feature fusion network interactive image segmentation method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0117] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0118] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the attention-based feature fusion network interactive image segmentation method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0119] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0120] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0121] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0122] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned embodiment of the attention-based feature fusion network interactive image segmentation method, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0123] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0124] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An attention-based feature fusion network interactive image segmentation method, characterized in that: The following steps are involved: Step S1: Data preparation, providing image data, the actual segmentation mask of the original image, and click data, converting the image data into RGB form, and encoding the click data into spatial form in the form of coordinates; Step S2: Data fusion: Input the image data and click data into the Click-aware pixel attention fusion module in the model to obtain fusion features. Step S2 specifically includes: Step S21: extract features from the click data and image data respectively to obtain deep features; Step S22: directly fuse the acquired deep features to obtain coarse fusion features; Step S23: Use the channel attention model to perform noise filtering and click enhancement on the coarse fusion features, and then perform secondary fusion of the clicks and the coarse fusion features; Step S24: Filtering heterogeneous information in the spatial dimension of the obtained coarse fusion features using the spatial attention model and image features; Step S25: using a deep feature encoder to characterize the fused features; Step S3: Mask segmentation: Input the obtained fusion features into the segmentation network, first use the HRNet backbone network in the segmentation network for deep encoding, then use the OCR network and linear processing layer in the segmentation network to decode the deep-encoded fusion features, and finally the segmentation network outputs the final mask of the target; Step S4: Model training. The final mask obtained in step S3 and the actual segmentation mask of the original image are compared and optimized through Balanced Normalized Focal Loss to complete the image segmentation.

2. The method according to claim 1, characterized in that Step S22 specifically includes: Step S221: Given a group , where click , , as well as Represents positive click, negative click and previous click in click, image , , and Respectively represent the length, width and number of channels of the image; Step S222: Design a feature encoding module Used to encode clicks The features are: Among them, click For click The encoded features, is a convolution mapping function based on disks with a fixed radius; Step S223: Design an image encoder To obtain and Corresponding image features , specifically: in, is the convolution mapping function; Step S224: Given a combination of click and image features , first click and image features The features are directly fused to obtain the fusion features , specifically: in, For addition operation.

3. The method according to claim 2, characterized in that Step S23 specifically includes: Channel Attention Model The weakened click feature is enhanced mainly at the channel level, which is expressed by the following formula: To avoid fusion features After the channel self-attention model loses a lot of content information, the click is directly added. and features Direct fusion is performed to enhance click information, specifically: 。 4. The method according to claim 1, wherein Step S24 specifically includes: To suppress features Heterogeneous information in the spatial dimension, using the spatial attention module To achieve the filtering of heterogeneous information in the spatial dimension, specifically: After filtering the above features, the features The other information in the image is weakened, and the click feature is enhanced again, but considering the image content loss in the process of click enhancement, the original image feature is Add enhanced features middle: Through the above feature layer, features The click features in the fusion are enhanced, the specificity of the fusion results is improved, and the sensitivity of the model to clicks is enhanced.

5. The method according to claim 1, wherein Step S25 specifically includes: Using a deep feature encoder To characterize the fused features, specifically: Among them, the characteristics It is the fusion feature obtained after processing that can be directly used in the segmentation part.

6. The method according to claim 1, wherein Step S4 specifically includes: The final mask obtained according to step S3 And the actual mask Ground Truth corresponding to the original image , according to the Focal loss commonly used in interactive segmentation tasks, it is defined as: right Perform the following Taylor transform: Taking the derivative we get: exist Introduce a balance weight based on Acting on the above formula to control the gradient centroid direction of positive click pixels and negative click pixels, so as to achieve the purpose of alleviating pixel gradient imbalance, It is expressed by the following formula: From this we can get The loss is expressed as follows: The final form of the loss can be expressed as: Where, and They are When taking different values The total gradient sum of the corresponding network output, and Not included in the back propagation calculation process, only used for The loss value is normalized to solve the problem of gradient decay over time.

7. An electronic device, characterized in that: include: at least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the attention-based feature fusion network interactive image segmentation method described in any one of claims 1-6.

8. A storage medium, characterized in that: Processor-executable instructions are stored, and when the processor executes the processor-executable instructions, the attention-based feature fusion network interactive image segmentation method described in any one of claims 1-6 is executed.

Images