Image segmentation method, device, equipment, program product and medium

Abstract

The present invention provides an image segmentation method, an image segmentation method, an apparatus, a device, and a medium. The method comprises: extracting intra-category image features corresponding to a feature image; obtaining a block sequence of the feature image through convolution calculation based on the intra-category image features; performing pixel set calculation based on the intra-category image features to obtain a category center feature of the feature image; interacting the block sequence with the category center feature to obtain an enhanced category center feature; concatenating the enhanced category center feature with the intra-category image feature to obtain an inter-category image feature; and segmenting the feature image using the inter-category image feature to obtain a segmentation result of the feature image of the target object. The present invention can improve image segmentation accuracy and adapt to different image segmentation scenarios.

Description

Technical Field

[0001] The present invention relates to machine learning technology, and in particular to an image segmentation method, device, electronic device, computer program product and storage medium. Background Art

[0002] Various types of recognition based on deep learning have always been an important tool for solving large-scale data classification in various application scenarios. For example, in application scenarios such as image and natural language processing, large-scale classification and recognition of large amounts of data can quickly and accurately obtain relevant classification prediction results, accelerating the implementation of the application's functions.

[0003] When performing image classification prediction, the specific images used for classification prediction and how to implement it vary depending on the application scenario being deployed, such as in AI+medical scenarios. Taking AI+medical scenarios as an example, various medical images generated by different diagnostic and treatment equipment are constantly being generated. For example, as a patient's condition progresses, images are continuously captured at different time points or within departments, generating a large amount of data. This urgently requires the use of image segmentation to segment medical images.

[0004] However, in related technologies, although deep convolutional neural network algorithms are widely used in the segmentation and enhancement of medical images, they cannot take into account both overall and local features, and it is also difficult to take into account both network accuracy and memory consumption. Although the linear regression model can improve the processing speed, it cannot fit the features in complex environments, resulting in poor processing accuracy of the neural network model. At the same time, the model migration ability in complex environments is poor, which affects the processing speed of the neural network model. Summary of the Invention

[0005] In view of this, the embodiments of the present invention provide an image segmentation method, device, electronic device, computer program product and storage medium, which can fully learn the characteristics of image features within a category and image features between categories, solve the problem of insufficient consistency within a category and insufficient difference between categories, achieve feature consistency within a category, and at the same time strengthen the feature distinctiveness between different categories through inter-category constraints, so that the segmentation results of the feature image of the target object are more accurate, the segmentation boundaries are clearer, and can adapt to image segmentation environments with various complex image backgrounds.

[0006] The technical solution of the embodiment of the present invention is achieved as follows:

[0007] An embodiment of the present invention provides an image segmentation method, the method comprising:

[0008] Obtaining a feature image of the target object;

[0009] Extracting intra-category image features corresponding to the feature image;

[0010] According to the image features within the category, a block sequence of the feature image is obtained through convolution calculation;

[0011] Perform pixel set calculation based on the image features within the category to obtain the category center features of the feature image;

[0012] Interacting the block sequence with the category center feature to obtain an enhanced category center feature;

[0013] splicing the enhanced category center feature and the intra-category image feature to obtain an inter-category image feature;

[0014] The feature image is segmented using the inter-category image features to obtain a segmentation result of the feature image of the target object.

[0015] An embodiment of the present invention further provides an image segmentation device, comprising:

[0016] An information transmission module is used to obtain a characteristic image of a target object;

[0017] An information processing module, configured to extract intra-category image features corresponding to the feature image;

[0018] The information processing module is used to obtain a block sequence of the feature image through convolution calculation according to the image features within the category;

[0019] The information processing module is used to perform pixel set calculation based on the image features within the category to obtain the category center features of the feature image;

[0020] The information processing module is used to interact with the block sequence and the category center feature to obtain an enhanced category center feature;

[0021] The information processing module is used to combine the enhanced category center features and the intra-category image features to obtain inter-category image features;

[0022] The information processing module is used to segment the feature image using the inter-category image features to obtain a segmentation result of the feature image of the target object.

[0023] In the above solution, the information processing module is used to perform iterative and decoupling processing on the feature image to obtain a decoupled feature image set;

[0024] The information processing module is used to calculate the query features corresponding to the target category feature images in the decoupled feature image set;

[0025] The information processing module is used to perform weighted processing on the query features to obtain a saliency perception map;

[0026] The information processing module is used to extract salient features of the target category feature image according to the saliency perception map;

[0027] The information processing module is used to calculate the intra-category image features based on the salient features of the target category feature image.

[0028] In the above solution, the information processing module is used to obtain a sliding window that matches the target object;

[0029] The information processing module is configured to divide the saliency perception map by the sliding window to obtain at least two non-overlapping sub-regions;

[0030] The information processing module is used to determine a significant position in each sub-region;

[0031] The information processing module is configured to extract salient features from the target category feature image based on the salient positions;

[0032] The information processing module is used to project the significant features to obtain key values ​​and value items;

[0033] The information processing module is used to calculate the features of enhanced sequences of different categories using the key and value items through a multi-head attention mechanism;

[0034] The information processing module is used to calculate the intra-category image features based on the features of the enhanced sequences of different categories.

[0035] In the above solution, the information processing module is used to determine the information extracted at different scales according to the image segmentation requirements of the target object;

[0036] The information processing module is configured to determine the total number of scales based on the information extracted at the different scales;

[0037] The information processing module is configured to determine each sliding window that matches the target object according to the total scale quantity.

[0038] In the above solution, the information processing module is used to determine the convolution step size that matches the image features within the category;

[0039] The information processing module is used to perform convolution processing on the image features within the category according to the convolution step size to obtain a block sequence of the feature image.

[0040] In the above solution, the information processing module is used to calculate the initial segmentation result of each category based on the image features within the category;

[0041] The information processing module is used to calculate a pixel representation set of a target category feature image based on the initial segmentation result;

[0042] The information processing module is used to determine the category center feature of the feature image based on the pixel representation set of the target category feature image.

[0043] In the above solution, the information processing module is used to splice the block sequence and the category center feature to obtain a spliced ​​feature;

[0044] The information processing module is used to perform feature complementation on the splicing features through an attention matrix to obtain complementary splicing features, wherein the attention matrix includes four groups of normalized similarity matrices;

[0045] The information processing module is used to split the complementary splicing features according to the splicing order to obtain enhanced block sequences and enhanced category center features.

[0046] In the above solution, the information processing module is used to obtain the feature image of the target object through the intra-category dynamic converter network model;

[0047] The information processing module is configured to extract intra-category image features corresponding to the feature image through the intra-category dynamic converter network model;

[0048] The information processing module is configured to obtain a block sequence of the feature image by performing convolution calculation based on the intra-category image features using the inter-category dynamic converter network model;

[0049] The information processing module is configured to perform pixel set calculation based on the intra-category image features through the inter-category dynamic converter network model to obtain the category center features of the feature image;

[0050] The information processing module is used to interact the block sequence and the category center feature through the inter-category dynamic converter network model to obtain an enhanced category center feature;

[0051] The information processing module is used to splice the enhanced category center features and the intra-category image features through the inter-category dynamic converter network model to obtain inter-category image features.

[0052] An embodiment of the present invention further provides an electronic device, including:

[0053] a memory for storing executable instructions;

[0054] The processor is configured to implement the preceding image segmentation method when running the executable instructions stored in the memory.

[0055] An embodiment of the present invention further provides a computer program product, which implements the aforementioned image segmentation method when the computer program or instructions are executed by a processor.

[0056] An embodiment of the present invention further provides a computer-readable storage medium storing executable instructions, wherein the executable instructions implement the preceding image segmentation method when executed by a processor.

[0057] The embodiments of the present invention have the following beneficial effects:

[0058] The embodiment of the present invention obtains a feature image of a target object; extracts the intra-category image features corresponding to the feature image; obtains a block sequence of the feature image based on the intra-category image features through convolution calculation; performs pixel set calculation based on the intra-category image features to obtain the category center features of the feature image; interacts the block sequence with the category center features to obtain an enhanced category center feature; splices the enhanced category center feature with the intra-category image features to obtain inter-category image features; and segments the feature image based on the inter-category image features to obtain a segmentation result of the feature image of the target object. In this way, the characteristics of the intra-category image features and the inter-category image features can be fully learned, the problem of insufficient intra-category consistency and insufficient inter-category difference can be solved, and the intra-category feature consistency can be achieved. At the same time, the feature distinction between different categories is strengthened through inter-category constraints, so that the segmentation result of the feature image of the target object is more accurate, the segmentation boundary is clearer, and it can adapt to image segmentation environments with various complex image backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a schematic diagram of an application environment of an image segmentation method provided by an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of the principle of using the Transformer network for image segmentation in the traditional solution;

[0061] Figure 3 This is a schematic diagram of the effect of using the Transformer network for image segmentation in the traditional solution;

[0062] Figure 4 An optional flowchart of the image segmentation method provided in this application;

[0063] Figure 5 Schematic diagram of the model structure of the class perception network model in an embodiment of the present invention;

[0064] Figure 6 A schematic diagram of the working process of the intra-category dynamic converter network model according to an embodiment of the present invention;

[0065] Figure 7 A schematic diagram of the working process of the inter-category dynamic converter network model according to an embodiment of the present invention;

[0066] Figure 8 Schematic diagram of the comparison between the image segmentation method according to an embodiment of the present invention and related technologies in the Synapse dataset;

[0067] Figure 9 Schematic diagram of the comparison between the image segmentation method according to the embodiment of the present invention and related technologies in the ACDC dataset and the MoNuSeg dataset;

[0068] Figure 10 Schematic diagram showing comparison of segmentation effects of the image segmentation method according to an embodiment of the present invention and related technologies in a test on the Synapse dataset;

[0069] Figure 11 Schematic diagram showing comparison of segmentation effects of the image segmentation method according to an embodiment of the present invention and related technologies in tests on the ACDC dataset;

[0070] Figure 12 Schematic diagram showing comparison of segmentation effects of the image segmentation method according to an embodiment of the present invention and related technologies in tests on the MoNuSeg dataset;

[0071] Figure 13 A schematic diagram of a usage scenario of the image segmentation method provided by an embodiment of the present invention;

[0072] Figure 14 An optional flowchart of the image segmentation method provided by an embodiment of the present invention;

[0073] Figure 15 An optional schematic diagram of the image processing method provided by an embodiment of the present invention.

[0074] Figure 16 Schematic diagram of a front-end for segmenting medical images using an image segmentation method according to an embodiment of the present invention;

[0075] Figure 17 A schematic diagram of the structure of an image segmentation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0077] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0078] Before further explaining the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0079] 1) Convolutional Neural Networks (CNNs) are a type of feed-forward neural network with a deep structure that incorporates convolutional computations. They are a representative algorithm for deep learning. CNNs possess representation learning capabilities and can perform shift-invariant classification on input information based on their hierarchical structure.

[0080] 2) Model training: Multi-classification learning is performed on image datasets. This model can be built using deep learning frameworks such as TensorFlow and Torch, using multiple layers of neural network layers such as CNN to form a multi-segmentation network. The model input is a three-channel or original channel matrix generated by reading the image using tools such as OpenCV. The model output is multi-class probabilities, and the image segmentation results are finally output using algorithms such as softmax. During training, the model approaches the correct trend using objective functions such as cross entropy.

[0081] 3) Neural Network (NN): Artificial Neural Network (ANN), also known as neural network or neural network-like network, is a mathematical model or computational model that imitates the structure and function of biological neural networks (the central nervous system of animals, especially the brain) in the fields of machine learning and cognitive science. It is used to estimate or approximate functions.

[0082] 4) Computer Aided Diagnosis (AD): AD is used to assist in the discovery of lesions and improve the accuracy of diagnosis through imaging, medical image processing technology and other possible physiological and biochemical means, combined with computer analysis and calculation.

[0083] 5) Endoscopic video stream: used to collect pathological information of the video state generated by image acquisition of body parts (different target organs of the human body or lesions in the body) through image acquisition equipment (such as endoscope).

[0084] Figure 1 Schematic diagram of the use scenario of the image segmentation method provided in the embodiment of the present invention, wherein the image segmentation method provided in this application can be used to process medical images in a medical environment, see Figure 1 The terminal (including terminal 10-1 and terminal 10-2) is provided with a corresponding client that can perform different functions. Among them, the client is that the terminal (including terminal 10-1 and terminal 10-2) obtains different medical images of corresponding target objects from the corresponding server 200 through the network 300 for browsing, or obtains corresponding medical images and analyzes the target area (such as the area of ​​lesion tissue) shown in the medical image. The terminal is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and a wireless link is used to realize data transmission. Among them, the types of medical images of corresponding target objects obtained by the terminal (including terminal 10-1 and terminal 10-2) from the corresponding server 200 through the network 300 can be the same or different. For example: the terminal (including terminal 10-1 and terminal 10-2) can obtain pathological images or medical image sets matching the target object from the corresponding server 200 through the network 300, or obtain a medical image set (such as CT images) that only matches the current target from the corresponding server 200 through the network 300 for browsing. The server 200 may store medical images of target objects corresponding to different target objects, and may also store auxiliary analysis information that matches the medical images of target objects corresponding to the target objects. In some embodiments of the present invention, the different types of medical images of target objects stored in the server 200 may be endoscopic images acquired by an endoscope, or CT images of patients acquired by a CT machine.

[0085] Among them, medical imaging refers to the technology and processing process of obtaining internal tissue images of the human body or a part of the human body in a non-invasive manner for medical treatment or medical research, including but not limited to: CT, MRI, ultrasound, X-ray, electrocardiogram, electroencephalogram, optical photography, etc., which are generated by medical instruments. It is an important means and reference factor for assisting clinical diagnosis. The intrinsic heterogeneity of different symptoms is also reflected in their imaging phenotypes (appearance and shape). Therefore, the use of medical imaging to perform etiology diagnosis or image segmentation of lesion tissue areas can effectively assist doctors in making accurate pathological diagnoses. In related technologies, deep convolutional neural network algorithms have been widely used in image segmentation. Various medical images generated by different diagnostic and treatment equipment are constantly generated. For example, as the patient's condition progresses, they need to be continuously photographed at different time points or in departments, which in turn become a large amount of data. It is urgent to use the execution of classification prediction to achieve large-scale classification and recognition.

[0086] However, in related technologies, although deep convolutional neural network algorithms are widely used in image segmentation, most classic segmentation methods cannot take into account both overall and local features, and it is also difficult to take into account both network accuracy and memory consumption. Medical images have more details that need to be captured by the network, so it is necessary to obtain auxiliary diagnostic information for large 2D diagnostic images and 3D diagnostic images through deeper networks and more local feature extraction.

[0087] Among them, conventional neural network technology often adopts an encoder-decoder structure (encoder-decoder technology) for segmentation, that is, first downsampling the image to extract features, and then upsampling to return to the original image size. It maintains a high-resolution image channel to retain valid information, but due to high-resolution operations, the image itself occupies a large amount of memory, and multi-scale will bring more channels, resulting in an increase in parameter memory usage. Therefore, it may only be able to construct a shallow network and cannot be applied to larger 2D or 3D images.

[0088] Taking a step further, we can also use the reversible residual network RevNet technology. The advantage of RevNet is that it can be connected in series almost infinitely without increasing the memory consumption caused by image calculations. It only adds some intermediate calculation parameters. However, the problem with the reversible residual network is that its structure is too simple and calculations must be maintained at a certain resolution scale. Therefore, the limited complexity cannot improve the accuracy of network tasks.

[0089] In the use environment of medical images, taking endoscopic images as an example, there are no less than two original endoscopic images in the endoscopic video stream, which are a collection of multi-field pathological images obtained by the doctor repeatedly observing the suspected target object area by moving the camera, switching the magnification, etc. during the use of the endoscope, and integrate the information of the specific field of view under the endoscope. Since the endoscopic video stream records all the information in the endoscopic field of view during the doctor's observation of the patient's target object (such as the area of ​​lesion tissue), the information of the doctor observing a single patient target object (such as the area of ​​lesion tissue) in the endoscope's field of view is used as a continuous video stream to avoid the doctor ignoring tiny lesion areas during the rapid movement of the endoscope, thereby providing more information than a single frame image to assist the doctor in diagnosing and discovering tiny lesion areas. During this process, clear endoscopic images are needed to assist the doctor's diagnosis. However, due to the limitations of the mechanical endoscopic imaging environment or the operator's operational limitations, the endoscopic images presented are often endoscopic images of the general environment of the lesion, and cannot be focused on the specific location of the lesion for imaging. This is not conducive to the doctor's assistance in classification through endoscopic images focused on the lesion. Therefore, it is necessary to use the image segmentation method provided in this application to segment the endoscopic image using image segmentation to form a clear endoscopic image of the lesion for use by the doctor.

[0090] The embodiments of the present invention may be implemented in conjunction with cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network or local area network to enable data computing, storage, processing, and sharing. It can also be understood as a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business models. Backend services of technical network systems, such as video websites, image websites, and more portal websites, require a large amount of computing and storage resources. Therefore, cloud technology needs to be supported by cloud computing.

[0091] It should be noted that cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, the resources in the "cloud" appear to be infinitely scalable and can be accessed at any time, used on demand, and expanded at any time, with a pay-per-use fee. As a provider of cloud computing's basic capabilities, a cloud computing resource pool platform, often referred to as Infrastructure as a Service (IaaS), is established. Various types of virtual resources are deployed within the resource pool for external customers to choose from. The cloud computing resource pool primarily includes computing devices (which can be virtualized machines, including operating systems), storage devices, and network devices.

[0092] Combined with the embodiment Figure 1 As shown, the target object determination method provided in the embodiment of the present invention can be implemented by corresponding cloud devices. For example, terminals (including terminal 10-1 and terminal 10-2) are connected to server 200 located in the cloud via network 300. Network 300 can be a wide area network or a local area network, or a combination of the two. It is worth noting that server 200 can be a physical device or a virtualized device.

[0093] Specifically, in combination with the above embodiment Figure 1 As shown, the server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0094] It should be noted that the patient target object (e.g., the area of ​​lesion tissue) viewed under an endoscope (a medical device connected to the target object) can include a variety of different application scenarios, such as diabetic retinopathy screening, early cervical cancer screening, and other different video stream screenings. The image segmentation method based on this embodiment can be deployed in a variety of application scenarios, thereby facilitating remote viewing and use by doctors.

[0095] The server 200 transmits medical image information of corresponding target objects of the same target object to the terminal (terminal 10-1 and / or terminal 10-2) via the network 300 to enable the user of the terminal (terminal 10-1 and / or terminal 10-2) to analyze the image information of the corresponding target object of the target object. As an example, the server 200 deploys corresponding image segmentation, including the class-aware network model provided by the present application, for outputting inter-class image features, so that the image segmentation can use the inter-class image features to segment the feature image and obtain a segmentation result of the feature image of the target object.

[0096] Before introducing the image segmentation method provided by the embodiment of the present invention, the process of image segmentation in the related art of the present invention is first introduced, wherein: Figure 2 This is a schematic diagram of the principle of using the Transformer network for image segmentation in the traditional solution. Figure 3 This is a schematic diagram of the effect of using the Transformer network for image segmentation in the traditional solution, as shown in Figure 2 As shown, the Transformer network mainly consists of two structures: a multi-head attention mechanism (MHSA) and a position fully connected feedforward network (FFN). First, a feature map is input, and three nonlinear mappings can be used to obtain: query (Q), key value (K), and value item (V). First, the dot product between the transpose of Q and K is calculated, and then the obtained attention matrix (size is HW×HW) is normalized to a probability distribution using the Softmax operation, and then multiplied by the matrix V to obtain the weighted sum representation. The enhanced features are further input into the FFN to learn more nonlinear relationships between features. Use Figure 2 When the neural network model is used to segment medical images, Figure 3 As shown, Figure 3 (a) and (c) show the segmentation results of TransUNet. Figure 3 (b) and (d) show the segmentation results of SwinUNet. Although the pixels on the liver can capture the relationship with the pixels on the stomach, left kidney, right kidney and other organs from a distance, the probability of the stomach being segmented into the spleen is high, and the edges of the segmentation results are blurred. Therefore, the traditional Transformer method usually acts on feature maps that do not distinguish categories, that is, all category objects are highlighted at the same time. The weighted collection of all these salient regions (based on multi-object reasoning) will lead to confusing pixel-to-pixel relationships, which may impair intra-category consistency learning, for example Figure 3Incomplete stomach segmentation shown. Meanwhile, traditional Transformers typically only model pixel-level dependencies, but often ignore object-to-object correlations between semantic categories. This limits the ability to accurately segment different organs, especially for organs with similar contextual information and close locations, which can easily cause confusion, for example Figure 3 A portion of the midstomach was mistakenly divided into the spleen.

[0097] To solve the above defects, refer to Figure 4 , Figure 4 This is an optional flow chart of the image segmentation method provided in this application. It can be understood that: Figure 4 The steps shown can be performed by various electronic devices running segmentation network devices, such as a medical image processing terminal, a server or server cluster with image segmentation function, to achieve accurate segmentation of complex images adapted to different usage scenarios. Figure 4 The steps shown are explained.

[0098] Step 401: The image segmentation device obtains a feature image of a target object.

[0099] In some embodiments of the present invention, the image segmentation method provided by the present application can be implemented based on a class-aware network model, which includes at least an intra-class dynamic converter network model and an inter-class dynamic converter network model. Figure 5 , Figure 5 Figure 1 is a schematic diagram of the model structure of the class-aware network model in an embodiment of the present invention. The architecture of the class-aware network model (ClassFormer) includes two parts: the intra-class dynamic transformer network model (IDT Intra-Class Dynamic Transformer) and the inter-class dynamic transformer network model (IIT Inter-Class Interactive Transformer). The feature image X is extracted from a certain layer of the network as the input of the class-aware network model; it is processed by the IDT module on the left and the IIT module on the right; finally, the enhanced inter-class image feature X is output. inter By using the image features between categories, complex images can be segmented to obtain clear edges and accurate segmentation results. Figure 5In the structure shown, inter-class and intra-class are terms in image classification, which are used to describe two different features. For example, we all know that the data set used for training in face recognition tasks is divided into many different people. Each person is a category, and each category has several separate pictures. Among them, inter-class represents the features between categories, such as the differences between facial features of different people in face recognition; intra-class represents the features within the category, such as the differences in facial features of the same person in different states.

[0100] The following takes the image segmentation method provided in this application executed by the class perception network model as an example to continue to explain the processing process of steps 402 to 406.

[0101] Step 402: The image segmentation device extracts intra-category image features corresponding to the feature image.

[0102] In some embodiments of the present invention, when extracting intra-category image features corresponding to feature images through an intra-category dynamic transformer network model, it can be achieved in the following manner:

[0103] Iterate and decouple the feature image to obtain a set of decoupled feature images; calculate the query features corresponding to the target category feature image in the decoupled feature image set; perform weighted processing on the query features to obtain a saliency perception map; extract the salient features of the target category feature image based on the saliency perception map; calculate the intra-category image features based on the salient features of the target category feature image. Where H / W / D are the height, width and feature dimension of the feature image respectively. Then, the feature map that does not distinguish between categories is iterated C times to obtain Where C represents the number of categories. The repeated feature image is input into the group convolution layer with C groups to generate a decoupled feature image set in Indicates the feature of the Cth class, [·] represents the concatenation operation. In order to make the decoupled feature X C More explicitly include the C-th category information, let each decoupled feature generate a corresponding binary prediction result, that is, each predicted pixel value represents the possibility that the pixel belongs to that category. By using the binary cross entropy loss function L bce , each decoupled feature map gradually has explicit class-aware clues.

[0104] refer to Figure 6 , Figure 6The figure is a schematic diagram of the working process of the intra-category dynamic transformer network model in an embodiment of the present invention. In order to capture the long-range dependencies that distinguish categories, a simple method is to use a Transformer with global relationships on the feature graph of each class. However, in traditional Transformers, each query Q has to process a large number of key values ​​K, resulting in redundant correlation calculations between irrelevant K. Therefore, by Figure 6 The intra-category dynamic transformer network model shown can adaptively select the significant K in each category, thereby significantly reducing redundant computations and focusing more on significant features.

[0105] According to the salient features of the target category feature image, the intra-category image features are calculated, which can be achieved by: obtaining a sliding window that matches the target object; dividing the saliency perception map through the sliding window to obtain at least two non-overlapping sub-regions; determining a salient position in each sub-region; extracting salient features from the target category feature image based on the salient position; projecting the salient features to obtain key values ​​and value items; using the multi-head attention mechanism, using the key values ​​and value items, calculate the features of the enhanced sequences of different categories; and calculating the intra-category image features based on the features of the enhanced sequences of different categories. Figure 6 As shown, the C-th feature image X after input decoupling C , the query feature Q can be obtained by linear projection C =X c W q , and then input the query features into a simple saliency-aware network θ SN (including convolutional layers and activation layers), reweighting the pixels of the target category feature image, that is, increasing the weight of significant pixels and suppressing the weight of irrelevant pixels. SN In

[15] , the input features are first subjected to a 3*3 convolution operation and a GELU activation function to extract local features. The activation layer can use a GELU activation function, a Softmax activation function, or a ReLU activation function, which is not specifically limited in this embodiment of the present invention.

[0106] Since pooling along the channel dimension can effectively activate the information region, we can pool the maximum and average values ​​of the channel and then concatenate them. Figure 6 The concatenated features are processed through a 3*3 convolutional layer to obtain a saliency perception map. Where H / W are the corresponding height and width respectively. In order to realize the dynamic key value K selection in the entire space Λ, we first use a sliding window of size δ to divide the saliency perception map M into a set of non-overlapping sub-regions Λ s In each The position with the largest value is selected as the significant position Where p is the position of each pixel. Therefore, a set of salient positions is formed over the entire space Λ in Indicates the number of positions.

[0107] Since the image segmentation method provided by this application needs to segment different complex images, it is necessary to set the class perception network model as a plug-and-play module and integrate the class perception network model into the ResUNet network to segment complex images. Different image segmentation environments have different segmentation requirements and extraction scales. Therefore, in order to increase the versatility of the image segmentation method of this application and adapt to different image segmentation requirements, it is possible to determine the information extracted at different scales according to the image segmentation requirements of the target object; determine the total number of scales according to the information extracted at different scales; and determine each sliding window that matches the target object according to the total number of scales, referring to Figure 6 , a pyramid mechanism can be used to adaptively sample salient points from multiple scales. Specifically, for each scale, a different saliency perception network θ is used SN and the corresponding δ (the size of the sliding window), can adaptively learn unique patterns at different scales. The information extracted from different scales is integrated, where S is the total number of scales. From the input X C Extract salient features from Further through linear projection, the corresponding key value K is obtained c Sum value term V C ,in, Continue to capture long-range dependencies through the multi-head attention mechanism: Combine enhanced sequences from different classes Convert it into two-dimensional features and concatenate them with the original input features, then input them into a 1*1 convolutional layer to generate the final output category image features Where H / W / D are the height, width and feature dimension of the feature image respectively.

[0108] Through the processing of step 401-step 402, the intra-category dynamic converter network model completes the image processing process and outputs the intra-category image features to the inter-category dynamic converter network model. The inter-category dynamic converter network model transforms the image features within the category through steps 403 to 406. After processing, inter-category image features can be output. The working process of the inter-category dynamic converter network model is explained below.

[0109] Step 403: The image segmentation device obtains a block sequence of feature images through convolution calculation based on the image features within the category.

[0110] refer to Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the working process of the inter-category dynamic converter network model according to an embodiment of the present invention. Figure 7 As shown on the left side of the figure, first determine the convolution step size that matches the image features within the category; according to the convolution step size, perform convolution processing on the image features within the category to obtain a block sequence of feature images. Specifically, the image features within the input category Using a maximum pooling layer with a convolution step size of r and two consecutive 3*3 convolutional layers, a high-level representation of size H / r*W / r*D is obtained, and then its shape is further transformed into a block sequence (patch sequence) with rich semantics.

[0111]

[0112] Step 404: The image segmentation device performs pixel set calculation based on the image features within the category to obtain the category center features of the feature image.

[0113] for Figure 7 The category center features of the feature image used in the calculation can be used to calculate the initial segmentation results of each category based on the image features within the category; then, based on the initial segmentation results, the pixel representation set of the target category feature image is calculated; finally, based on the pixel representation set of the target category feature image, the category center features of the feature image are determined. Figure 7 As shown, first, we use the image feature X within the category intra Predict an initial segmentation result for each class Among them, y c is the initial segmentation result, H / W are the height and width of the feature image, and the representations of all pixels are combined according to the probability of belonging to the cth category: in represents the feature of the i-th pixel, is the probability that the i-th pixel belongs to the c-th class. It is represented as the central feature of the cth category. This overcomes the defect of related technologies that only focus on pixel-level relationships and ignore the category dependencies between semantic objects.

[0114] Step 405: The image segmentation device interacts with the block sequence and the category center feature to obtain an enhanced category center feature.

[0115] In some embodiments of the present invention, by interacting between a block sequence with high-level semantics and different category center features, the category correlation between the block sequence and the different category center features can be determined. When the block sequence and the category center feature interact, the block sequence and the category center feature are spliced ​​to obtain a spliced ​​feature; the spliced ​​feature is complemented by the attention matrix to obtain a complementary spliced ​​feature, wherein the attention matrix includes 4 groups of normalized similarity matrices; the complementary spliced ​​features are split according to the splicing order to obtain an enhanced block sequence and an enhanced category center feature. Specifically, referring to Figure 7 , for the block sequence and category center features The concatenation is then performed through Transformer to capture long-range interactions. Therefore, the shape of the features Q / K / V after the three linear projections is (C+HW / r 2 )*D, the obtained attention matrix A can be expressed as:

[0116]

[0117] Among them c→c , Α p→p , Α c→p , Α p→c , is the normalized similarity matrix, Α c→c Represents class to class, A p→p Indicates patch to patch, A c→p Indicates class to patch, Α p→c Indicates patch to class. c→c and A p→p is the normalized similarity matrix of the self-attention mechanism, Α c→p , Α p→c Cross-attention can be used to learn complementary clues and achieve feature interaction. Therefore, the block sequence of high-level semantics can explore explicit category dependencies from the category center features, thereby achieving class-guided feature enhancement; while the category center features can capture rich semantics from the block sequence and increase the difference between categories. Then, according to the previous splicing order, the enhanced sequence is split into block sequence and category center features. In order to further improve the difference of features, the enhanced category center features are subjected to the Euclidean distance loss function L EU Constraints are set to maximize the distance between enhanced class tokens.

[0118] Step 406: The image segmentation device combines the enhanced category center features and the intra-category image features to obtain inter-category image features.

[0119] like Figure 7As shown, when calculating the inter-category image features, in order to retain the source information, the input intra-category image features X intra After splicing, the final output image features between categories are obtained after a 1*1 convolution layer

[0120] Through the processing of steps 403 to 406, the inter-category dynamic converter network model converts the input intra-category image features into inter-category image features. The inter-category image features obtained in step 406 inherit the intra-category consistency, so that the image segmentation network can maintain the intra-category consistency and increase the inter-category difference, thereby improving the feature representation ability and producing high-quality segmentation performance.

[0121] Step 407: The image segmentation device segments the feature image using the inter-category image features to obtain a segmentation result of the feature image of the target object.

[0122] Since the image segmentation method provided in this application needs to segment different complex images, it is necessary to set the class-aware network model as a plug-and-play module and integrate the class-aware network model into the ResUNet network to segment complex images. Therefore, it is necessary to train the class-aware network model of the intra-class dynamic converter network model and the inter-class dynamic converter network model, and determine the parameters of the intra-class dynamic converter network model and the inter-class dynamic converter network model. The datasets are Synapse, ACDC, and MoNuSeg, which contain medical images with different numbers of categories and collected by different devices.

[0123] in:

[0124] The Synapse dataset contains 9 categories (8 abdominal organs and 1 background category), involving a total of 30 Computed Tomography (CT) scan cases, of which 18 cases are used for training and 12 cases are used for testing.

[0125] The ACDC dataset contains 4 categories (left ventricle (LV), right ventricle (RV), myocardium (MYO) and 1 background category), involving a total of 100 Magnetic Resonance Imaging (MRI) cases, of which 70 are used for training, 10 are used for validation, and 20 are used for testing.

[0126] The MoNuSeg dataset contains two categories (cell foreground and background) and a total of 44 pathology images, 30 of which are used for training images and 14 for testing images.

[0127] To avoid overfitting, two data augmentation methods, random rotation and random flipping, are used. Next, the commonly used batch size (bs), learning rate (lr), maximum training epoch (ep), and solver (opt) for the three datasets are listed in turn:

[0128] Synapse: bs=8; lr=3e-3; ep=600; opt=SGD;

[0129] ACDC: bs=8; lr=3e-3; ep=200; opt=SGD;

[0130] MoNuSeg: bs=4; lr=1e-3; ep=200; opt=Adam;

[0131] All datasets used momentum = 0.9 and weight decay = 0.0001.

[0132] After determining the parameters of the intra-category dynamic transformer network model and the inter-category dynamic transformer network model, the trained class-aware network model can be integrated into the ResUNet network. The encoder part of ResUNet has 4 convolutional modules composed of basic ResNet-34 blocks, where the number of channels increases with the depth of the network, while the resolution of the feature map decreases accordingly. Given an input image I∈R^(224×224), the ResNet encoder first maps the pixels in I into nonlinear features X∈R^(28×28×512) and inputs them into the class-aware network model module to explore class-aware dependencies. The enhanced features are further sent to the decoder for progressive upsampling. Finally, a segmentation layer is passed to obtain the final segmentation result.

[0133] When testing the image segmentation method you provide, refer to Figure 8 and Figure 9 , Figure 8 Schematic diagram of the comparison between the image segmentation method in the embodiment of the present invention and related technologies in the Synapse dataset. Figure 9 Schematic diagram of the comparison between the image segmentation method in the embodiment of the present invention and related technologies in the ACDC dataset and the MoNuSeg dataset, wherein: Figure 8 and Figure 9The ClassFormer shown in is the test result of the image segmentation method of this application. The Dice Coefficient (DSC index) and Hausdorff Distance (HD index) are used on the Synapse dataset, and the DSC indexes of 8 abdominal organs are listed respectively. The DSC index is used on the ACDC dataset, and the DSC indexes of the left ventricle (LV), right ventricle (RV), and myocardium (MYO) are listed respectively. The DSC index and Intersection over Union (IoU index) are used on the MoNuSeg dataset. Figure 8 and Figure 9 It can be seen that the image segmentation method implemented by the class perception network model in this application always outperforms the test results of all related technologies and significantly improves the performance on all data sets. Figure 8 Among the parameters shown, the image segmentation method implemented by the perceptual network model in this application reduces the number of parameters by 76.15M compared to the previous best method ScaleFormer, but improves the accuracy by DSC: +1.54 and HD: -4.59mm.

[0134] refer to Figure 10 、 Figure 11 and Figure 12 , Figure 10 Schematic diagram showing comparison of the segmentation effect of the image segmentation method in an embodiment of the present invention and the related art in the Synapse dataset test. Figure 11 Schematic diagram showing comparison of the segmentation effect of the image segmentation method in an embodiment of the present invention and the related art in the ACDC dataset test. Figure 12 FIG. 1 is a schematic diagram showing a comparison of the segmentation effects of the image segmentation method in an embodiment of the present invention and the related art in the MoNuSeg dataset. Figure 10-12 As shown by the segmentation results of the three test data sets, the image segmentation method provided by this application can accurately segment significant organs of varying sizes, shapes, and locations. Furthermore, the boundaries of the segmentation results are clearer than those of other methods, reducing the probability of false segmentation and making it easier for users to accurately segment complex medical images.

[0135] In order to better illustrate the working process of the image segmentation method, the image segmentation method provided by the present invention is described below by taking internal bleeding such as cerebral hemorrhage and fundus hemorrhage of the target object as an example.

[0136] Figure 13 Schematic diagram of the use scenario of the image segmentation method provided by the embodiment of the present invention, see Figure 13 , Figure 13This is a schematic diagram of an application scenario of the vascular image processing system 10 provided in an embodiment of the present invention. The terminal 200 can be located in various institutions with medical attributes (such as hospitals, medical research institutes), and can be used to collect (for example, the image collection device of the terminal 200, or through other image collection devices 400) the patient's fundus image (i.e., the vascular image to be processed).

[0137] In some embodiments, the terminal 200 locally executes the vascular image processing method provided by the embodiment of the present invention to complete vascular segmentation and vascular classification of the fundus image, and outputs the results of the vascular segmentation and vascular classification in a graphical manner, so as to provide doctors and researchers with the ability to diagnose, follow up, and study treatment methods for diseases. For example, based on the vascular segmentation results and vascular classification results of the fundus image, the morphological manifestations of different types of blood vessels can be determined, thereby assisting or directly diagnosing whether the patient is at risk of cardiovascular and cerebrovascular diseases or hypertensive retinopathy.

[0138] The terminal 200 can also send fundus images to the server 100 through the network 300 and call the remote diagnosis service function provided by the server 100. The server 100 performs the multi-task of vascular segmentation and vascular classification through the vascular image processing method provided by the embodiment of the present invention, and returns the results of vascular segmentation and vascular classification to the terminal 200 for doctors and researchers to diagnose, follow up and study treatment methods of the disease.

[0139] The terminal 200 may display various intermediate results and final results of the blood vessel image processing in the graphical interface 210 , such as the fundus image, the segmentation result and the classification result of the fundus blood vessels, and the like.

[0140] Continuing to explain the structure of the blood vessel image processing device provided by the embodiment of the present invention, the blood vessel image processing device can be various terminals, such as medical diagnostic equipment, computers, etc., or can be such as Figure 1 Server 100 is shown.

[0141] The image segmentation method provided by the present invention is described below by taking the medical information of cerebral hemorrhage cases as an example, wherein various images forming medical images are continuously generated, for example, as CT plain scan images and MRI images are continuously captured and generated, they become a large amount of data, which urgently needs to be implemented with the help of classification prediction execution to achieve large-scale classification and recognition.

[0142] refer to Figure 14 , Figure 14 An optional flowchart of the image segmentation method provided in an embodiment of the present invention, wherein the user may be a doctor and the target object is a patient, specifically comprising the following steps:

[0143] Step 1401: Acquire a feature image of blood vessels to be segmented of a target object.

[0144] Figure 15 An optional schematic diagram of the image processing method provided by the embodiment of the present invention, the foreground A (eg Figure 10 The medical terminal 400 shown in FIG receives image data (e.g., the medical image to be processed by the user in the previous embodiment), and then processes the image data through pre-processing algorithms, including but not limited to data augmentation such as translation, rotation, symmetry, and segmentation algorithms for selected organs, and then uploads the data to the backend (e.g., Figure 10 The server 100 shown in FIG2 is used, and the background uses the image processing method provided by the present application to segment the medical image by the image segmentation method, and then outputs it to the front-end B (for example, Figure 10 The doctor can clearly observe the segmented medical image through the display device at the front desk B (terminal 200 shown in FIG).

[0145] Step 1402: Extract intra-category image features corresponding to the blood vessel feature image.

[0146] Among them, when segmenting vascular feature images, the morphology of tumor microvessels is extremely diverse, and multiple signaling pathways are involved in regulating the growth of microvessels. Current medical research shows that the different morphologies and densities of microvessels in patient tumor tissue are related to the patient's prognosis and responsiveness to treatment. Therefore, when performing image segmentation, separating microvessels from mature blood vessels, distinguishing different microvessel morphologies from each other, reducing intra-class variations, and increasing inter-class variations can significantly improve the accuracy of image segmentation.

[0147] Step 1403: Obtain a block sequence of the blood vessel feature image through convolution calculation, perform pixel set calculation based on the image features within the category, and obtain the category center feature of the blood vessel feature image.

[0148] Step 1404: interacting the block sequence and the category center feature to obtain enhanced category center feature, and concatenating it with the intra-category image feature to obtain the inter-category image feature.

[0149] Because the image segmentation method provided in this application needs to segment different complex images, it is necessary to set up the class-aware network model as a plug-and-play module and integrate the class-aware network model into the ResUNet network to segment complex vascular images. Different vascular image segmentation environments have different segmentation requirements and extraction scales. Therefore, in order to increase the versatility of the image segmentation method of this application and adapt to different vascular image segmentation requirements, it is possible to determine the information extracted at different scales based on the vascular image segmentation requirements; determine the total number of scales based on the information extracted at different scales; and determine each sliding window that matches the target object based on the total number of scales to obtain accurate inter-category image features for different types of vascular images.

[0150] Step 1405: Segment the blood vessel feature image using the inter-category image features to obtain the segmentation result of the blood vessel feature image.

[0151] in, Figure 16 The present invention is a front-end schematic diagram of segmenting a medical image using an image segmentation method in an embodiment of the present invention; wherein, when displaying a medical image in medical information, the segmented area of ​​the medical image in the display user interface is locked by the control component; the medical image is segmented using the image segmentation method to achieve segmentation of the medical image; and the segmentation results of the CT image and angiography image corresponding to the target object's internal bleeding such as cerebral hemorrhage, fundus hemorrhage and pulmonary hemorrhage are presented in the display interface through the user interface.

[0152] In order to implement the image segmentation method provided by the present application, the present application also provides corresponding hardware equipment. The structure of the image segmentation device of the embodiment of the present invention is described in detail below. The image segmentation device can be implemented in various forms, such as a dedicated terminal with a logic rule network training processing function, or a server with an image segmentation device processing function, such as the preamble. Figure 1 Server 200 in. Figure 17 The schematic diagram of the structure of the image segmentation device provided in the embodiment of the present invention can be understood as follows: Figure 17 Only an exemplary structure of the image segmentation device is shown, not all structures, and can be implemented as needed. Figure 17 Partial or complete structure shown.

[0153] The image segmentation device provided in the embodiment of the present invention includes: at least one processor 201, a memory 202, a user interface 203 and at least one network interface 204. The various components in the image segmentation device are coupled together via a bus system 205. It is understood that the bus system 205 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 205 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 205 is not described in detail. Figure 17 Various buses are labeled as bus system 205.

[0154] The user interface 203 may include a display, a keyboard, a mouse, a trackball, a click wheel, keys, buttons, a touch pad or a touch screen.

[0155] It is understood that the memory 202 can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The memory 202 in the embodiment of the present invention can store data to support the operation of the terminal (such as 10-1). Examples of such data include: any computer program used to operate on the terminal (such as 10-1), such as an operating system and an application program. Among them, the operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application program can include various application programs.

[0156] In some embodiments, the image segmentation device provided by the embodiments of the present invention can be implemented in a combination of software and hardware. As an example, the image segmentation device provided by the embodiments of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the image segmentation method provided by the embodiments of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0157] As an example of an image segmentation device provided by an embodiment of the present invention being implemented by a combination of software and hardware, the image segmentation device provided by an embodiment of the present invention can be directly embodied as a combination of software modules executed by a processor 201. The software module can be located in a storage medium, and the storage medium is located in the memory 202. The processor 201 reads the executable instructions included in the software module in the memory 202, and combines with the necessary hardware (for example, including the processor 201 and other components connected to the bus 205) to complete the image segmentation method provided by the embodiment of the present invention.

[0158] As an example, the processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0159] As an example of hardware implementation of the image segmentation device provided in an embodiment of the present invention, the device provided in an embodiment of the present invention can be directly executed by a processor 201 in the form of a hardware decoding processor, for example, one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the image segmentation method provided in an embodiment of the present invention.

[0160] The memory 202 in the embodiment of the present invention is used to store various types of data to support the operation of the image segmentation apparatus. Examples of such data include any executable instructions for operating on the image segmentation apparatus, such as executable instructions. A program implementing the image segmentation method according to the embodiment of the present invention may be included in the executable instructions.

[0161] In other embodiments, the image segmentation device provided by the embodiment of the present invention can be implemented in software. Figure 17 The image segmentation device stored in the memory 202 is shown. The image segmentation device may be software in the form of a program or plug-in, and may include a series of modules. As an example of a program stored in the memory 202, the image segmentation device may include the following software modules:

[0162] Information transmission module 2081 and information processing module 2082. When the software modules in the image segmentation device are read into RAM and executed by the processor 201, the image segmentation method provided by the embodiment of the present invention will be implemented. The functions of each software module in the image segmentation device include:

[0163] Information transmission module 2081, used to obtain a characteristic image of a target object;

[0164] Information processing module 2082, used to extract intra-category image features corresponding to the feature image;

[0165] The information processing module 2082 is used to obtain a block sequence of feature images through convolution calculation based on the image features within the category;

[0166] Information processing module 2082, used to perform pixel set calculation based on the image features within the category to obtain the category center features of the feature image;

[0167] Information processing module 2082, used for interacting the block sequence and the category center feature to obtain an enhanced category center feature;

[0168] An information processing module 2082 is used to combine the enhanced category center features and the intra-category image features to obtain inter-category image features;

[0169] The information processing module 2082 is used to segment the feature image using the inter-category image features to obtain a segmentation result of the feature image of the target object.

[0170] according to Figure 17 In one aspect of the electronic device shown, the present application further provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of the electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform different embodiments and combinations of embodiments provided in various optional implementations of the above-mentioned image segmentation method.

[0171] The present invention has the following beneficial technical effects:

[0172] 1) The embodiment of the present invention obtains a feature image of a target object; extracts intra-category image features corresponding to the feature image; obtains a block sequence of the feature image through convolution calculation based on the intra-category image features; performs pixel set calculation based on the intra-category image features to obtain the category center features of the feature image; interacts the block sequence and the category center features to obtain enhanced category center features; splices the enhanced category center features and the intra-category image features to obtain inter-category image features; and segments the feature image through the inter-category image features to obtain the segmentation results of the feature image of the target object. In this way, the characteristics of the intra-category image features and the inter-category image features can be fully learned, the problem of insufficient intra-category consistency and insufficient inter-category difference can be solved, and the intra-category feature consistency can be achieved. At the same time, the feature distinction between different categories is strengthened through inter-category constraints, so that the segmentation results of the feature image of the target object are more accurate, the segmentation boundaries are clearer, and the image segmentation environment with various complex image backgrounds can be adapted.

[0173] 2) Through the processing of the intra-category dynamic transformer network model, by decoupling the representations of different categories and adaptively selecting prominent keys / values ​​from multiple scales, compact learning can be achieved to obtain more compact intra-category features. It can also provide more representative initial intra-category center features.

[0174] 3) Through the processing of the inter-category dynamic converter network model, the dependencies between different categories can be obtained, fully mining the category information and improving the accuracy of complex image segmentation. The above is only an embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. An image segmentation method, characterized in that: The method comprises: Obtaining a feature image of the target object; Extracting intra-category image features corresponding to the feature image; According to the image features within the category, a block sequence of the feature image is obtained through convolution calculation; Perform pixel set calculation based on the image features within the category to obtain the category center features of the feature image; Splicing the block sequence and the category center feature to obtain a splicing feature; Performing feature complementation on the splicing features through an attention matrix to obtain complementary splicing features, wherein the attention matrix includes a normalized similarity matrix from category to category, a normalized similarity matrix from block to block, a normalized similarity matrix from category to block, and a normalized similarity matrix from block to category; Splitting the complementary splicing features according to the splicing order to obtain enhanced category center features; splicing the enhanced category center feature and the intra-category image feature to obtain an inter-category image feature; The feature image is segmented using the inter-category image features to obtain a segmentation result of the feature image of the target object.

2. The method according to claim 1, characterized in that The extracting the intra-category image features corresponding to the feature image includes: Performing iterative and decoupling processing on the feature image to obtain a decoupled feature image set; Calculating query features corresponding to the target category feature images in the decoupled feature image set; performing weighted processing on the query features to obtain a saliency perception map; Extracting salient features of the target category feature image according to the saliency perception map; The intra-category image features are calculated based on the salient features of the target category feature image.

3. The method according to claim 2, characterized in that The calculating the intra-category image features according to the salient features of the target category feature image includes: Obtaining a sliding window that matches the target object; Dividing the saliency perception map by the sliding window to obtain at least two non-overlapping sub-regions; determining a significant location in each of the sub-regions; Extracting salient features in the target category feature image according to the salient position; Projecting the significant features to obtain key values ​​and value items; By using the key value and the value item through a multi-head attention mechanism, features of enhanced sequences of different categories are calculated; The intra-category image features are calculated according to the features of the enhanced sequences of different categories.

4. The method according to claim 3, characterized in that The method further comprises: Determining information to be extracted at different scales according to image segmentation requirements of the target object; determining the total number of scales based on the information extracted at the different scales; Each sliding window matching the target object is determined according to the total number of scales.

5. The method according to claim 1, characterized in that The step of obtaining a block sequence of the feature image by performing convolution calculation based on the image features within the category includes: Determining a convolution step size that matches the image features within the category; According to the convolution step size, convolution processing is performed on the image features within the category to obtain a block sequence of the feature image.

6. The method according to claim 1, characterized in that The pixel set calculation is performed based on the image features within the category to obtain the category center features of the feature image, including: Calculating an initial segmentation result for each category based on the image features within the category; Calculating a pixel representation set of a target category feature image based on the initial segmentation result; According to the pixel representation set of the target category feature image, a category center feature of the feature image is determined.

7. The method according to claim 1, characterized in that The method is implemented based on a class-aware network model, which includes at least an intra-class dynamic converter network model and an inter-class dynamic converter network model. Acquiring a feature image of a target object includes: Acquire a feature image of the target object through the intra-category dynamic transformer network model; The extracting the intra-category image features corresponding to the feature image includes: Extracting intra-category image features corresponding to the feature image through the intra-category dynamic transformer network model; The step of obtaining a block sequence of the feature image by performing convolution calculation based on the image features within the category includes: Obtaining a block sequence of the feature image by performing convolution calculation based on the intra-category image features through the inter-category dynamic converter network model; The pixel set calculation is performed based on the image features within the category to obtain the category center features of the feature image, including: Performing pixel set calculation based on the intra-category image features through the inter-category dynamic converter network model to obtain the category center features of the feature image; The interacting the block sequence and the category center feature to obtain an enhanced category center feature includes: Interacting the block sequence and the category center feature through the inter-category dynamic transformer network model to obtain an enhanced category center feature; The step of splicing the enhanced category center feature and the intra-category image feature to obtain the inter-category image feature includes: The enhanced category center features and the intra-category image features are spliced ​​through the inter-category dynamic transformer network model to obtain inter-category image features.

8. An image segmentation device, characterized in that: The device comprises: An information transmission module is used to obtain a characteristic image of a target object; An information processing module, configured to extract intra-category image features corresponding to the feature image; The information processing module is used to obtain a block sequence of the feature image through convolution calculation according to the image features within the category; The information processing module is used to perform pixel set calculation based on the image features within the category to obtain the category center features of the feature image; The information processing module is configured to concatenate the block sequence and the category center feature to obtain a concatenated feature; perform feature complementation on the concatenated feature through an attention matrix to obtain a complementary concatenated feature, wherein the attention matrix includes a normalized similarity matrix from category to category, a normalized similarity matrix from block to block, a normalized similarity matrix from category to block, and a normalized similarity matrix from block to category; and split the complementary concatenated feature according to the concatenation order to obtain an enhanced category center feature; The information processing module is used to combine the enhanced category center features and the intra-category image features to obtain inter-category image features; The information processing module is used to segment the feature image using the inter-category image features to obtain a segmentation result of the feature image of the target object.

9. The device according to claim 8, characterized in that The information processing module is further configured to iteratively and decouple the feature images to obtain a set of decoupled feature images; calculate query features corresponding to target category feature images in the set of decoupled feature images; perform weighted processing on the query features to obtain a saliency perception map; extract salient features of the target category feature images based on the saliency perception map; and calculate intra-category image features based on the salient features of the target category feature images.

10. The device according to claim 9, characterized in that The information processing module is further used to obtain a sliding window that matches the target object; divide the saliency perception map through the sliding window to obtain at least two non-overlapping sub-regions; determine a salient position in each of the sub-regions; extract salient features from the target category feature image based on the salient position; project the salient features to obtain key values ​​and value items; calculate the features of enhanced sequences of different categories using the key values ​​and the value items through a multi-head attention mechanism; and calculate the image features within the category based on the features of the enhanced sequences of different categories.

11. An electronic device, characterized in that: The electronic device comprises: a memory for storing executable instructions; A processor, configured to implement the image segmentation method according to any one of claims 1 to 7 when running the executable instructions stored in the memory.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the image segmentation method according to any one of claims 1 to 7 is implemented.

13. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the image segmentation method according to any one of claims 1 to 7 is implemented.

Images