Image semantic segmentation method, device, equipment, storage medium and program product
By calculating the feature similarity value of the target image and the reference image as the weighted value, the problem of image semantic segmentation information loss in small sample scenes is solved, and a more accurate image semantic segmentation effect is achieved.
Patent Information
- Application Number
- CN202210267787.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-17
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-03-17
AI Technical Summary
In small sample scenarios, in the prior art, the reconstruction of the target image by random sampling will lead to information loss, resulting in poor image semantic segmentation effect.
By obtaining the features and mask of the target image and reference image, the similarity value between the features is calculated and used as a weighted value and reference mask to obtain the mask of the target image to achieve more accurate image semantic segmentation .
It improves the accuracy of image semantic segmentation, reduces information loss, and obtains more accurate segmentation results.
Smart Images

Figure CN114612902B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image semantic segmentation method, apparatus, device, storage medium and program product. Background Art
[0002] Image semantic segmentation is a technique that distinguishes different objects contained in an image and identifies the categories to which each object belongs.
[0003] In the related art, the target image is reconstructed in the reference image by utilizing the similarity between the target image and the reference image, that is, the target image is processed according to the similarity between the target image and the reference image, so as to obtain the final image semantic segmentation result based on the reconstructed target image.
[0004] However, in small sample scenarios, the related art randomly samples the reference image when reconstructing the target image, which may lead to potential information loss, resulting in poor semantic segmentation effect of the final image. Summary of the invention
[0005] The present application provides an image semantic segmentation method, apparatus, device, storage medium and program product, which can obtain relatively accurate image semantic segmentation results. The technical solution is as follows:
[0006] According to one aspect of the present application, a method for image semantic segmentation is provided, the method comprising:
[0007] Acquire a target feature corresponding to a target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image, wherein the reference image is an image that belongs to the same category as the target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image;
[0008] Inputting the target feature, the reference feature and the reference mask into an image semantic segmentation model to calculate a similarity value between the target feature and the reference feature;
[0009] In the image semantic segmentation model, the similarity value is used as a weighted value to be calculated with the reference mask to obtain a target mask corresponding to the target image, and the target mask is used to indicate different semantic areas after image semantic segmentation is performed on the target image.
[0010] According to one aspect of the present application, a method for training an image semantic segmentation model is provided, the method comprising:
[0011] Obtaining a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask, wherein the reference image is an image that belongs to the same category as the target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image;
[0012] The target feature, the reference feature and the reference mask are input into the image semantic segmentation model to calculate the similarity value between the target feature and the reference feature
[0013] In the image semantic segmentation model, the similarity value is used as a weighted value to calculate with the reference mask to obtain a predicted target mask corresponding to the sample target image, wherein the predicted target mask is used to indicate different semantic areas after image semantic segmentation is performed on the sample target image;
[0014] Calculating a loss function value based on the predicted target mask and the true target mask;
[0015] The model parameters of the image semantic segmentation model are updated based on the loss function value.
[0016] According to one aspect of the present application, a device for image semantic segmentation is provided, the device comprising:
[0017] an acquisition module, used to acquire a target feature corresponding to a target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image, wherein the reference image is an image that belongs to the same category as the target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image;
[0018] A calculation module, used for inputting the target feature, the reference feature and the reference mask into an image semantic segmentation model to calculate a similarity value between the target feature and the reference feature;
[0019] The calculation module is also used to calculate the similarity value as a weighted value with the reference mask in the image semantic segmentation model to obtain a target mask corresponding to the target image, and the target mask is used to indicate different semantic areas after image semantic segmentation of the target image.
[0020] According to one aspect of the present application, a training device for an image semantic segmentation model is provided, the device comprising:
[0021] An acquisition module, used to acquire a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask, wherein the reference image is an image that belongs to the same category as the sample target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image;
[0022] A first calculation module, used for inputting the target feature, the reference feature and the reference mask into the image semantic segmentation model to calculate the similarity value between the target feature and the reference feature;
[0023] The first calculation module is further used to calculate the similarity value as a weighted value with the reference mask in the image semantic segmentation model to obtain a predicted target mask corresponding to the sample target image, wherein the predicted target mask is used to indicate different semantic areas after image semantic segmentation is performed on the sample target image;
[0024] A second calculation module, used for calculating a loss function value based on the predicted target mask and the real target mask;
[0025] An updating module is used to update the model parameters of the image semantic segmentation model based on the loss function value.
[0026] According to another aspect of the present application, a computer device is provided, which includes: a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the image semantic segmentation method described above or the image semantic segmentation model training method described above.
[0027] According to another aspect of the present application, a computer storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the image semantic segmentation method or the image semantic segmentation model training method as described above.
[0028] According to another aspect of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes the image semantic segmentation method or the image semantic segmentation model training method as described above.
[0029] The beneficial effects of the technical solution provided by this application include at least:
[0030] By obtaining the target features corresponding to the target image, the reference features corresponding to at least one reference image, and the reference mask corresponding to the reference image; the target features, the reference features, and the reference mask are input into the image semantic segmentation model to calculate the similarity value between the target features and the reference features, and the similarity value is used as a weighted value to calculate with the corresponding reference mask to obtain the target mask corresponding to the target image. The present application obtains the target mask corresponding to the target image by calculating the similarity value between all features of the target image and the reference image, and using the similarity value as a weighted value of the reference mask. Based on the above method, a relatively accurate image semantic segmentation result can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0032] Figure 1 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0033] Figure 2 is a schematic diagram of the architecture of a computer system provided by an exemplary embodiment of the present application;
[0034] Figure 3 is a flow chart of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0035] Figure 4 is a flow chart of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0036] Figure 5 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0037] Figure 6 is a flow chart of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0038] Figure 7 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0039] Figure 8 is a flow chart of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0040] Fig. 9 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0041] Fig.10 It is a schematic diagram comparing the training efficiency of an image semantic segmentation method provided by an exemplary embodiment of the present application and a comparative solution;
[0042] Fig.11 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0043] Fig.12 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application;
[0044] Fig.13 It is a framework diagram of image semantic segmentation model generation and image semantic segmentation provided by an exemplary embodiment of the present application;
[0045] Fig.14 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application;
[0046] Fig.15 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application;
[0047] Fig.16 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application;
[0048] Fig.17 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application;
[0049] Fig.18 is a block diagram of an image semantic segmentation device provided by an exemplary embodiment of the present application;
[0050] Fig.19 is a block diagram of a training device for an image semantic segmentation model provided by an exemplary embodiment of the present application;
[0051] Fig. 20 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0052] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0053] The embodiment of the present application provides an image semantic segmentation method. For ease of understanding, several terms involved in the present application are explained below.
[0054] 1) Artificial Intelligence (AI)
[0055] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0056] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The display device containing an image acquisition component shown in this application mainly involves computer vision technology, machine learning / deep learning, autonomous driving, smart transportation, and other directions.
[0057] 2) Machine Learning (ML)
[0058] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0059] 3) Computer Vision (CV)
[0060] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image semantic segmentation, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0061] 4) Self Attention Mechanism
[0062] The role of the self-attention mechanism is to better learn the dependencies between global features. Self-attention directly calculates the relationship between any two nodes in the graph structure and obtains the global geometric features of the graph structure in one step.
[0063] Self-attention uses the attention mechanism and is calculated in three stages: (1) Introducing different functions and calculation mechanisms to calculate the similarity or correlation between the query and a key. The most common methods include: finding the dot product of their vectors, finding the similarity of their vectors, or introducing an additional neural network to evaluate. (2) Introducing a calculation method similar to softmax to convert the scores of the first stage into numerical values. On the one hand, it can be normalized to organize the original calculated scores into a probability distribution in which the sum of the weights of all elements is 1. On the other hand, the inherent mechanism of softmax can be used to highlight the weights of important elements. (3) The calculation result of the second stage is the corresponding weight coefficient, which is then weighted summed to obtain the attention value.
[0064] 5) Few-shot learning is a branch of machine learning that aims to solve machine learning tasks with limited data. In few-shot learning, there are fewer samples available, and the use of traditional machine learning methods can easily cause overfitting and reduce the generalization performance of the model algorithm. Few-shot learning hopes to use as few samples as possible while ensuring good performance.
[0065] The present application embodiment provides a technical solution for an image semantic segmentation method, such as Figure 1 As shown, the method can be executed by a computer device, and the computer device can be a terminal or a server.
[0066] Exemplarily, the computer device obtains a target image 101, a reference image 102, and a reference mask 103 corresponding to the reference image 102; and extracts features from the target image 101 through a feature extraction network 104 to obtain a multi-scale target feature 106; and extracts features from the reference image 102 through the feature extraction network 104 to obtain a reference feature 107 having the same scale as the target feature 106. The computer device obtains a reference mask 103 of a corresponding scale by downsampling 105 the reference mask 103.
[0067] For example, four scale target features 106 corresponding to the target image 101 are obtained, and the four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, but are not limited to this. The embodiments of the present application are not limited to this.
[0068] The computer device inputs the target features 106, reference features 107 and reference masks 103 corresponding to different scales into a dense cross-query-and-support Attention weighted Mask Aggregation (DCAMA) network 108 for calculation to obtain a similarity value between the target features 106 and the reference features 107; in the multi-layer dense cross-attention weighted mask aggregation network 108, the similarity value is used as a weighted value of the reference mask 103 to calculate a target mask 113 corresponding to the target image 101.
[0069] For example, the computer device inputs the target feature 106 corresponding to the 1 / 32 resolution, the reference feature 107 corresponding to the 1 / 32 resolution, and the reference mask 103 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 108 for calculation, and obtains the similarity value between the target feature 106 corresponding to the 1 / 32 resolution and the reference feature 107 corresponding to the 1 / 32 resolution; in the multi-layer dense cross-attention weighted mask aggregation network 108, the similarity value is used as the weighted value of the reference mask 103 to calculate the first target mask corresponding to the 1 / 32 resolution.
[0070] After calculating the target masks corresponding to different scales, the computer device aggregates the target masks corresponding to different scales to obtain an intermediate target mask.
[0071] For example, the computer device inputs a first target mask corresponding to a resolution of 1 / 32 into the convolution network 109 for three convolution calculations, and upsamples 110 the convolution result of the target mask corresponding to the resolution of 1 / 32 to obtain a first intermediate target mask; the computer device inputs a second target mask corresponding to a resolution of 1 / 16 into the convolution network 109 for three convolution calculations, adds the convolution result of the target mask corresponding to the resolution of 1 / 16 to the first intermediate target mask element by element, and performs three convolution calculations and upsamples 110 on the result of the element-by-element addition to obtain a second intermediate target mask; the above steps are repeated to add the convolution result of the second intermediate target mask with the third target mask corresponding to the resolution of 1 / 8 element by element, and perform three convolution calculations on the result of the element-by-element addition to finally obtain the intermediate target mask.
[0072] The computer device performs three convolution calculations on the intermediate target mask, performs residual connection 111 on the convolution result of the intermediate target mask and the target feature 106 corresponding to 1 / 8 resolution to obtain a first residual connection result, upsamples 110 the first residual connection result and performs residual connection 111 again with the target feature 106 corresponding to 1 / 4 resolution to obtain a second residual connection result, inputs the second residual connection result into the hybrid network 112 for fusion, and finally obtains a target mask 113 corresponding to the target image 101.
[0073] In summary, the method provided in this embodiment obtains multi-scale target features corresponding to the target image, reference features corresponding to the reference image, and a reference mask; inputs the target features, reference features, and reference masks corresponding to different scales into the image semantic segmentation model for calculation to obtain target masks corresponding to different scales; aggregates the target masks corresponding to different scales to obtain an intermediate target mask; and fuses the intermediate target mask with the target features to obtain a target mask corresponding to the target image. The present application obtains the target mask corresponding to the target image by calculating the similarity value between all features of the target image and the reference image, and using the similarity value as the weighted value of the reference mask. Based on the above method, a more accurate image semantic segmentation result can be obtained.
[0074] Figure 2 The schematic diagram of the architecture of a computer system provided by an embodiment of the present application is shown. The computer system may include: a terminal 100 and a server 200.
[0075] The terminal 100 may be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a personal computer (PC), an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, an unmanned vending terminal, etc. The terminal 100 may be installed with a client that runs a target application, and the target application may be an application for reference image semantic segmentation, or may be other applications that provide image semantic segmentation functions, which is not limited in this application. In addition, this application does not limit the form of the target application, including but not limited to an application (Application, App) installed in the terminal 100, a mini-program, etc., and may also be in the form of a web page.
[0076] The server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, cloud database, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server 200 may be a backend server of the above-mentioned target application, used to provide backend services for the client of the target application.
[0077] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.
[0078] In some embodiments, the above-mentioned server can also be implemented as a node in a blockchain system. Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.
[0079] The terminal 100 and the server 200 may communicate with each other via a network, such as a wired or wireless network.
[0080] In the image semantic segmentation method or the image semantic segmentation model training method provided in the embodiments of the present application, the execution subject of each step may be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 2 Taking the implementation environment of the scheme shown as an example, the image semantic segmentation method or the image semantic segmentation model training method can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executes the image semantic segmentation method or the image semantic segmentation model training method), or the image semantic segmentation method or the image semantic segmentation model training method can be executed by the server 200, or the terminal 100 and the server 200 can interact and cooperate to execute it, and this application does not limit this.
[0081] Figure 3 is a flowchart of an image semantic segmentation method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0082] Step 302: Obtain a target feature corresponding to the target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image.
[0083] Exemplarily, the target image refers to an image to be subjected to image semantic segmentation, and the target image is also called a query image. The target feature refers to a feature representation obtained by extracting features from the target image.
[0084] Image semantic segmentation refers to the process of identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs. For example, all pixels related to trees in a picture are classified into one category, and all pixels related to cars in the picture are classified into another category.
[0085] The reference image is an image that belongs to the same category as the target image and has been subjected to image semantic segmentation. The reference image is also called a support image. The reference image is an image that belongs to the same category as the target image to be subjected to image semantic segmentation. The number of reference images is selected to be at least one, but is not limited thereto. The embodiment of the present application does not specifically limit the number of reference images.
[0086] For example, for the airplane A in the semantic segmentation target image, one or n images of the same category that have been semantically segmented are selected as reference images. The reference image contains airplane B, and the airplane B has been semantically segmented. The area corresponding to the airplane B is the activated part of the reference mask.
[0087] The reference mask is used to indicate different semantic regions after image semantic segmentation of the reference image.
[0088] For example, if image semantic segmentation has been performed on airplane B in the reference image, the mask value of the reference mask corresponding to the area corresponding to airplane B in the reference image is 1, and the mask value of the reference mask corresponding to the remaining areas of the reference image is 0.
[0089] The reference feature refers to a feature representation obtained by extracting features from a reference image. The reference feature corresponding to at least one reference image refers to a feature obtained by extracting features from each reference image individually.
[0090] It can be understood that in the present application, performing feature extraction on at least one reference image means performing separate feature extraction on each reference image. For example, performing feature extraction on three reference images, and finally obtaining three reference features corresponding to the three reference images.
[0091] Step 304: Input the target feature, reference feature and reference mask into the image semantic segmentation model to calculate the similarity value between the target feature and the reference feature, and use the similarity value as a weighted value to calculate with the reference mask to obtain the target mask corresponding to the target image.
[0092] The target mask is used to indicate different semantic regions after image semantic segmentation of the target image.
[0093] The image semantic segmentation model refers to a model used to segment an image into regional blocks with certain semantic meanings.
[0094] The image semantic segmentation model calculates the similarity value between the target feature and all reference features based on the target feature of the target image, the reference feature of the reference image and the reference mask, and finally obtains the target mask corresponding to the area to be semantically segmented in the target image by calculating the similarity value between the target feature and all reference features, and calculating the similarity value with the reference mask.
[0095] Step 306: Output the target mask corresponding to the target image.
[0096] Exemplarily, the computer device outputs a target mask corresponding to the target image.
[0097] Optionally, the target mask is a mask value of 1 for the area to be segmented corresponding to the target image, and the rest is 0, or the mask value of 0 for the area to be segmented corresponding to the target image, and the rest is 1, but is not limited to this and is not limited to this in the embodiments of the present application.
[0098] For example, if the part to be semantically segmented in the target image is airplane A, the mask value of the target mask corresponding to the airplane A area is 1, and the rest of the target image is 0; or, the mask value of the target mask corresponding to the airplane A area is 0, and the rest of the target image is 1.
[0099] In summary, the method provided in this embodiment obtains the target features corresponding to the target image, the reference features corresponding to the reference image, and the reference mask; the target features, reference features, and reference masks are input into the image semantic segmentation model to calculate the similarity value between the target features and the reference features, and the similarity value is used as a weighted value to calculate with the reference mask to obtain the target mask corresponding to the target image. This application calculates the target features, reference features, and reference masks using the image semantic segmentation model to obtain the target mask corresponding to the target image. Based on the above method, a relatively accurate image semantic segmentation result can be obtained.
[0100] An embodiment of the present application provides an image semantic segmentation model, which includes: a feature extraction network and a multi-layer dense cross-attention weighted mask aggregation network.
[0101] The computer device acquires a target image and a reference image, and extracts features from the target image through a feature extraction network to obtain target features, and extracts features from the reference image through a feature extraction network to obtain reference features.
[0102] The computer device inputs the acquired target features, reference features and reference masks corresponding to the reference features into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain a target mask corresponding to the target image.
[0103] Based on the image semantic segmentation model, the following image semantic segmentation method is provided.
[0104] Figure 4 is a flowchart of an image semantic segmentation method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0105] Step 402: Obtain a target feature corresponding to a target image, a reference feature corresponding to a reference image, and a reference mask corresponding to the reference image.
[0106] Exemplarily, the target image refers to an image to be subjected to image semantic segmentation, and the target feature refers to a feature representation obtained by extracting features from the target image.
[0107] Image semantic segmentation refers to the pixel-level recognition of images and the prediction of the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs. The reference image is an image that belongs to the same category as the target image to be semantically segmented.
[0108] For example, in the medical field, by performing semantic segmentation on medical images and labeling different organs in medical images based on the results of image semantic segmentation, medical staff can be better assisted in medical diagnosis. For example, by performing semantic segmentation on chest X-rays, the areas corresponding to the lungs and heart in the X-rays can be labeled.
[0109] The reference image refers to an image that has been semantically segmented. The reference mask is used to indicate different semantic regions after semantic segmentation of the reference image.
[0110] Reference features refer to feature representations obtained by extracting features from reference images.
[0111] In one possible implementation, the image semantic segmentation model includes a feature extraction network; a computer device obtains a target image and a reference image; the computer device extracts features of the target image through the feature extraction network to obtain target features; and extracts features of the reference image through the feature extraction network to obtain reference features.
[0112] Optionally, the feature extraction network is at least one of a residual neural network (ResNet) and a deep convolutional network VGG, but is not limited thereto and is not limited to this in the embodiments of the present application.
[0113] Step 404: Input the target feature, the reference feature and the reference mask into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain a similarity value between the target feature and the reference feature.
[0114] Exemplarily, the computer device inputs the target feature, the reference feature and the reference mask into a multi-layer dense cross-attention weighted mask aggregation network to perform a scaled dot product attention calculation to obtain a similarity value between the target feature and the reference feature.
[0115] Among them, the target feature and the reference feature are both pixel-level feature representations, and the reference feature includes the feature representations of foreground pixels and background pixels in the reference image.
[0116] Exemplarily, the computer device uses the target feature as the first vector matrix Q (Query), the reference feature as the second vector matrix K (Key), and the reference mask as the third vector matrix V (Value). The computer device inputs the target feature, the reference feature, and the reference mask into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the similarity value between the target feature Q and the reference feature K, that is, to obtain the similarity value between the target feature Q corresponding to any pixel in the target image and the reference feature K corresponding to all pixels in the reference image.
[0117] The similarity calculation formula between the target feature Q and the reference feature K can be expressed as:
[0118] C=QK T
[0119] Among them, C represents the similarity value, Q is the target feature, and K is the reference feature.
[0120] Step 406: In a multi-layer dense cross-attention weighted mask aggregation network, the similarity value is used as a weighted value of the reference mask to calculate a target mask corresponding to the target image.
[0121] The target mask is used to indicate different semantic regions after image semantic segmentation of the target image.
[0122] The computer device multiplies the similarity value with the mask value of the reference mask in a multi-layer dense cross-attention weighted mask aggregation network to calculate a target mask corresponding to the target image.
[0123] The calculation formula of the target mask can be expressed as:
[0124]
[0125] Among them, Attn is the mask value corresponding to each pixel in the target mask, Q is the target feature, K is the reference feature, V is the reference mask, QK T is the similarity value, d is the vector dimension of Query and Key, and softmax is the normalization function.
[0126] like Figure 5 As shown, the computer device obtains a target image 501, a reference image 502 and a reference mask 503 corresponding to the reference image 502; and extracts features from the target image 501 through a feature extraction network 504 to obtain a target feature 505 of a scale; and extracts features from the reference image 502 through the feature extraction network 504 to obtain a reference feature 506 of a scale.
[0127] The computer device inputs the target feature 505, the reference feature 506 and the reference mask 503 into the multi-layer dense cross-attention weighted mask aggregation network 507 for calculation to obtain the similarity value between the target feature 505 and the reference feature 506. For example, the similarity values of the target feature 505 corresponding to the pixel A in the target image 501 and the reference feature 506 corresponding to all pixels in the reference image 502 are calculated respectively; in the multi-layer dense cross-attention weighted mask aggregation network 507, the similarity value is multiplied by the mask value of the reference mask 503 to calculate the target mask 508 corresponding to the target image 501.
[0128] Step 408: Output the target mask corresponding to the target image.
[0129] Exemplarily, the computer device outputs a target mask corresponding to the target image.
[0130] In summary, the method provided in this embodiment obtains a target feature of a scale corresponding to the target image, a reference feature of a scale corresponding to the reference image, and a reference mask; the target feature, reference feature, and reference mask corresponding to the same scale are input into the image semantic segmentation model for calculation to obtain a target mask corresponding to the target image. This application calculates the similarity value between all features of the target image and the reference image, and uses the similarity value as the weighted value of the reference mask to obtain the target mask corresponding to the target image. Based on the above method, a more accurate image semantic segmentation result can be obtained.
[0131] An embodiment of the present application provides an image semantic segmentation model, which includes: a feature extraction network, a multi-layer dense cross-attention weighted mask aggregation network, and a convolutional network.
[0132] The computer device acquires a target image and at least one reference image, and extracts features from the target image through a feature extraction network to obtain target features, and extracts features from the reference image through a feature extraction network to obtain reference features.
[0133] The computer device inputs the acquired target feature, the reference feature, and the reference mask corresponding to the reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain a target mask corresponding to the target image. In the case where the reference feature is a multi-scale reference feature, the acquired target feature, the reference feature, and the reference mask corresponding to the reference feature are input into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain an i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1.
[0134] The computer device inputs the target masks corresponding to different scales into the convolutional network for convolution, and adds the convolution results of the target masks corresponding to different scales element by element to obtain the target mask corresponding to the target image.
[0135] Based on the image semantic segmentation model, the following image semantic segmentation method is provided.
[0136] Figure 6 is a flowchart of an image semantic segmentation method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0137] Step 602: Obtain a target feature corresponding to a target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image.
[0138] Exemplarily, the target image refers to an image to be subjected to image semantic segmentation, and the target feature refers to a feature representation obtained by extracting features from the target image.
[0139] Image semantic segmentation refers to identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs.
[0140] The reference image refers to an image that has been subjected to image semantic segmentation. The reference image is an image that belongs to the same category as the target image to be subjected to image semantic segmentation. The number of reference images is selected to be at least one, but is not limited thereto. The embodiment of the present application does not specifically limit the number of reference images.
[0141] For example, for the airplane A in the semantic segmentation target image, one or n images of the same category that have been semantically segmented are selected as reference images. The reference image contains airplane B, and the airplane B has been semantically segmented. The area corresponding to the airplane B is the activated part of the reference mask.
[0142] The reference mask is used to indicate different semantic regions after image semantic segmentation of the reference image.
[0143] The reference feature refers to a feature representation obtained by extracting features from a reference image. The reference feature corresponding to at least one reference image refers to a feature representation obtained by extracting features from each reference image individually.
[0144] Step 604: Input the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0145] Exemplarily, the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature.
[0146] The scale refers to the resolution corresponding to the target feature. For example, four scale target features corresponding to the target image are obtained, and the four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, but are not limited to this. The embodiments of the present application are not limited to this.
[0147] The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0148] Exemplarily, the computer device uses the target feature as the first vector matrix Q (Query), the reference feature as the second vector matrix K (Key), and the reference mask as the third vector matrix V (Value). The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network to perform a scaled dot product attention calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature. The formula for calculating the i-th similarity value between the i-th scale target feature and the i-th scale reference feature can refer to the formula for calculating the similarity value in step 404, which will not be repeated here.
[0149] For example, for example, Figure 7 As shown, the computer device obtains target features 706 of four scales corresponding to the target image 701, and the four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution respectively.
[0150] The computer device inputs the target feature 706 corresponding to the 1 / 32 resolution, the reference feature 707 corresponding to the 1 / 32 resolution, and the reference mask 103 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 708 for calculation, and obtains the similarity value between the target feature 706 corresponding to the 1 / 32 resolution and the reference feature 707 corresponding to the 1 / 32 resolution.
[0151] The computer device inputs the target feature 706 corresponding to the 1 / 16 resolution, the reference feature 707 corresponding to the 1 / 16 resolution, and the reference mask 703 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 708 for calculation, and obtains the similarity value between the target feature 706 corresponding to the 1 / 16 resolution and the reference feature 707 corresponding to the 1 / 16 resolution.
[0152] The computer device inputs the target feature 706 corresponding to the 1 / 8 resolution, the reference feature 707 corresponding to the 1 / 8 resolution, and the reference mask 703 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 708 for calculation, and obtains the similarity value between the target feature 706 corresponding to the 1 / 8 resolution and the reference feature 707 corresponding to the 1 / 8 resolution.
[0153] Exemplarily, the computer device obtains a reference mask 703 of a corresponding scale by downsampling 705 the reference mask 703 .
[0154] Downsampling 705 refers to reducing the reference mask 703 so that the reference mask 703 matches the corresponding reference feature 707. For example, if the resolution of the reference feature 707 is 1 / 8, the reference mask 703 is reduced by downsampling to a reference mask 703 that matches the reference feature 707 corresponding to the 1 / 8 resolution.
[0155] Step 606: multiply the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature to obtain the i-th target mask corresponding to the i-th scale target feature of the target image.
[0156] The computer device multiplies the i-th similarity value by the mask value of the reference mask matching the i-th scale reference feature in a multi-layer dense cross-attention weighted mask aggregation network to obtain the i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1.
[0157] The calculation formula of the i-th target mask may refer to the formula for calculating the target mask in step 406, and will not be repeated here.
[0158] Exemplarily, when a computer device obtains a similarity value between a target feature corresponding to a 1 / 32 resolution and a reference feature corresponding to a 1 / 32 resolution, the similarity value is multiplied by a mask value of a reference mask that matches the reference feature corresponding to the 1 / 32 resolution to obtain a first target mask corresponding to the target feature corresponding to the 1 / 32 resolution of the target image.
[0159] When the computer device obtains a similarity value between a target feature corresponding to a 1 / 16 resolution and a reference feature corresponding to a 1 / 16 resolution, the similarity value is multiplied by a mask value of a reference mask that matches the reference feature corresponding to the 1 / 16 resolution to obtain a second target mask corresponding to the target feature corresponding to the 1 / 16 resolution of the target image.
[0160] When the computer device obtains a similarity value between a target feature corresponding to a 1 / 8 resolution and a reference feature corresponding to a 1 / 8 resolution, the similarity value is multiplied by a mask value of a reference mask that matches the reference feature corresponding to the 1 / 8 resolution to obtain a third target mask corresponding to the target feature corresponding to the 1 / 8 resolution of the target image.
[0161] Step 608: Aggregate the target masks corresponding to different scales to obtain the target mask corresponding to the target image.
[0162] Exemplarily, the image semantic segmentation model includes a convolutional network; the computer device convolves the target masks corresponding to different scales through the convolutional network, and adds the target masks corresponding to different scales element by element to obtain an intermediate target mask.
[0163] Optionally, the image semantic segmentation model includes three convolutional networks, but is not limited to this, and the embodiments of the present application do not impose any limitations on this.
[0164] For example, Figure 7 As shown, the computer device inputs the first target mask corresponding to the 1 / 32 resolution into the convolution network 709 for three convolution calculations, and upsamples the convolution result of the target mask corresponding to the 1 / 32 resolution 710 to obtain a first intermediate target mask.
[0165] The computer device inputs the second target mask corresponding to the 1 / 16 resolution into the convolution network 709 for three convolution calculations, adds the convolution result of the second target mask corresponding to the 1 / 16 resolution to the first intermediate target mask element by element, performs three convolution calculations on the element by element addition result, and performs upsampling 710 to obtain the second intermediate target mask. Repeat the above steps, add the convolution result of the second intermediate target mask to the third target mask corresponding to the 1 / 8 resolution element by element, perform three convolution calculations on the element by element addition result, and finally obtain the target mask corresponding to the target image.
[0166] Step 610: Output the target mask corresponding to the target image.
[0167] Exemplarily, the computer device outputs a target mask corresponding to the target image.
[0168] In summary, the method provided in this embodiment obtains multi-scale target features corresponding to the target image, reference features corresponding to the reference image, and a reference mask; inputs the target features, reference features, and reference masks corresponding to different scales into the image semantic segmentation model for calculation to obtain target masks corresponding to different scales; aggregates the target masks corresponding to different scales to obtain a target mask corresponding to the target image. This application calculates the similarity value between all features of the target image and the reference image, and uses the similarity value as the weighted value of the reference mask to obtain the target mask corresponding to the target image. Based on the above method, a more accurate image semantic segmentation result can be obtained.
[0169] An embodiment of the present application provides an image semantic segmentation model, which includes: a feature extraction network, a multi-layer dense cross-attention weighted mask aggregation network, a convolutional network, and a hybrid network.
[0170] The computer device acquires a target image and at least one reference image, and extracts features from the target image through a feature extraction network to obtain target features, and extracts features from the reference image through a feature extraction network to obtain reference features.
[0171] The computer device inputs the acquired target feature, the reference feature, and the reference mask corresponding to the reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain a target mask corresponding to the target image. In the case where the reference feature is a multi-scale reference feature, the acquired target feature, the reference feature, and the reference mask corresponding to the reference feature are input into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain an i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1.
[0172] The computer device inputs the target masks corresponding to different scales into the convolutional network for convolution, and adds the convolution results of the target masks corresponding to different scales element by element to obtain an intermediate target mask.
[0173] The computer device performs a residual connection between the obtained intermediate target mask and the target feature corresponding to at least one scale, and inputs the residual connection result of the intermediate target mask and the target feature into the hybrid network for fusion, and finally obtains the target mask corresponding to the target image.
[0174] Based on the image semantic segmentation model, the following image semantic segmentation method is provided.
[0175] Figure 8 is a flowchart of an image semantic segmentation method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0176] Step 802: Obtain a target feature corresponding to a target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image.
[0177] Exemplarily, the target image refers to an image to be subjected to image semantic segmentation, and the target feature refers to a feature representation obtained by extracting features from the target image.
[0178] Image semantic segmentation refers to identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs.
[0179] The reference image refers to an image that has been semantically segmented. The reference mask is used to indicate different semantic regions after semantic segmentation of the reference image.
[0180] The reference feature refers to a feature representation obtained by extracting features from a reference image. The reference feature corresponding to at least one reference image refers to a feature representation obtained by extracting features from each reference image individually.
[0181] In one possible implementation, the image semantic segmentation model includes a feature extraction network; a computer device obtains a target image and a reference image; the computer device extracts features of the target image through the feature extraction network to obtain target features; and extracts features of the reference image through the feature extraction network to obtain reference features.
[0182] Optionally, the feature extraction network is at least one of a residual neural network (ResNet) and a deep convolutional network VGG, but is not limited thereto and is not limited to this in the embodiments of the present application.
[0183] Step 804: input the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into the image semantic segmentation model for calculation to obtain the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0184] Exemplarily, the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature.
[0185] The scale refers to the resolution corresponding to the target feature. For example, four scale target features corresponding to the target image are obtained, and the four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, but are not limited to this. The embodiments of the present application are not limited to this.
[0186] The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into the image semantic segmentation model for calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0187] Exemplarily, the computer device uses the target feature as the first vector matrix Q (Query), the reference feature as the second vector matrix K (Key), and the reference mask as the third vector matrix V (Value). The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network to perform a scaled dot product attention calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature. The formula for calculating the i-th similarity value between the i-th scale target feature and the i-th scale reference feature can refer to the formula for calculating the similarity value in step 404, which will not be repeated here.
[0188] For example, for example, Figure 1 As shown, the computer device obtains target features 106 of four scales corresponding to the target image 101, and the four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution respectively.
[0189] The computer device inputs the target feature 106 corresponding to the 1 / 32 resolution, the reference feature 107 corresponding to the 1 / 32 resolution, and the reference mask 103 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 108 for calculation, and obtains the similarity value between the target feature 106 corresponding to the 1 / 32 resolution and the reference feature 107 corresponding to the 1 / 32 resolution.
[0190] The computer device inputs the target feature 106 corresponding to the 1 / 16 resolution, the reference feature 107 corresponding to the 1 / 16 resolution, and the reference mask 103 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 108 for calculation, and obtains the similarity value between the target feature 106 corresponding to the 1 / 16 resolution and the reference feature 107 corresponding to the 1 / 16 resolution.
[0191] The computer device inputs the target feature 106 corresponding to the 1 / 8 resolution, the reference feature 107 corresponding to the 1 / 8 resolution, and the reference mask 103 of the corresponding scale into the multi-layer dense cross-attention weighted mask aggregation network 108 for calculation, and obtains the similarity value between the target feature 106 corresponding to the 1 / 8 resolution and the reference feature 107 corresponding to the 1 / 8 resolution.
[0192] Exemplarily, the computer device obtains a reference mask 103 of a corresponding scale by downsampling 105 the reference mask 103 .
[0193] Downsampling 105 refers to reducing the reference mask 103 so that the reference mask 103 matches the corresponding reference feature 107. For example, if the resolution of the reference feature 107 is 1 / 8, the reference mask 103 is reduced by downsampling to a reference mask 103 that matches the reference feature 107 corresponding to the 1 / 8 resolution.
[0194] Step 806: multiply the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature to obtain the i-th target mask corresponding to the i-th scale target feature of the target image.
[0195] The computer device multiplies the i-th similarity value by the mask value of the reference mask matching the i-th scale reference feature in a multi-layer dense cross-attention weighted mask aggregation network to obtain the i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1.
[0196] The calculation formula of the i-th target mask may refer to the formula for calculating the target mask in step 406, and will not be repeated here.
[0197] Exemplarily, when a computer device obtains a similarity value between a target feature corresponding to a 1 / 32 resolution and a reference feature corresponding to a 1 / 32 resolution, the similarity value is multiplied by a mask value of a reference mask that matches the reference feature corresponding to the 1 / 32 resolution to obtain a first target mask corresponding to the target feature corresponding to the 1 / 32 resolution of the target image.
[0198] When the computer device obtains a similarity value between a target feature corresponding to a 1 / 16 resolution and a reference feature corresponding to a 1 / 16 resolution, the similarity value is multiplied by a mask value of a reference mask that matches the reference feature corresponding to the 1 / 16 resolution to obtain a second target mask corresponding to the target feature corresponding to the 1 / 16 resolution of the target image.
[0199] When the computer device obtains a similarity value between a target feature corresponding to a 1 / 8 resolution and a reference feature corresponding to a 1 / 8 resolution, the similarity value is multiplied by a mask value of a reference mask that matches the reference feature corresponding to the 1 / 8 resolution to obtain a third target mask corresponding to the target feature corresponding to the 1 / 8 resolution of the target image.
[0200] Step 808: Aggregate the target masks corresponding to different scales to obtain an intermediate target mask.
[0201] Exemplarily, the image semantic segmentation model includes a convolutional network; the computer device convolves the target masks corresponding to different scales through the convolutional network, and adds the target masks corresponding to different scales element by element to obtain an intermediate target mask.
[0202] Optionally, the image semantic segmentation model includes three convolutional networks, but is not limited to this, and the embodiments of the present application do not impose any limitations on this.
[0203] For example, Figure 1 As shown, the computer device inputs the first target mask corresponding to the 1 / 32 resolution into the convolution network 109 for three convolution calculations, and upsamples the convolution result of the target mask corresponding to the 1 / 32 resolution 110 to obtain a first intermediate target mask.
[0204] The computer device inputs the second target mask corresponding to the 1 / 16 resolution into the convolution network 109 for three convolution calculations, adds the convolution result of the second target mask corresponding to the 1 / 16 resolution to the first intermediate target mask element by element, performs three convolution calculations on the element by element addition result, and performs upsampling 110 to obtain the second intermediate target mask. Repeat the above steps, add the convolution result of the second intermediate target mask to the third target mask corresponding to the 1 / 8 resolution element by element, perform three convolution calculations on the element by element addition result, and finally obtain the intermediate target mask.
[0205] For example, Fig. 9 A schematic diagram of an image semantic segmentation method is shown, and a computer device obtains a target feature 901 corresponding to a target image, a reference feature 902 corresponding to at least one reference image, and a reference mask 903 corresponding to the reference image. The target feature 901 is represented as [h, w, c], where h represents the height of the target feature 901, w represents the width of the target feature 901, and c represents the length of the feature vector corresponding to each pixel in the target feature 901 (also known as the number of channels); the reference feature 902 is represented as [h, w, c], where h represents the height of the reference feature 902, w represents the width of the reference feature 902, and c represents the length of the feature vector corresponding to each pixel in the reference feature 902 (also known as the number of channels); the reference mask 903 is represented as [h, w, 1], where h represents the height of the reference mask 903, w represents the width of the reference mask 903, and 1 represents that the length of the feature vector corresponding to each pixel in the reference mask 903 is 1 (also known as the number of channels is 1).
[0206] The computer device flattens 904 the target feature 901, the reference feature 902 corresponding to at least one reference image, and the reference mask 903 corresponding to the reference image. There are n reference images, that is, the target feature 901, the reference feature 902, and the reference mask 903 are flattened 904, that is, the target feature 901, the reference feature 902, and the reference mask 903 are respectively reduced from two dimensions to one dimension, and the obtained target feature 901 is expressed as [h*w, c], the reference feature 902 is expressed as [n*h*w, c], and the reference mask 903 is expressed as [n*h*w, 1].
[0207] For example, the target feature 901 is a 3*3 vector matrix. By performing flattening processing 904 on the target feature 901, the target feature 901 obtained is a 9*1, or 1*9 vector matrix.
[0208] The computer device regards each pixel in the target feature 901 and the n reference features 902 as a processing unit, performs position encoding 905 on the target feature 901 and the n reference features 902 by using sine and cosine functions of different frequencies, and performs linear projection 906, thereby obtaining a one-dimensional sequence corresponding to the target image. The computer device implements the transformation 907 of the target feature 901, the reference feature 902, and the reference mask 903 by dividing the length of the feature vector corresponding to each pixel into multiple heads, and the obtained transformation target feature 908 is expressed as [heads, h*w, c / heads], the transformation reference feature 909 is expressed as [heads, n*h*w, c / heads], and the transformation reference mask 910 is expressed as [heads, n*h*w, 1].
[0209] Linear projection 906 is used to perform linear transformation on the vector matrix of target feature 901, thereby obtaining a one-dimensional sequence corresponding to the target image; linear projection 906 is used to perform linear transformation on the vector matrix of reference feature 902, thereby obtaining a one-dimensional sequence corresponding to the reference image.
[0210] The computer device calculates the similarity value 911 between the conversion target feature 908 and each head in the conversion reference feature 909, which can be expressed as [heads, h*w, n*h*w], and calculates the similarity value 911 as the weighted value of the conversion reference mask 910 912, and multiplies it with the mask value of each head in the conversion reference mask 910, so as to obtain the mask value of the target mask corresponding to the target image, which can be expressed as [heads, n*h*w, 1]. The computer device averages the mask values of multiple heads corresponding to each pixel 913 to obtain the averaged mask value of the target mask, which can be expressed as [h*w, 1], and then reshapes the target mask 914, upgrading the averaged target mask from one dimension to two dimensions, which can be expressed as [h, w, 1], and finally outputs the target mask 915, which is expressed as [h, w, 1].
[0211] Step 810: Fuse the intermediate target mask with the target feature to obtain a target mask corresponding to the target image.
[0212] Exemplarily, the image semantic segmentation model includes a hybrid network; a computer device performs a residual connection between an intermediate target mask and a target feature corresponding to at least one scale, and inputs the residual connection result of the intermediate target mask and the target feature into the hybrid network for fusion to obtain a target mask corresponding to the target image.
[0213] Optionally, the image semantic segmentation model includes three hybrid networks, each of which includes two groups of convolutional layers and two groups of activation layers ReLU; the intermediate target mask is residually connected with the target feature corresponding to at least one scale, and the residual connection results are sequentially input into the convolutional layer and the activation layer ReLU for fusion to obtain the target mask corresponding to the target image.
[0214] For example, Figure 1 As shown, the computer device performs three convolution calculations on the intermediate target mask, performs residual connection 111 on the convolution result of the intermediate target mask and the target feature 106 corresponding to 1 / 8 resolution to obtain a first residual connection result, upsamples the first residual connection result through bilinear interpolation 110, and performs residual connection 111 again with the target feature 106 corresponding to 1 / 4 resolution to obtain a second residual connection result, and inputs the second residual connection result into three hybrid networks 112 for fusion, and finally obtains a target mask 113 corresponding to the target image 101.
[0215] Step 812: Output the target mask corresponding to the target image.
[0216] Exemplarily, the computer device outputs a target mask corresponding to the target image.
[0217] Exemplarily, in order to verify the performance of the image semantic segmentation method in the present application, the performance of the image semantic segmentation method based on the multi-layer dense cross-attention weighted mask aggregation network in the present application is compared with the performance of the image semantic segmentation method published on the PASCAL-5 dataset, the COCO-20 dataset, and the FSS-1000 dataset. Each dataset is averaged and divided into four small datasets, namely Fold-1, Fold-2, Fold-3, and Fold-4. The evaluation is completed by cross-validation: each small dataset Fold is selected as the test set D in turn. test , the other three small data sets Fold are training sets D train , from the test set D test 1000 test tasks are randomly selected for evaluation. For example, the FSS-1000 dataset contains 1000 classes, of which the training set, validation set, and test set contain 520, 240, and 240 classes respectively, and 2400 test tasks are sampled from the test set for evaluation. The performance comparison tables of image semantic segmentation methods on the PASCAL-5 dataset, COCO-20 dataset, and FSS-1000 dataset are shown in Tables 1, 2, and 3.
[0218] Table 1 shows the performance of the image semantic segmentation methods published on the PASCAL-5 dataset. It can be seen from Table 1 that the image semantic segmentation method based on multi-layer DCAMA has the highest mean intersection over Union (MIoU) when there is one reference image or five reference images; the image semantic segmentation method based on multi-layer DCAMA has the highest foreground-background intersection over Union (FB-IoU) when there is one reference image or five reference images.
[0219] The average intersection-union ratio refers to the ratio of the intersection and union of the predicted value and the true value, and is used as an evaluation indicator for the performance of image semantic segmentation methods.
[0220] Table 1 Image semantic segmentation methods on the PASCAL-5 dataset
[0221]
[0222] Table 2 shows the performance of the image semantic segmentation methods published on the COCO-20 dataset. It can be seen from Table 2 that the image semantic segmentation method based on multi-layer DCAMA has the highest MIoU when there is one reference image or five reference images; the image semantic segmentation method based on multi-layer DCAMA has the highest FB-IoU when there is one reference image or five reference images.
[0223] Table 3 shows the performance of the image semantic segmentation methods published on the Fss-1000 dataset. It can be seen from Table 3 that the image semantic segmentation method based on multi-layer DCAMA has the highest MIoU when there is one reference image or five reference images; the image semantic segmentation method based on multi-layer DCAMA has the highest FB-IoU when there is one reference image or five reference images.
[0224] Table 2 Image semantic segmentation methods on COCO-20 dataset
[0225]
[0226] Table 3 Image semantic segmentation methods on COCO-20 dataset
[0227]
[0228] For example, Fig.10 The figure shows a schematic diagram comparing the training efficiency of the image semantic segmentation method based on multi-layer DCAMA and the comparison scheme. Fig.10As shown, compared with the comparative scheme, the image semantic segmentation method based on multi-layer DCAMA in the present application requires fewer training Epochs to achieve fitting, and under the same training Epoch, the MIoU of the present application scheme is higher.
[0229] by Fig.10 Taking Figure (a) in the figure as an example, the image semantic segmentation method based on multi-layer DCAMA in the present application can achieve fitting when the training Epoch is 30, while the comparison scheme can only achieve fitting when the training Epoch is 105.
[0230] In summary, the method provided in this embodiment obtains multi-scale target features corresponding to the target image, reference features corresponding to the reference image, and a reference mask; inputs the target features, reference features, and reference masks corresponding to different scales into the image semantic segmentation model for calculation to obtain target masks corresponding to different scales; aggregates the target masks corresponding to different scales to obtain an intermediate target mask; and fuses the intermediate target mask with the target features to obtain a target mask corresponding to the target image. The present application obtains the target mask corresponding to the target image by calculating the similarity value between all features of the target image and the reference image, and using the similarity value as the weighted value of the reference mask. Based on the above method, a more accurate image semantic segmentation result can be obtained.
[0231] Fig.11 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0232] In the medical field, medical images have become an important auxiliary tool for medical auxiliary diagnosis. Image semantic segmentation of medical images and labeling of skin lesions in medical images based on the results of image semantic segmentation can better assist medical staff in medical diagnosis, such as Fig.11 As shown, the target skin image 1101 , the reference skin image 1102 , and the reference skin mask 1103 are input into the image semantic segmentation model 1204 to perform image semantic segmentation on the target skin image 1101 , and obtain the image semantic segmentation result: the target skin mask 1105 .
[0233] The reference skin image 1102 includes lesions A, B, C, D, E, and F. The reference skin image 1102 is an image that belongs to the same category as the target skin image 1101 and has been subjected to image semantic segmentation. The reference skin masks 1103 corresponding to the reference skin image 1102 are mask A, mask B, mask C, mask D, mask E, and mask F, respectively, wherein mask A is the image semantic segmentation result corresponding to lesion A, mask B is the image semantic segmentation result corresponding to lesion B, mask C is the image semantic segmentation result corresponding to lesion C, mask D is the image semantic segmentation result corresponding to lesion D, mask E is the image semantic segmentation result corresponding to lesion E, and mask F is the image semantic segmentation result corresponding to lesion F. The reference skin mask 1103 is used to indicate different semantic regions after image semantic segmentation of the reference skin image 1102.
[0234] When performing image semantic segmentation on the target skin image 1101, the target feature corresponding to the target skin image 1101, the reference feature corresponding to at least one reference skin image 1102, and the reference skin mask 1103 corresponding to the reference skin image 1102 are obtained. The similarity value between the target feature and the reference feature is calculated in the image semantic segmentation model, and the similarity value is used as a weighted value to calculate with the reference skin mask 1103 to obtain the target skin mask 1105 corresponding to the target skin image 1101, that is, the lesion area in the target skin image 1101 is displayed in the target skin mask 1105.
[0235] In the process of image semantic segmentation, the accuracy of image semantic segmentation affects the accuracy of information annotation in medical images, and further affects the accuracy of medical diagnosis made by medical staff based on information annotation. Therefore, in the medical image-assisted diagnosis scenario, by using the image semantic segmentation method provided by the present application, the accuracy of semantic segmentation of medical images can be improved, thereby improving the ability to accurately segment lesions and improving the accuracy of medical-assisted diagnosis.
[0236] Fig.12 is a schematic diagram of an image semantic segmentation method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0237] In the autonomous driving scenario, in order to add necessary perception to the vehicle and obtain the environment in which the vehicle is located so that the autonomous driving vehicle can drive safely, it is necessary to obtain road scene images in real time and perform image semantic segmentation on the road scene images to obtain the location of obstacles in the road scene and control the autonomous driving of the vehicle based on the obstacle location.
[0238] like Fig.12 As shown, the target road image 1201 , the reference road image 1202 , and the reference road mask 1203 are input into the image semantic segmentation model 1204 to perform image semantic segmentation on the target road image 1201 , and obtain the image semantic segmentation result: the target road mask 1205 .
[0239] The reference road image 1202 includes obstacles, and the reference road image 1202 is an image that belongs to the same category as the target road image 1201 and has been subjected to image semantic segmentation. The reference road mask 1203 corresponding to the reference road image 1202 performs semantic segmentation on the obstacles in the reference road image 1202, and the reference road mask 1203 is used to indicate different semantic regions after the reference road image 1202 is subjected to image semantic segmentation.
[0240] When performing image semantic segmentation on the target road image 1201, the target feature corresponding to the target road image 1201, the reference feature corresponding to at least one reference road image 1202, and the reference road mask 1203 corresponding to the reference road image 1202 are obtained. The similarity value between the target feature and the reference feature is calculated in the image semantic segmentation model, and the similarity value is used as a weighted value to calculate with the reference road mask 1203 to obtain the target road mask 1205 corresponding to the target road image 1201, that is, the obstacles in the target road image 1201 are displayed in the target road mask 1205.
[0241] In this process, by adopting the image semantic segmentation method provided by the present application, the accuracy of image semantic segmentation can be improved, and then the precise location of obstacles in the road scene can be obtained, thereby improving the safety of autonomous driving.
[0242] The image semantic segmentation method involved in the present application can be implemented based on an image semantic segmentation model. The scheme includes an image semantic segmentation model generation stage and an image semantic segmentation stage. Fig.13 is a framework diagram of image semantic segmentation model generation and image semantic segmentation shown in an exemplary embodiment of the present application, such as Fig.13 As shown, in the image semantic segmentation model generation stage, the image semantic segmentation model generation device 1310 obtains the image semantic segmentation model through a pre-set training sample data set (including a sample target image and a real target mask corresponding to the sample target image), and then generates an image semantic segmentation result based on the image semantic segmentation model. In the image semantic segmentation stage, the image semantic segmentation device 1320 processes the input target image based on the image semantic segmentation model to obtain the image semantic segmentation result of the target image, such as determining the area and position of each object in the target image.
[0243] Among them, the above-mentioned image semantic segmentation model generation device 1310 and image semantic segmentation device 1320 can be computer devices. For example, the computer device can be a fixed computer device such as a personal computer or a server, or the computer device can also be a mobile computer device such as a tablet computer or an e-book reader.
[0244] Optionally, the above-mentioned image semantic segmentation model generation device 1310 and image semantic segmentation device 1320 can be the same device, or the image semantic segmentation model generation device 1310 and the image semantic segmentation device 1320 can also be different devices. In addition, when the image semantic segmentation model generation device 1310 and the image semantic segmentation device 1320 are different devices, the image semantic segmentation model generation device 1310 and the image semantic segmentation device 1320 can be the same type of device, such as the image semantic segmentation model generation device 1310 and the image semantic segmentation device 1320 can both be servers; or the image semantic segmentation model generation device 1310 and the image semantic segmentation device 1320 can also be different types of devices, such as the image semantic segmentation device 1320 can be a personal computer or a terminal, and the image semantic segmentation model generation device 1310 can be a server, etc. The embodiment of the present application does not limit the specific types of the image semantic segmentation model generation device 1310 and the image semantic segmentation device 1320.
[0245] The above embodiment describes the image semantic segmentation method, and the following describes the training method of the image semantic segmentation model. The training method of the image semantic segmentation model provided in the embodiment of the present application can be applied to scenarios with a small number of samples (i.e., small samples). Accordingly, the training method can also be called a small sample learning method. The goal of small sample learning is to achieve relatively good model training accuracy when the samples are limited.
[0246] Fig.14 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0247] Step 1402: Obtain a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask.
[0248] Exemplarily, the sample target image refers to a sample image to be subjected to image semantic segmentation. The target feature refers to a feature representation obtained by extracting features from the sample target image.
[0249] Image semantic segmentation refers to the process of identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs. For example, all pixels related to trees in a picture are classified into one category, and all pixels related to cars in the picture are classified into another category.
[0250] The reference image refers to an image that has been subjected to image semantic segmentation. The reference image is an image that belongs to the same category as the target image to be subjected to image semantic segmentation. The number of reference images is selected to be at least one, but is not limited thereto. The embodiment of the present application does not specifically limit the number of reference images.
[0251] The reference mask is used to indicate different semantic regions after the reference image is semantically segmented. For example, if the image semantic segmentation has been performed on airplane B in the reference image, the mask value of the reference mask corresponding to the region corresponding to airplane B in the reference image is 1, and the mask value of the reference mask corresponding to the rest of the reference image is 0.
[0252] The reference feature refers to a feature representation obtained by extracting features from a reference image. The reference feature corresponding to at least one reference image refers to a feature representation obtained by extracting features from each reference image individually.
[0253] It can be understood that in the present application, performing feature extraction on at least one reference image means performing separate feature extraction on each reference image. For example, performing feature extraction on three reference images, and finally obtaining three reference features corresponding to the three reference images.
[0254] The true target mask is used to indicate different semantic regions after image semantic segmentation of the sample target image.
[0255] Optionally, the real target mask corresponding to the sample target image may be annotated manually by relevant personnel on the sample target image.
[0256] Step 1404: Input the target features, reference features and reference masks into the image semantic segmentation model to calculate the similarity value between the target features and the reference features, and use the similarity value as a weighted value to calculate with the reference mask to obtain a predicted target mask corresponding to the sample target image.
[0257] The predicted target mask is used to indicate different semantic regions after image semantic segmentation of the sample target image.
[0258] The image semantic segmentation model is a model used to segment an image into blocks with certain semantic meanings. The image semantic segmentation model calculates the similarity between the target features and the reference features based on the target features of the sample target image, the reference features of the reference image, and the reference mask, and multiplies the similarity value with the reference mask as a weighted value, and finally obtains the predicted target mask corresponding to the area to be semantically segmented in the sample target image.
[0259] Step 1406: Calculate the loss function value based on the predicted target mask and the true target mask.
[0260] Exemplarily, the computer device calculates a loss function value based on the predicted target mask and the true target mask.
[0261] Optionally, the loss function value is at least one of the cross entropy between the predicted target mask and the true target mask, the mean square error between the predicted target mask and the true target mask, and the absolute difference between the predicted target mask and the true target mask, but is not limited to this and is not limited to this in the embodiments of the present application.
[0262] Step 1408: Update the model parameters of the image semantic segmentation model based on the loss function value.
[0263] Exemplarily, the computer device updates the model parameters of the image semantic segmentation model according to the loss function value.
[0264] Model parameter updating refers to updating the network parameters in the image semantic segmentation model, or updating the network parameters of each network module in the model, or updating the network parameters of each network layer in the model, but is not limited to this, and the embodiments of the present application do not limit this.
[0265] To summarize, the method provided in this embodiment obtains the target features corresponding to the sample target image, the reference features and the reference mask corresponding to the reference image; inputs the target features, the reference features and the reference mask into the image semantic segmentation model for calculation, obtains the predicted target mask corresponding to the sample target image, and calculates the loss function value based on the predicted target mask and the true target mask; updates the model parameters of the image semantic segmentation model according to the loss function value, so that the trained image semantic segmentation model can have a higher image semantic segmentation accuracy, thereby generating a more accurate image semantic segmentation result.
[0266] Fig.15 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0267] Step 1502: Obtain a target feature corresponding to a sample target image, a reference feature corresponding to a reference image, a reference mask corresponding to the reference image, and a true target mask.
[0268] Exemplarily, the sample target image refers to a sample image to be subjected to image semantic segmentation. The target feature refers to a feature representation obtained by extracting features from the sample target image.
[0269] Image semantic segmentation refers to the process of identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs. For example, all pixels related to trees in a picture are classified into one category, and all pixels related to cars in the picture are classified into another category.
[0270] The reference image refers to the image that has been semantically segmented.
[0271] The reference mask is used to indicate different semantic regions after image semantic segmentation of the reference image.
[0272] Reference features refer to feature representations obtained by extracting features from reference images.
[0273] The true target mask is used to indicate different semantic regions after image semantic segmentation of the sample target image.
[0274] Optionally, the real target mask corresponding to the sample target image may be annotated manually by relevant personnel on the sample target image.
[0275] In one possible implementation, the image semantic segmentation model includes a feature extraction network; a computer device obtains a sample target image and a reference image; the computer device extracts features from the sample target image through the feature extraction network to obtain target features; and extracts features from the reference image through the feature extraction network to obtain reference features.
[0276] Optionally, the feature extraction network is at least one of a residual neural network (ResNet) and a deep convolutional network VGG, but is not limited thereto and is not limited to this in the embodiments of the present application.
[0277] Step 1504: Input the target feature, reference feature and reference mask into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain a similarity value between the target feature and the reference feature.
[0278] Exemplarily, the computer device inputs the target feature, the reference feature and the reference mask into a multi-layer dense cross-attention weighted mask aggregation network to perform a scaled dot product attention calculation to obtain a similarity value between the target feature and the reference feature.
[0279] Both the target feature and the reference feature are pixel-level feature representations, and the reference feature includes the feature representations of foreground pixels and background pixels in the reference image.
[0280] Exemplarily, the computer device uses the target feature as the first vector matrix Q (Query), the reference feature as the second vector matrix K (Key), and the reference mask as the third vector matrix V (Value). The computer device inputs the target feature, the reference feature, and the reference mask into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the similarity value between the target feature Q and the reference feature K.
[0281] The similarity calculation formula between the target feature Q and the reference feature K can be expressed as:
[0282] C=QK T
[0283] Among them, C represents the similarity value, Q is the target feature, and K is the reference feature.
[0284] Step 1506: In a multi-layer dense cross-attention weighted mask aggregation network, the similarity value is multiplied by the mask value of the reference mask to calculate the predicted target mask corresponding to the sample target image.
[0285] The predicted target mask is used to indicate different semantic regions after image semantic segmentation of the sample target image.
[0286] The computer device multiplies the similarity value with the mask value of the reference mask in a multi-layer dense cross-attention weighted mask aggregation network to calculate a predicted target mask corresponding to the target image.
[0287] The calculation formula for predicting the target mask may refer to the calculation formula for the target mask in step 406, which will not be described in detail here.
[0288] Step 1508: Calculate the loss function value based on the predicted target mask and the true target mask.
[0289] Exemplarily, the computer device calculates a loss function value based on the predicted target mask and the true target mask.
[0290] Optionally, the loss function value is at least one of the cross entropy between the predicted target mask and the true target mask, the mean square error between the predicted target mask and the true target mask, and the absolute difference between the predicted target mask and the true target mask, but is not limited to this and is not limited to this in the embodiments of the present application.
[0291] Step 1510: Update the model parameters of the image semantic segmentation model based on the loss function value.
[0292] Exemplarily, the computer device updates the model parameters of the image semantic segmentation model according to the loss function value.
[0293] Model parameter updating refers to updating the network parameters in the image semantic segmentation model, or updating the network parameters of each network module in the model, or updating the network parameters of each network layer in the model, but is not limited to this, and the embodiments of the present application do not limit this.
[0294] To summarize, the method provided in this embodiment obtains a target feature corresponding to a sample target image, a reference feature corresponding to a reference image, a reference mask corresponding to the reference image, and a true target mask; the target feature, the reference feature, and the reference mask are input into the image semantic segmentation model for calculation to obtain a similarity value between the target feature and the reference feature, the similarity value is multiplied by the mask value of the reference mask, the predicted target mask corresponding to the sample target image is calculated, and the loss function value is calculated based on the predicted target mask and the true target mask; the model parameters of the image semantic segmentation model are updated according to the loss function value, so that the trained image semantic segmentation model can have a higher image semantic segmentation accuracy, thereby generating a more accurate image semantic segmentation result.
[0295] Fig.16 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0296] Step 1602: Obtain a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask.
[0297] Exemplarily, the sample target image refers to a sample image to be subjected to image semantic segmentation. The target feature refers to a feature representation obtained by extracting features from the sample target image.
[0298] Image semantic segmentation refers to the process of identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs. For example, all pixels related to trees in a picture are classified into one category, and all pixels related to cars in the picture are classified into another category.
[0299] The reference image refers to the image that has been semantically segmented.
[0300] The reference mask is used to indicate different semantic regions after image semantic segmentation of the reference image.
[0301] The reference feature refers to a feature representation obtained by extracting features from a reference image. The reference feature corresponding to at least one reference image refers to a feature representation obtained by extracting features from each reference image individually.
[0302] The true target mask is used to indicate different semantic regions after image semantic segmentation of the sample target image.
[0303] Optionally, the real target mask corresponding to the sample target image may be annotated manually by relevant personnel on the sample target image.
[0304] Step 1604: Input the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0305] Exemplarily, the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature.
[0306] Scale refers to the resolution corresponding to the target feature. For example, four scale target features corresponding to the sample target image are obtained. The four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, but are not limited to this. The embodiments of the present application are not limited to this.
[0307] The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into the image semantic segmentation model for calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0308] Exemplarily, the computer device uses the target feature as the first vector matrix Q (Query), the reference feature as the second vector matrix K (Key), and the reference mask as the third vector matrix V (Value). The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network to perform a scaled dot product attention calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature. The formula for calculating the i-th similarity value between the i-th scale target feature and the i-th scale reference feature can refer to the formula for calculating the similarity value in step 404, which will not be repeated here.
[0309] Step 1606: multiply the ith similarity value by the mask value of the reference mask matched with the ith scale reference feature to obtain the ith predicted target mask corresponding to the ith scale target feature of the sample target image.
[0310] The computer device multiplies the i-th similarity value by the mask value of the reference mask matching the i-th scale reference feature in a multi-layer dense cross-attention weighted mask aggregation network to obtain the i-th predicted target mask corresponding to the i-th scale target feature of the sample target image, where i is a positive integer greater than 1.
[0311] The calculation formula of the i-th predicted target mask may refer to the formula for calculating the target mask in step 406, and will not be repeated here.
[0312] Step 1608: Aggregate the predicted target masks corresponding to different scales to obtain the predicted target mask corresponding to the sample target image.
[0313] Exemplarily, the image semantic segmentation model includes a convolutional network; the computer device convolves the predicted target masks corresponding to different scales through the convolutional network, and adds the predicted target masks corresponding to different scales element by element, and finally obtains the predicted target mask corresponding to the sample target image.
[0314] Optionally, the image semantic segmentation model includes three convolutional networks, but is not limited to this, and the embodiments of the present application do not impose any limitations on this.
[0315] Step 1610: Calculate the loss function value based on the predicted target mask and the true target mask.
[0316] Exemplarily, the computer device calculates a loss function value based on the predicted target mask and the true target mask.
[0317] Optionally, the loss function value is at least one of the cross entropy between the predicted target mask and the true target mask, the mean square error between the predicted target mask and the true target mask, and the absolute difference between the predicted target mask and the true target mask, but is not limited to this and is not limited to this in the embodiments of the present application.
[0318] Step 1612: Update the model parameters of the image semantic segmentation model based on the loss function value.
[0319] Exemplarily, the computer device updates the model parameters of the image semantic segmentation model according to the loss function value.
[0320] In summary, the method provided in this embodiment obtains multi-scale target features corresponding to the sample target image, reference features corresponding to the reference image, reference mask corresponding to the reference image, and true target mask; inputs the target features, reference features, and reference masks corresponding to different scales into the image semantic segmentation model for calculation to obtain target masks corresponding to different scales; aggregates the target masks corresponding to different scales to obtain a predicted target mask corresponding to the sample target image, and calculates the loss function value based on the predicted target mask and the true target mask; updates the model parameters of the image semantic segmentation model according to the loss function value, so that the trained image semantic segmentation model can have higher image semantic segmentation accuracy, thereby generating more accurate image semantic segmentation results.
[0321] Fig.17 is a flowchart of a method for training an image semantic segmentation model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment. The method comprises:
[0322] Step 1702: Obtain a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask.
[0323] Exemplarily, the sample target image refers to a sample image to be subjected to image semantic segmentation. The target feature refers to a feature representation obtained by extracting features from the sample target image.
[0324] Image semantic segmentation refers to the process of identifying images at the pixel level and predicting the class label of each pixel in the image, that is, marking the object category to which each pixel in the image belongs. For example, all pixels related to trees in a picture are classified into one category, and all pixels related to cars in the picture are classified into another category.
[0325] The reference image refers to the image that has been semantically segmented.
[0326] The reference mask is used to indicate different semantic regions after image semantic segmentation of the reference image.
[0327] The reference feature refers to a feature representation obtained by extracting features from a reference image. The reference feature corresponding to at least one reference image refers to a feature representation obtained by extracting features from each reference image individually.
[0328] The true target mask is used to indicate different semantic regions after image semantic segmentation of the sample target image.
[0329] Optionally, the real target mask corresponding to the sample target image may be annotated manually by relevant personnel on the sample target image.
[0330] Step 1704: Input the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into the image semantic segmentation model for calculation to obtain the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0331] Exemplarily, the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature.
[0332] Scale refers to the resolution corresponding to the target feature. For example, four scale target features corresponding to the sample target image are obtained. The four scales are 1 / 4 resolution, 1 / 8 resolution, 1 / 16 resolution, and 1 / 32 resolution, but are not limited to this. The embodiments of the present application are not limited to this.
[0333] The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into the image semantic segmentation model for calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature.
[0334] Exemplarily, the computer device uses the target feature as the first vector matrix Q (Query), the reference feature as the second vector matrix K (Key), and the reference mask as the third vector matrix V (Value). The computer device inputs the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network to perform a scaled dot product attention calculation, and obtains the i-th similarity value between the i-th scale target feature and the i-th scale reference feature. The formula for calculating the i-th similarity value between the i-th scale target feature and the i-th scale reference feature can refer to the formula for calculating the similarity value in step 404, which will not be repeated here.
[0335] Step 1706: multiply the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature to obtain the i-th predicted target mask corresponding to the i-th scale target feature of the sample target image.
[0336] The computer device multiplies the i-th similarity value by the mask value of the reference mask matching the i-th scale reference feature in a multi-layer dense cross-attention weighted mask aggregation network to obtain the i-th predicted target mask corresponding to the i-th scale target feature of the sample target image, where i is a positive integer greater than 1.
[0337] The calculation formula of the i-th predicted target mask may refer to the formula for calculating the target mask in step 406, and will not be repeated here.
[0338] Step 1708: Aggregate the prediction target masks corresponding to different scales to obtain an intermediate prediction target mask.
[0339] Exemplarily, the image semantic segmentation model includes a convolutional network; the computer device convolves the predicted target masks corresponding to different scales through the convolutional network, and adds the predicted target masks corresponding to different scales element by element to obtain an intermediate predicted target mask.
[0340] Optionally, the image semantic segmentation model includes three convolutional networks, but is not limited to this, and the embodiments of the present application do not impose any limitations on this.
[0341] Step 1710: Fuse the intermediate predicted target mask with the target feature to obtain a predicted target mask corresponding to the sample target image.
[0342] Exemplarily, the image semantic segmentation model includes a hybrid network; a computer device performs a residual connection between an intermediate predicted target mask and a target feature corresponding to at least one scale, and inputs the residual connection result of the intermediate predicted target mask and the target feature into the hybrid network for fusion to obtain a predicted target mask corresponding to the sample target image.
[0343] Optionally, the image semantic segmentation model includes three hybrid networks, each of which includes two groups of convolutional layers and two groups of activation layers ReLU; the intermediate predicted target mask is residually connected with the target feature corresponding to at least one scale, and the residual connection results are sequentially input into the convolutional layer and the activation layer ReLU for fusion to obtain the predicted target mask corresponding to the sample target image.
[0344] Step 1712: Calculate the loss function value based on the predicted target mask and the true target mask.
[0345] Exemplarily, the computer device calculates a loss function value based on the predicted target mask and the true target mask.
[0346] Optionally, the loss function value is at least one of the cross entropy between the predicted target mask and the true target mask, the mean square error between the predicted target mask and the true target mask, and the absolute difference between the predicted target mask and the true target mask, but is not limited to this and is not limited to this in the embodiments of the present application.
[0347] Step 1714: Update the model parameters of the image semantic segmentation model based on the loss function value.
[0348] Exemplarily, the computer device updates the model parameters of the image semantic segmentation model according to the loss function value.
[0349] Model parameter updating refers to updating the network parameters in the image semantic segmentation model, or updating the network parameters of each network module in the model, or updating the network parameters of each network layer in the model, but is not limited to this, and the embodiments of the present application do not limit this.
[0350] The model parameters of the image semantic segmentation model include at least one of the network parameters of a multi-layer dense cross-attention weighted mask aggregation network, the network parameters of a convolutional network, and the network parameters of a hybrid network in the image semantic segmentation model.
[0351] When the loss function value is obtained, the computer device updates the network parameters of the multi-layer dense cross-attention weighted mask aggregation network, the convolutional network, and the hybrid network in the image semantic segmentation model based on the loss function value to obtain updated multi-layer dense cross-attention weighted mask aggregation network, the convolutional network, and the hybrid network, thereby obtaining a trained image semantic segmentation model.
[0352] In some embodiments, the model parameter update of the image semantic segmentation model includes updating the network parameters of all network modules in the image semantic segmentation model, or fixing the network parameters of some network modules in the image semantic segmentation model and only updating the network parameters of the remaining network modules. For example, when updating the model parameters of the image semantic segmentation model, the network parameters of the convolutional network and the network parameters of the convolutional network in the image semantic segmentation model are fixed, and only the network parameters of the multi-layer dense cross-attention weighted mask aggregation network are updated.
[0353] In summary, the method provided in this embodiment obtains multi-scale target features corresponding to the sample target image, reference features corresponding to the reference image, reference masks corresponding to the reference image, and true target masks; inputs the target features, reference features, and reference masks corresponding to different scales into the image semantic segmentation model for calculation to obtain target masks corresponding to different scales; aggregates the target masks corresponding to different scales to obtain an intermediate predicted target mask; fuses the intermediate predicted target mask with the target features to obtain a predicted target mask corresponding to the sample target image, and calculates the loss function value based on the predicted target mask and the true target mask; updates the model parameters of the image semantic segmentation model according to the loss function value, so that the trained image semantic segmentation model can have higher image semantic segmentation accuracy, thereby generating more accurate image semantic segmentation results.
[0354] It is understandable that in different application fields, the training image sets used to train the image semantic segmentation model are different. Several exemplary application fields are described below.
[0355] 1. Assisted driving field: Since the trained image semantic segmentation model is used to identify different road elements in the image, the training images in the training image set are road images containing road elements.
[0356] 2. Object detection field: Since the trained image semantic segmentation model is used to identify objects contained in the image, the training images in the training image set need to contain objects in different postures and angles.
[0357] Optionally, the training images in the training image set are actually taken photos (which require manual category labeling later), or screenshots taken from the game screen (the category of the object to which each pixel point belongs can be obtained directly from the game application), but are not limited to this and are not limited to the embodiments of the present application.
[0358] Regarding the method of training an image semantic segmentation model, in one possible implementation, a computer device inputs a sample target image into an initial image semantic segmentation model to obtain a predicted category corresponding to each pixel point output by the initial image semantic segmentation model, i.e., a predicted target mask. Then, based on the error between the predicted target mask and the true target mask, the parameters in the initial image semantic segmentation model are updated through a back propagation algorithm, and finally the image semantic segmentation model is trained.
[0359] Illustratively, the application scenarios of the image semantic segmentation method provided in the embodiments of the present application include but are not limited to the following scenarios:
[0360] 1) Autonomous driving scenarios;
[0361] In an autonomous driving scenario, in order to add necessary perception to the vehicle and obtain the environment in which the vehicle is located so that the autonomous driving vehicle can drive safely, it is necessary to obtain road scene images in real time and perform image semantic segmentation on the road scene images to obtain the positions of obstacles in the road scene and control the autonomous driving of the vehicle based on the positions of the obstacles. In this process, in order to improve the semantic segmentation effect of the road scene images, the image segmentation method provided in the present application can be used to improve the accuracy of image semantic segmentation, and then obtain the precise position of the obstacle position in the road scene, thereby improving the safety of autonomous driving.
[0362] 2) Medical image-assisted diagnosis scenarios;
[0363] In the medical field, medical images have become an important auxiliary tool for medical-assisted diagnosis. Performing image semantic segmentation on medical images and labeling different organs in medical images based on the results of image semantic segmentation can better assist medical staff in making medical diagnoses. For example, performing image semantic segmentation on chest X-rays can label the areas corresponding to the lungs and heart in the X-rays. In this process, the accuracy of image semantic segmentation affects the accuracy of information labeling in medical images, and further affects the accuracy of medical diagnosis made by medical staff based on information labeling. Therefore, in the medical image-assisted diagnosis scenario, by using the image semantic segmentation method provided in this application, the accuracy of semantic segmentation of medical images can be improved, thereby improving the accuracy of information standards and the accuracy of medical-assisted diagnosis.
[0364] Of course, in addition to being applied to the above-mentioned scenarios, the method provided in the embodiments of the present application can also be applied to other scenarios that require training image semantic segmentation models, especially small sample scenarios. The embodiments of the present application do not limit the specific application scenarios.
[0365] Fig.18 The schematic diagram of the structure of an image semantic segmentation device provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device includes:
[0366] The acquisition module 1801 is used to acquire target features corresponding to the target image, reference features corresponding to at least one reference image, and a reference mask corresponding to the reference image, wherein the reference mask is used to indicate different semantic areas after image semantic segmentation is performed on the reference image.
[0367] The calculation module 1802 is used to input the target feature, the reference feature and the reference mask into the image semantic segmentation model to calculate the similarity value between the target feature and the reference feature, and use the similarity value as a weighted value to calculate with the reference mask to obtain the target mask corresponding to the target image, and the target mask is used to indicate different semantic areas after image semantic segmentation of the target image.
[0368] The output module 1803 is used to output the target mask corresponding to the target image.
[0369] In a possible implementation, the calculation module 1802 is also used to input the target feature, the reference feature and the reference mask into the multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the similarity value between the target feature and the reference feature; in the multi-layer dense cross-attention weighted mask aggregation network, the similarity value is used as the weighted value of the reference mask to calculate the target mask corresponding to the target image.
[0370] In a possible implementation, the calculation module 1802 is also used to multiply the similarity value with the mask value of the reference mask in the multi-layer dense cross-attention weighted mask aggregation network to calculate the target mask corresponding to the target image.
[0371] In a possible implementation, the calculation module 1802 is further used to input the i-th scale target feature, the i-th scale reference feature, and the reference mask matching the i-th scale reference feature into the image semantic segmentation model for calculation to obtain the i-th similarity value between the i-th scale target feature and the i-th scale reference feature; multiply the i-th similarity value by the mask value of the reference mask matching the i-th scale reference feature to obtain the i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1; aggregate the target masks corresponding to different scales to obtain an intermediate target mask; and fuse the intermediate target mask with the target feature to obtain the target mask corresponding to the target image.
[0372] In a possible implementation, the computing module 1802 is further configured to perform convolution on the target masks corresponding to different scales through the convolution network; and to add the convolution results of the target masks corresponding to different scales element by element to obtain the intermediate target mask.
[0373] In one possible implementation, the computing module 1802 is further used to perform a residual connection between the intermediate target mask and the target feature corresponding to at least one scale, and input the residual connection result of the intermediate target mask and the target feature into the hybrid network for fusion to obtain the target mask corresponding to the target image.
[0374] In a possible implementation, the computing module 1802 is further used to perform a residual connection between the intermediate target mask and the target feature corresponding to at least one scale, and input the residual connection result into the convolution layer and the activation layer ReLU in sequence for fusion to obtain the target mask corresponding to the target image.
[0375] In a possible implementation, the acquisition module 1801 is further used to acquire a target image and a reference image;
[0376] The target image is subjected to feature extraction by the feature extraction network to obtain the target feature; and the reference image is subjected to feature extraction by the feature extraction network to obtain the reference feature.
[0377] Fig.19 A schematic diagram of the structure of a training device for an image semantic segmentation model provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device includes:
[0378] An acquisition module 1901 is used to acquire a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask, wherein the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image;
[0379] A first calculation module 1902 is used to input the target feature, the reference feature and the reference mask into the image semantic segmentation model to calculate the similarity value between the target feature and the reference feature, and use the similarity value as a weighted value to calculate with the reference mask to obtain a predicted target mask corresponding to the sample target image, wherein the predicted target mask is used to indicate different semantic regions after image semantic segmentation is performed on the target image;
[0380] A second calculation module 1903, used to calculate a loss function value based on the predicted target mask and the real target mask;
[0381] The updating module 1904 is used to update the model parameters of the image semantic segmentation model based on the loss function value.
[0382] In a possible implementation, the first calculation module 1902 is also used to input the target feature, the reference feature and the reference mask into the multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain the similarity value between the target feature and the reference feature; in the multi-layer dense cross-attention weighted mask aggregation network, the similarity value is multiplied by the mask value of the reference mask to calculate the predicted target mask corresponding to the sample target image.
[0383] In a possible implementation, the first calculation module 1902 is further used to input the i-th scale target feature, the i-th scale reference feature, and the reference mask matched with the i-th scale reference feature into the image semantic segmentation model for calculation to obtain an i-th similarity value between the i-th scale target feature and the i-th scale reference feature;
[0384] Multiplying the i-th similarity value by the mask value of the reference mask matching the i-th scale reference feature to obtain an i-th predicted target mask corresponding to the i-th scale target feature of the sample target image, where i is a positive integer greater than 1;
[0385] Aggregating the predicted target masks corresponding to different scales to obtain an intermediate predicted target mask;
[0386] The intermediate predicted target mask is fused with the target feature to obtain the predicted target mask corresponding to the sample target image.
[0387] In a possible implementation, the first computing module 1902 is further used to convolve the predicted target masks corresponding to different scales through the convolutional network, and add the convolution results of the predicted target masks corresponding to different scales element by element to obtain the intermediate predicted target mask.
[0388] In one possible implementation, the first computing module 1902 is also used to perform a residual connection between the intermediate predicted target mask and the target feature corresponding to at least one scale, and input the residual connection result of the intermediate predicted target mask and the target feature into the hybrid network for fusion to obtain the predicted target mask corresponding to the sample target image.
[0389] In a possible implementation, the first computing module 1902 is also used to perform a residual connection between the intermediate predicted target mask and the target feature corresponding to at least one scale, and input the residual connection result into the convolution layer and the activation layer ReLU in sequence for fusion to obtain the predicted target mask corresponding to the sample target image.
[0390] In a possible implementation, the acquisition module 1901 is further used to acquire a sample target image and a reference image;
[0391] The target feature is obtained by extracting features from the sample target image through the feature extraction network; and the reference feature is obtained by extracting features from the reference image through the feature extraction network.
[0392] Fig. 20The structural block diagram of a computer device 2000 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as the server in the above-mentioned solution of the present application. The image computer device 2000 includes a central processing unit (CPU) 2001, a system memory 2004 including a random access memory (RAM) 2002 and a read-only memory (ROM) 2003, and a system bus 2005 connecting the system memory 2004 and the central processing unit 2001. The image computer device 2000 also includes a large-capacity storage device 2006 for storing an operating system 2009, an application program 2010 and other program modules 2011.
[0393] The mass storage device 2006 is connected to the central processing unit 2001 via a mass storage controller (not shown) connected to the system bus 2005. The mass storage device 2006 and its associated computer readable medium provide non-volatile storage for the image computer device 2000. That is, the mass storage device 2006 may include a computer readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.
[0394] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 2004 and mass storage device 2006 can be collectively referred to as memory.
[0395] According to various embodiments of the present disclosure, the image computer device 2000 can also be connected to a remote computer on a network such as the Internet for operation. That is, the image computer device 2000 can be connected to the network 2008 via the network interface unit 2007 connected to the system bus 2005, or the network interface unit 2007 can be used to connect to other types of networks or remote computer systems (not shown).
[0396] The memory also includes at least one computer program, which is stored in the memory. The central processing unit 2001 implements all or part of the steps in the image semantic segmentation method or the image semantic segmentation model training method shown in the above-mentioned embodiments by executing the at least one program.
[0397] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the image semantic segmentation method or the image semantic segmentation model training method provided by the above-mentioned method embodiments.
[0398] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the image semantic segmentation method or image semantic segmentation model training method provided by the above-mentioned method embodiments.
[0399] An embodiment of the present application also provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes to implement the image semantic segmentation method or the image semantic segmentation model training method provided in the above-mentioned method embodiments.
[0400] It will be understood that in the specific implementation of the present application, the data involved, historical data, and data related to user data processing such as portraits related to user identity or characteristics, when the above embodiments of the present application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0401] It should be understood that the "plurality" mentioned in this article refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.
[0402] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0403] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A method for image semantic segmentation, characterized in that: The method comprises: Acquire a target feature corresponding to a target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image, wherein the reference image is an image that belongs to the same category as the target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image; the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature; Inputting the i-th scale target feature, the i-th scale reference feature and the reference mask matched with the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain an i-th similarity value between the i-th scale target feature and the i-th scale reference feature; In the multi-layer dense cross-attention weighted mask aggregation network, multiplying the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature to obtain an i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1; Aggregating the target masks corresponding to different scales to obtain an intermediate target mask; The intermediate target mask is merged with the target feature to obtain a target mask corresponding to the target image, wherein the target mask is used to indicate different semantic regions after image semantic segmentation is performed on the target image; The target feature and the reference feature are both pixel-level feature representations, and the reference feature includes feature representations of foreground pixels and background pixels in the reference image.
2. The method according to claim 1, characterized in that The multi-layer dense cross-attention weighted mask aggregation network includes a convolutional network; The step of aggregating the target masks corresponding to different scales to obtain an intermediate target mask includes: Convolving the target masks corresponding to different scales through the convolutional network; The convolution results of the target masks corresponding to different scales are added element by element to obtain the intermediate target mask.
3. The method according to claim 2, characterized in that The multi-layer dense cross-attention weighted mask aggregation network includes a hybrid network; The step of fusing the intermediate target mask with the target feature to obtain a target mask corresponding to the target image includes: The intermediate target mask is residually connected to the target feature corresponding to at least one scale, and the residual connection result of the intermediate target mask and the target feature is input into the hybrid network for fusion to obtain the target mask corresponding to the target image.
4. The method according to claim 3, characterized in that The hybrid network includes two groups of convolutional layers and two groups of activation layers ReLU; The step of performing a residual connection between the intermediate target mask and the target feature corresponding to at least one scale, and inputting the residual connection result between the intermediate target mask and the target feature into the hybrid network for fusion to obtain the target mask corresponding to the target image includes: The intermediate target mask is residually connected to the target feature corresponding to at least one scale, and the residual connection result is sequentially input into the convolution layer and the activation layer ReLU for fusion to obtain the target mask corresponding to the target image.
5. The method according to any one of claims 1 to 4, characterized in that: The acquiring of a target feature corresponding to the target image and a reference feature corresponding to at least one reference image includes: Acquire a target image and a reference image; The target image is subjected to feature extraction through a feature extraction network to obtain the target feature; and the reference image is subjected to feature extraction through the feature extraction network to obtain the reference feature.
6. A method for training an image semantic segmentation model, characterized in that: The method comprises: Obtaining a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask, wherein the reference image is an image that belongs to the same category as the sample target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image; the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature; Inputting the i-th scale target feature, the i-th scale reference feature and the reference mask matched with the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain an i-th similarity value between the i-th scale target feature and the i-th scale reference feature; In the multi-layer dense cross-attention weighted mask aggregation network, multiplying the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature to obtain an i-th predicted target mask corresponding to the i-th scale target feature of the sample target image, where i is a positive integer greater than 1; Aggregating the predicted target masks corresponding to different scales to obtain an intermediate predicted target mask; The intermediate predicted target mask is merged with the target feature to obtain a predicted target mask corresponding to the sample target image, wherein the predicted target mask is used to indicate different semantic regions after image semantic segmentation is performed on the sample target image; Calculating a loss function value based on the predicted target mask and the true target mask; Based on the loss function value, the model parameters of the image semantic segmentation model are updated, wherein the image semantic segmentation model includes the multi-layer dense cross-attention weighted mask aggregation network; The target feature and the reference feature are both pixel-level feature representations, and the reference feature includes feature representations of foreground pixels and background pixels in the reference image.
7. The method according to claim 6, characterized in that The image semantic segmentation model includes a convolutional network; The step of aggregating the predicted target masks corresponding to different scales to obtain an intermediate predicted target mask includes: Convolving the predicted target masks corresponding to different scales through the convolutional network; The convolution results of the prediction target masks corresponding to different scales are added element by element to obtain the intermediate prediction target mask.
8. The method according to claim 7, characterized in that The image semantic segmentation model includes a hybrid network; The step of fusing the intermediate predicted target mask with the target feature to obtain a predicted target mask corresponding to the sample target image includes: The intermediate predicted target mask is residually connected to the target feature corresponding to at least one scale, and the residual connection result of the intermediate predicted target mask and the target feature is input into the hybrid network for fusion to obtain the predicted target mask corresponding to the sample target image.
9. The method according to claim 8, characterized in that The hybrid network includes two groups of convolutional layers and two groups of activation layers ReLU; The step of performing a residual connection between the intermediate predicted target mask and the target feature corresponding to at least one scale, and inputting the residual connection result between the intermediate predicted target mask and the target feature into the hybrid network for fusion to obtain the predicted target mask corresponding to the sample target image, comprises: The intermediate predicted target mask is residually connected to the target feature corresponding to at least one scale, and the residual connection result is sequentially input into the convolution layer and the activation layer ReLU for fusion to obtain the predicted target mask corresponding to the sample target image.
10. The method according to any one of claims 6 to 9, characterized in that: The image semantic segmentation model includes a feature extraction network; The step of acquiring a target feature corresponding to the sample target image and a reference feature corresponding to at least one reference image includes: Acquire a sample target image and a reference image; The target feature is obtained by extracting features from the sample target image through the feature extraction network; and the reference feature is obtained by extracting features from the reference image through the feature extraction network.
11. An image semantic segmentation device, characterized in that: The device comprises: an acquisition module, used to acquire a target feature corresponding to a target image, a reference feature corresponding to at least one reference image, and a reference mask corresponding to the reference image, wherein the reference image is an image that belongs to the same category as the target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image; the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature; A calculation module, used for inputting the i-th scale target feature, the i-th scale reference feature and the reference mask matched with the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation, to obtain an i-th similarity value between the i-th scale target feature and the i-th scale reference feature; The calculation module is further used to multiply the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature in the multi-layer dense cross-attention weighted mask aggregation network to obtain an i-th target mask corresponding to the i-th scale target feature of the target image, where i is a positive integer greater than 1; Aggregating the target masks corresponding to different scales to obtain an intermediate target mask; The intermediate target mask is merged with the target feature to obtain a target mask corresponding to the target image, wherein the target mask is used to indicate different semantic regions after image semantic segmentation is performed on the target image; The target feature and the reference feature are both pixel-level feature representations, and the reference feature includes feature representations of foreground pixels and background pixels in the reference image.
12. A training device for an image semantic segmentation model, characterized in that: The device comprises: An acquisition module, used to acquire a target feature corresponding to a sample target image, a reference feature corresponding to at least one reference image, a reference mask corresponding to the reference image, and a true target mask, wherein the reference image is an image that belongs to the same category as the sample target image and has been subjected to image semantic segmentation, and the reference mask is used to indicate different semantic regions after image semantic segmentation is performed on the reference image; the target feature is a multi-scale target feature, and the reference feature is a multi-scale reference feature; A first calculation module is used to input the i-th scale target feature, the i-th scale reference feature and the reference mask matched with the i-th scale reference feature into a multi-layer dense cross-attention weighted mask aggregation network for calculation to obtain an i-th similarity value between the i-th scale target feature and the i-th scale reference feature; The first calculation module is further used to multiply the i-th similarity value by the mask value of the reference mask matched with the i-th scale reference feature in the multi-layer dense cross-attention weighted mask aggregation network to obtain an i-th predicted target mask corresponding to the i-th scale target feature of the sample target image, where i is a positive integer greater than 1; Aggregating the predicted target masks corresponding to different scales to obtain an intermediate predicted target mask; The intermediate predicted target mask is merged with the target feature to obtain a predicted target mask corresponding to the sample target image, wherein the predicted target mask is used to indicate different semantic regions after image semantic segmentation is performed on the sample target image; A second calculation module, used for calculating a loss function value based on the predicted target mask and the real target mask; An updating module, configured to update model parameters of the image semantic segmentation model based on the loss function value, wherein the image semantic segmentation model includes the multi-layer dense cross-attention weighted mask aggregation network; The target feature and the reference feature are both pixel-level feature representations, and the reference feature includes feature representations of foreground pixels and background pixels in the reference image.
13. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein at least one computer program is stored in the memory, and at least one computer program is loaded and executed by the processor to implement the image semantic segmentation method as described in any one of claims 1 to 5, or the training method of the image semantic segmentation model as described in any one of claims 6 to 10.
14. A computer storage medium, characterized in that: The computer storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image semantic segmentation method as described in any one of claims 1 to 6, or the training method of the image semantic segmentation model as described in any one of claims 6 to 10.
15. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes the image semantic segmentation method as described in any one of claims 1 to 5, or the training method of the image semantic segmentation model as described in any one of claims 6 to 10.
