Artificial intelligence-based image segmentation method and device, computer device and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to an image segmentation method and device based on artificial intelligence, computer equipment and a medium. The method inputs a to-be-processed image and a reference image into a convolution model to obtain first image features and second image features, inputs the first image features and the second image features into an association model to obtain a feature association matrix, multiplies the feature association matrix and a label image to obtain aggregated features, inputs the aggregated features into a segmenter to obtain an image segmentation result, a small amount of reference images and labels thereof are used as reference information to assist a large amount of unlabeled to-be-processed images in image segmentation, the labeling cost of the images is reduced, when unknown segmentation categories exist in the to-be-processed images, feature difference information of the segmentation categories and existing segmentation categories can be provided according to the feature association information, and then the unknown segmentation categories can be accurately identified in the image segmentation process, and the accuracy of the image segmentation is improved.

Description

Image segmentation methods, apparatus, computer equipment, and media based on artificial intelligence Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image segmentation method, apparatus, computer equipment, and medium based on artificial intelligence. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, image segmentation technology is gradually being applied to various application scenarios, such as road information segmentation in autonomous driving scenarios and disease area segmentation in medical scenarios, realizing the intelligentization of application scenarios. Existing image segmentation technology usually adopts supervised training image segmentation models.

[0003] However, supervised training image segmentation models require a large number of image labels, and image segmentation requires labeling each pixel in the image, which is quite costly. The high labeling cost also makes it difficult for supervised training image segmentation models to be updated in a timely manner, resulting in low image segmentation accuracy. Therefore, how to improve the accuracy of image segmentation while reducing labeling cost has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image segmentation method, apparatus, computer device, and medium based on artificial intelligence to solve the problem of low accuracy in image segmentation when annotation costs are low.

[0005] In a first aspect, embodiments of the present invention provide an image segmentation method based on artificial intelligence, the image segmentation method comprising:

[0006] The acquired image to be processed is input into the trained convolutional model for feature extraction to obtain the first image features;

[0007] Obtain a reference image and its corresponding label image, and input the reference image into the trained convolutional model for feature extraction to obtain the second image features;

[0008] The first image feature and the second image feature are input into the trained association model to calculate the feature correlation, and the feature association matrix between the first image feature and the second image feature is obtained.

[0009] The feature association matrix and the label image are multiplied together to determine the aggregated feature. The aggregated feature is then input into the trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed.

[0010] Secondly, embodiments of the present invention provide an image segmentation device based on artificial intelligence, the image segmentation device comprising:

[0011] The first convolution module is used to input the acquired image to be processed into the trained convolution model for feature extraction to obtain the first image features;

[0012] The second convolutional module is used to acquire a reference image and its corresponding label image, and input the reference image into the trained convolutional model for feature extraction to obtain the second image features.

[0013] The feature association module is used to input the first image feature and the second image feature into the trained association model to calculate the feature correlation matrix between the first image feature and the second image feature.

[0014] The image segmentation module is used to multiply the feature association matrix and the label image, determine the multiplication result as the aggregated feature, input the aggregated feature into the trained segmenter for pixel category segmentation, and obtain the image segmentation result corresponding to the image to be processed.

[0015] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image segmentation method as described in the first aspect.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image segmentation method as described in the first aspect.

[0017] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0018] The acquired image to be processed is input into a trained convolutional model for feature extraction to obtain the first image feature. A reference image and its corresponding label image are obtained, and the reference image is input into the trained convolutional model for feature extraction to obtain the second image feature. The first and second image features are input into a trained association model for feature correlation calculation to obtain the feature association matrix between the first and second image features. The feature association matrix is ​​multiplied with the label image, and the result of the multiplication is determined as the aggregate feature. The aggregate feature is input into a trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed. By using a small number of reference images and their labels as reference information, image segmentation of a large number of images to be processed can be assisted, reducing the annotation cost of the image segmentation model. At the same time, when there is an unknown segmentation category in the image to be processed, the feature association information can provide feature difference information between the segmentation category and the existing segmentation categories, thereby accurately identifying the unknown segmentation category during the image segmentation process and improving the accuracy of image segmentation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 is a schematic diagram of an application environment for an image segmentation method based on artificial intelligence provided in Embodiment 1 of the present invention;

[0021] Figure 2 is a flowchart illustrating an image segmentation method based on artificial intelligence provided in Embodiment 1 of the present invention;

[0022] Figure 3 is a schematic diagram of an image segmentation device based on artificial intelligence provided in Embodiment 2 of the present invention;

[0023] Figure 4 is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0024] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0025] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0026] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0028] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0029] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0030] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0031] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0032] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0033] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0034] The first embodiment of this invention provides an image segmentation method based on artificial intelligence, which can be applied in the application environment shown in Figure 1, wherein the client and the server communicate with each other. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs), and other computer devices. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0035] Referring to Figure 2, which is a flowchart illustrating an artificial intelligence-based image segmentation method according to Embodiment 1 of the present invention, the image segmentation method can be applied to the client shown in Figure 1. The computer device corresponding to the client is connected to the server to obtain the image to be processed, the reference image, and the corresponding label image from the server. The image to be processed can refer to the image received by the server that needs to be segmented. The reference image can refer to the image stored on the server. The label image can refer to the image obtained by manually annotating the reference image. The computer device corresponding to the client has a pre-trained convolutional model, a pre-trained correlation model, and a pre-trained segmentation model deployed within it. The pre-trained convolutional model can be used to extract features from the input image. The pre-trained correlation model can be used to calculate the correlation between features. The pre-trained segmentation model can be used to segment the pixel categories of the features corresponding to the image. As shown in Figure 2, the image segmentation method may include the following steps:

[0036] Step S201: Input the acquired image to be processed into the trained convolutional model for feature extraction to obtain the first image features.

[0037] The image to be processed can refer to an image that needs to be segmented. The trained convolutional model can be used to extract the image features of the input image. In this embodiment, the input of the trained convolutional model is the image to be processed of a preset size, and the output of the trained convolutional model is the image features corresponding to the image to be processed, that is, the first image features.

[0038] Specifically, after obtaining the image to be processed, it is necessary to scale the image to meet the preset size. The preset size can be 648*648, that is, the image to be processed is a two-dimensional image. Each row of the two-dimensional image contains 648 pixels, and each column also contains 648 pixels. The purpose of scaling is to make the image to be processed meet the input size requirements of the trained convolutional model so that the trained convolutional model can perform convolution calculations.

[0039] Convolutional models can be such as VGG, ResNet, and MobileNet. In this embodiment, ResNet is selected as the convolutional model. ResNet includes a skip connection mechanism, meaning that for multiple convolutional layers in the model, the input of the i-th convolutional layer is the sum of the output and input of the (i-1)-th convolutional layer. This ensures that the extracted deep features do not degenerate when performing convolution calculations across multiple layers. After each convolutional layer outputs its result, the output is processed using an activation function, such as ReLU or Sigmoid. The purpose of applying activation functions to the convolutional layer output is to increase the non-linear representation capability of the features, thereby improving the fitting ability of the convolutional model.

[0040] After determining the convolutional model, the training method can adopt a self-supervised training approach to reduce the sample labeling cost during the training process. Training samples for the convolutional model are obtained, and each training sample is input into the convolutional model one by one to obtain sample features. These features are then input into the decoding model for reconstruction, resulting in reconstructed samples. The Euclidean distance between the training samples and the reconstructed samples is calculated as the training loss function. Based on the training loss function, gradient descent is used to train both the convolutional and decoding models, resulting in well-trained convolutional and decoding models. The well-trained decoding model is discarded, and only the well-trained convolutional model is retained. The decoding model contains multiple upsampling layers and is used to upsample sample features into reconstructed samples. The architecture of the decoding model is symmetrical to that of the convolutional model.

[0041] The above steps involve inputting the acquired image to be processed into a trained convolutional model for feature extraction to obtain the first image features. The convolutional model can effectively learn the corresponding sample features from a large number of training samples, avoiding a complex feature extraction process. At the same time, it extracts more abstract features from the original image through a nonlinear structure, reducing the complexity of the model and the number of weights. By relying on multiple convolutional layers for feature extraction, the features extracted by translating, scaling, tilting, or other forms of deformation of the input image have high invariance.

[0042] Step S202: Obtain the reference image and its corresponding label image, input the reference image into the trained convolutional model for feature extraction, and obtain the second image features.

[0043] The reference image can refer to a pre-acquired image or a historical image that has undergone image segmentation. The label image can refer to the annotation information corresponding to the reference image. The annotation information includes the true pixel category to which each pixel in the reference image belongs. That is, the size of the label image is consistent with the size of the reference image. It should be noted that since the reference image also needs to be used for feature extraction through a trained convolutional model, the size of the reference image also meets the preset size by default. That is, the size of the reference image is consistent with the size of the image to be processed. The second image feature can refer to the image features of the reference image.

[0044] Specifically, the reference image should be consistent with the applicable scenario of the image to be processed, so as to provide reference information for image segmentation of the image to be processed. For example, in the context of intelligent driving, if the image to be processed is a road image captured by a vehicle camera, then the reference image should also be a road image. Only when the reference image is consistent with the applicable scenario of the image to be processed can the annotation information of the reference image provide reference information for image segmentation of the processed image. For example, if both the reference image and the image to be processed are road images, then the features corresponding to pixels of known pixel categories such as road category pixels, road mark category pixels, and traffic light category pixels in the reference image can be applied to the image segmentation of the image to be processed. If the image to be processed is a medical image, and medical images generally do not include road category pixels, road mark category pixels, traffic light category pixels, etc., then the image features of the reference image cannot provide a good reference for the image segmentation of the image to be processed.

[0045] In the label image, the pixel values ​​of all pixels are of the same category as those of the pixels in the reference image. That is, one pixel value corresponds to one pixel category, and pixels of the same category have the same pixel value in the label image. The pixel category can be determined according to the specific application scenario of image segmentation. The semantic category of the pixels in the image acquired in the specific application scenario is the pixel category. For example, in the intelligent driving scenario, the pixel category can include four categories: road category pixels, road mark category pixels, traffic light category pixels, and vehicle category pixels. Accordingly, there are four possible pixel values ​​in the label image. In this embodiment, pixel value 1 represents a road category pixel, pixel value 2 represents a road mark category pixel, pixel value 3 represents a traffic light category pixel, and pixel value 4 represents a vehicle category pixel. The implementer can adjust the pixel values ​​and the correspondence between pixel values ​​and pixel categories according to the actual situation.

[0046] Optionally, obtaining the reference image and its corresponding label image includes:

[0047] The reference image is input into the trained convolutional model for feature extraction to obtain the second image features;

[0048] The second image features are input into the trained segmenter for image segmentation, and the image segmentation result is determined as the label image.

[0049] Since both the convolutional model and the segmenter are pre-trained, and the training samples used during their training may not be consistent with the reference image (i.e., the reference image should contain more semantic information about the scene), to reduce the manpower consumption of image annotation tasks, assuming that the pre-trained convolutional model can accurately extract input image features and the pre-trained segmenter can accurately segment the image based on those features, the reference image can be directly input into the pre-trained convolutional model and segmenter to obtain the image segmentation result as the label image of the reference image. It should be noted that in scenarios where high accuracy of the labeled data is required, manual annotation is still recommended. For example, in medical and autonomous driving scenarios, the accuracy of image segmentation is related to important factors such as personnel safety. In these cases, the accuracy of image segmentation should be guaranteed as much as possible, so manual annotation is used. However, in scenarios such as clothing image segmentation and face image segmentation, to improve training efficiency, the reference image can be input into the pre-trained convolutional model and segmenter to obtain the image segmentation result as the label image of the reference image.

[0050] This embodiment proposes a method to directly segment the reference image using a trained convolutional model and a trained segmenter to obtain a labeled image. This method can effectively save the resource consumption of labeling the reference image. Furthermore, since the labeled image is used for subsequent feature aggregation and not for training the convolutional model and segmenter, it will not affect the accuracy of the convolutional model and segmenter. Thus, without reducing the image segmentation accuracy, it can effectively reduce the human resource consumption of the image preprocessing process and improve the efficiency of image segmentation.

[0051] Optionally, a reference image and its corresponding label image are obtained, and the reference image is input into a trained convolutional model for feature extraction to obtain the second image features, including:

[0052] The reference image is reconstructed using a trained variational autoencoder to obtain N reconstructed images, and the label image corresponding to each reconstructed image is obtained.

[0053] The reference image and N reconstructed images are respectively input into the trained convolutional model for feature extraction, resulting in N+1 second image features.

[0054] Where N is a positive integer, the trained variational autoencoder can be used to reconstruct an input image into an output image that is similar to but not exactly the same as the input image. The reconstructed image can be an image that is similar to but not exactly the same as the reference image.

[0055] Specifically, the trained variational autoencoder includes a trained variational encoder and a trained variational decoder. The reference image is input into the trained variational encoder to obtain hidden layer features. At this time, Gaussian sampling is performed on the hidden layer features to subject them to random perturbation, so that the hidden layer features cannot be completely recovered from the input reference image. However, since the interference of random perturbation is small, the reconstructed image output by the trained variational decoder is relatively similar to the reference image. In this way, more reference information can be provided for image segmentation of the image to be processed.

[0056] The variational autoencoder also adopts a self-supervised approach during training. The training samples are input into the variational autoencoder to obtain reconstructed samples. The Euclidean distance between the reconstructed samples and the training samples is used as the reconstruction loss. At the same time, an additional KL divergence loss is added based on the distribution of the training samples and the distribution of the reconstructed samples, thereby constraining the reconstruction results to be sufficiently similar to the input, but not completely identical.

[0057] After obtaining N reconstructed images, the reconstructed images can be labeled manually, or the reconstructed images can be directly input into the trained convolutional model and the trained segmenter, and the output of the trained segmenter can be used as the label image corresponding to the reconstructed image.

[0058] Since there are N reconstructed images, the reconstructed images and the reference images are respectively input into the trained convolutional model to obtain N second image features corresponding to the reconstructed images and one second image feature corresponding to the reference image, for a total of N+1 second image features.

[0059] This embodiment uses a variational autoencoder to reconstruct the reference image, resulting in several reconstructed images. This effectively increases the number of reference images, thereby improving the generalization ability of the overall image segmentation and thus improving the accuracy of image segmentation.

[0060] Optionally, the first image features and the second image features are input into the trained association model to calculate feature correlation, thereby obtaining the feature association matrix between the first image features and the second image features. The feature association matrix is ​​then multiplied with the label image, and the result of the multiplication is determined as the aggregated features, including:

[0061] The first image features and N+1 second image features are input into the trained association model to obtain N+1 feature association matrices;

[0062] Multiply the N+1 feature association matrices by their corresponding label images to obtain N+1 aggregated features.

[0063] The first image feature is correlated with N+1 second image features through an association model to obtain N+1 feature association matrices corresponding to the second image features. Each second image feature corresponds to a label image, thus yielding N+1 aggregated features.

[0064] This embodiment calculates multiple aggregated features, thereby providing sufficient reference information for image segmentation of the image to be processed, which can effectively improve the accuracy of image segmentation of the image to be processed.

[0065] Optionally, the aggregated features are input into the trained segmenter for pixel category segmentation, resulting in image segmentation results for the image to be processed, including:

[0066] The mean of N+1 aggregated features is calculated to obtain the mean feature;

[0067] The mean features are input into the trained segmenter to perform pixel category segmentation, thus obtaining the image segmentation result corresponding to the image to be processed.

[0068] Among them, the mean feature can be used to represent the information after the fusion of N+1 aggregated features. The mean feature is input into the trained segmenter to perform pixel category segmentation, providing richer input information to the trained segmenter, thereby improving the accuracy of image segmentation by the trained segmenter.

[0069] In one embodiment, when calculating the mean of N+1 aggregated features, the implementer can use a weighted mean calculation method. In this case, the mean corresponding to each aggregated feature can be determined based on the similarity between the reconstructed image and the reference image corresponding to that aggregated feature. The similarity between the reconstructed image and the reference image can be calculated using cosine similarity, thereby directly obtaining the normalized similarity calculation result. The result of subtracting the similarity calculation is used as the weight of the corresponding aggregated feature. That is, the larger the difference between the reconstructed image and the reference image, the larger the weight of the aggregated feature corresponding to it. This is to avoid the aggregated features of the reference image having too much influence on the mean calculation, which would reduce the effect of data augmentation through reconstructed images to improve the generalization ability of image segmentation.

[0070] In this embodiment, multiple aggregated features are fused by mean fusion, thereby providing the segmenter with more sufficient and comprehensive additional information for image segmentation, thus improving the accuracy of image segmentation.

[0071] The steps described above—obtaining a reference image and its corresponding label image, inputting the reference image into a trained convolutional model for feature extraction to obtain second image features—extract the second image features of the reference image through the trained convolutional model, thereby serving as reference information for the image to be processed and assisting in image segmentation. This enables the image to be processed to achieve better image segmentation results even without labeled images, thus improving the accuracy of image segmentation.

[0072] Step S203: Input the first image features and the second image features into the trained association model to calculate the feature correlation matrix between the first image features and the second image features.

[0073] The trained association model can be used to extract feature association information between different input features. Feature correlation can refer to the degree of similarity between two features. The feature association matrix can be used to characterize the feature correlation between the first image feature and the second image feature.

[0074] Specifically, since the image to be processed and the reference image have the same image size, the image to be processed and the reference image are respectively input into the trained convolutional model to extract features, and the size of the first image feature and the second image feature extracted are also the same.

[0075] The input to the trained association model is the first image feature and the second image feature. That is, the first image feature and the second image feature are simultaneously input into the trained association model. The output of the trained association model is the feature association matrix between the input first image feature and the second image feature.

[0076] Optionally, the first image features and the second image features are input into the trained association model to calculate feature correlation, and the calculated feature correlation matrix between the first image features and the second image features includes:

[0077] The transpose of the second image feature is multiplied by the first image feature to obtain the multiplication result, which includes at least two rows of multiplied feature values.

[0078] Normalize the eigenvalues ​​of each row in the multiplication result, and concatenate the normalized results of all rows to obtain the feature correlation matrix.

[0079] Since the image features are represented as high-dimensional feature vectors, the feature vectors can be transposed. The result of the multiplication can be the feature correlation value between the first image feature and the second image feature. The purpose of the normalization process is to limit the feature correlation value to a preset range, thereby facilitating subsequent calculations.

[0080] Specifically, let the size of the first image feature be m*n. Since the sizes of the first and second image features are the same, the size of the second image feature is also m*n. The first and second image features cannot be directly multiplied. The second image feature is transposed, and the size of the transposed result is n*m, which allows for multiplication. The size of the multiplication result is m*m, meaning the multiplication result contains m rows and m columns, and contains m*m multiplied feature values. The m multiplied feature values ​​in a row are normalized. The normalization method can be a normalization exponential function to obtain the normalized value of each multiplied feature value in that row, which is the normalized result of that row. The normalized results of all rows are concatenated according to the column dimension, and the concatenated result is the feature correlation matrix.

[0081] This embodiment calculates the first image feature and the second image feature by transposing and multiplying, thereby effectively calculating the correlation between the first image feature and the second image feature, and performing normalization processing to facilitate subsequent calculation of aggregated features and improve the computational efficiency of image segmentation.

[0082] Step S204: Multiply the feature association matrix and the label image to determine the multiplication result as the aggregated feature. Input the aggregated feature into the trained segmenter to perform pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed.

[0083] Among them, the aggregated features can be used to characterize the image feature information of the image to be processed, the image feature information of the reference image, and the correlation information between the image features of the image to be processed and the image features of the reference image. The trained segmenter can take the form of a decoder. Pixel category segmentation can refer to classifying each pixel in the image to be processed into a pixel category. The image segmentation result can refer to the image in which all pixels in the image to be processed are adjusted to preset pixel values ​​according to their pixel categories.

[0084] Specifically, the feature correlation matrix and the label image are multiplied together. That is, based on the correlation information between the image features of the image to be processed and the image features of the reference image, the pixel category information contained in the label image corresponding to the reference image is fused with the image features of the image to be processed to obtain a more representative aggregated feature.

[0085] The trained segmenter can adopt a decoding model structure, which is consistent with the decoding model structure used in the training of the convolutional model in the above steps. However, since the decoder model trained in the above steps can only perform the task of image reconstruction, the trained decoding model is not used in this step. Instead, a new decoding model is connected with the trained convolutional model to obtain an image segmentation model. This image segmentation model is then trained. It should be noted that the implementer can randomly initialize the trained decoding model and use the randomly initialized decoding model as the new decoding model.

[0086] During the training of the image segmentation model, the training samples can still be the acquired sample images. The sample images are input into the encoding model in the image segmentation model. This encoding model is a pre-trained encoding model, which obtains the sample features of the sample images. Then, the sample features are input into the decoding model in the encoding model to obtain the decoding result. The training process of the image segmentation model is supervised training. At this time, it is necessary to manually label the pixel categories of the sample images pixel by pixel to obtain the sample labels of the corresponding sample images. The Euclidean distance between the sample labels and the decoding result is calculated. The Euclidean distance is used as the loss function during training. Based on this loss function, the gradient descent method is used to train the decoding model. It should be noted that the parameters of the pre-trained convolutional model in the image segmentation model are not updated at this time. That is, only the parameters of the decoding model are updated to obtain the trained segmenter.

[0087] In the image segmentation result corresponding to the image to be processed output by the trained segmenter, the pixel value of each pixel corresponds to a preset value, and each preset value corresponds to a pixel category. It should be noted that, in this embodiment, if the number of known pixel categories is C, then the number of preset values ​​is set to C+1, where the additional preset values ​​correspond to unknown pixel categories. This avoids the pixel category that the trained segmenter cannot accurately identify being segmented into a different pixel category, thus preventing misidentification and a decrease in the accuracy of image segmentation. In this embodiment, the preset value corresponding to the unknown pixel category is 0.

[0088] Optionally, the feature association matrix and the label image are multiplied together to determine the aggregated features, including:

[0089] The label image is dimensionality reduced to obtain a dimensionality-reduced image with the same feature dimensions as the first image;

[0090] Multiplying the dimensionality-reduced image and the feature correlation matrix yields the aggregated features.

[0091] Dimensionality reduction can refer to reducing the size of the label image while discarding its content. A dimensionality-reduced image is the image obtained after the label image has undergone dimensionality reduction processing.

[0092] Specifically, dimensionality reduction can be performed using methods such as principal component analysis and pooling. In this embodiment, max pooling is used for dimensionality reduction. The dimensionality-reduced image has the same feature dimension as the first image, that is, the size of the dimensionality-reduced image can be set to m*n. When multiplying the dimensionality-reduced image with the feature correlation matrix, the dimensionality-reduced image is right-multiplied by the feature correlation matrix, the size of the feature correlation matrix is ​​m*m, and the size of the resulting aggregated feature is m*n.

[0093] This embodiment performs dimensionality reduction on the label image, which facilitates the direct fusion of information from the label image and the feature association matrix to obtain aggregated features, thereby improving the computational efficiency of image segmentation.

[0094] The steps described above, which involve inputting the first image features and the second image features into the trained association model to calculate the feature correlation matrix between the first image features and the second image features, effectively extract the feature similarity information between the first image features and the second image features through feature correlation calculation. This facilitates the provision of additional information to assist in subsequent image segmentation and improve the image segmentation effect, thereby increasing the accuracy of image segmentation.

[0095] The steps described above, which involve multiplying the feature correlation matrix and the label image to determine the aggregated feature, and then inputting the aggregated feature into a trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed, fully aggregate the image feature information of the image to be processed, the image feature information of the reference image, and the pixel category information of the label image by multiplying the feature correlation matrix and the label image, thereby obtaining an aggregated feature with stronger representation ability. This allows the segmenter based on the aggregated feature as input to output a more accurate image segmentation result, thus improving the accuracy of image segmentation.

[0096] This embodiment uses a small number of reference images and their labels as reference information to assist in image segmentation of a large number of images to be processed, reducing the annotation cost of the image segmentation model. At the same time, when there are unknown segmentation categories in the images to be processed, the feature association information can provide feature difference information between the segmentation category and the existing segmentation categories, thereby accurately identifying the unknown segmentation category during the image segmentation process and improving the accuracy of image segmentation.

[0097] Corresponding to the AI-based image segmentation method in the above embodiments, Figure 3 shows a structural block diagram of the AI-based image segmentation device provided in Embodiment 2 of the present invention. This image segmentation device is applied to a client, whose computer device is connected to a server to obtain the image to be processed, a reference image, and a corresponding label image from the server. The image to be processed can refer to the image received by the server that requires image segmentation processing. The reference image can refer to the image stored on the server. The label image can refer to the image obtained by manually annotating the reference image. The computer device corresponding to the client has a pre-trained convolutional model, a pre-trained association model, and a pre-trained segmentation model deployed within it. The pre-trained convolutional model can be used to extract features from the input image, the pre-trained association model can be used to calculate the correlation between features, and the pre-trained segmentation model can be used to segment the pixel categories of the features corresponding to the image. For ease of explanation, only the parts related to the embodiments of the present invention are shown.

[0098] Referring to Figure 3, the image segmentation device includes:

[0099] The first convolution module 31 is used to input the acquired image to be processed into the trained convolution model for feature extraction to obtain the first image features;

[0100] The second convolutional module 32 is used to obtain a reference image and its corresponding label image. The reference image is input into the trained convolutional model for feature extraction to obtain the second image features.

[0101] Feature association module 33 is used to input the first image features and the second image features into the trained association model to calculate the feature correlation and obtain the feature association matrix between the first image features and the second image features.

[0102] The image segmentation module 34 is used to multiply the feature correlation matrix and the label image, determine the multiplication result as the aggregated feature, input the aggregated feature into the trained segmenter for pixel category segmentation, and obtain the image segmentation result corresponding to the image to be processed.

[0103] Optionally, the first convolutional module 31 mentioned above includes:

[0104] The reference image convolutional unit is used to input the reference image into the trained convolutional model for feature extraction to obtain the second image features.

[0105] The label determination unit is used to input the second image features into the trained segmenter for image segmentation and determine the image segmentation result as a label image.

[0106] Optionally, the feature association module 33 mentioned above includes:

[0107] The feature multiplication unit is used to multiply the transpose of the second image feature and the first image feature to obtain the multiplication result, which includes at least two rows of multiplied feature values.

[0108] The normalization unit is used to normalize the eigenvalues ​​of each row in the multiplication result, and concatenates the normalization results of all rows to obtain the feature correlation matrix.

[0109] Optionally, the image segmentation module 34 described above includes:

[0110] The image dimensionality reduction unit is used to reduce the dimensionality of the label image to obtain a dimensionality-reduced image with the same feature dimensions as the first image.

[0111] The aggregated feature calculation unit is used to multiply the dimensionality-reduced image and the feature correlation matrix to obtain aggregated features.

[0112] Optionally, the second convolutional module 32 mentioned above includes:

[0113] The image reconstruction unit is used to reconstruct the reference image using a trained variational autoencoder to obtain N reconstructed images and acquire the label image corresponding to each reconstructed image, where N is an integer greater than zero.

[0114] The reconstructed image convolutional unit is used to input the reference image and N reconstructed images into the trained convolutional model for feature extraction, resulting in N+1 second image features.

[0115] Optionally, the feature association module 33 mentioned above includes:

[0116] The feature association unit is used to input the first image features and N+1 second image features into the trained association model to obtain N+1 feature association matrices;

[0117] The feature aggregation unit is used to multiply the N+1 feature association matrices by their corresponding label images to obtain N+1 aggregated features.

[0118] Optionally, the image segmentation module 34 described above includes:

[0119] The mean calculation unit is used to calculate the mean of N+1 aggregated features to obtain the mean feature;

[0120] The image segmentation unit is used to input the mean features into the trained segmenter to perform pixel category segmentation and obtain the image segmentation result corresponding to the image to be processed.

[0121] It should be noted that the information interaction and execution process between the above modules and units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0122] Figure 4 is a schematic diagram of a computer device provided in Embodiment 3 of the present invention. As shown in Figure 4, the computer device of this embodiment includes: at least one processor (only one is shown in Figure 4), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps in any of the above-described image segmentation method embodiments.

[0123] The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that Figure 4 is merely an example of a computer device and does not constitute a limitation thereof. The computer device may include more or fewer components than illustrated, or a combination of certain components, or different components, such as a network interface, a display screen, and input devices.

[0124] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0125] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0126] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0127] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0129] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0130] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image segmentation method based on artificial intelligence, characterized in that, The image segmentation method includes: inputting the acquired image to be processed into a trained convolutional model for feature extraction to obtain a first image feature; acquiring a reference image, inputting the reference image into the trained convolutional model for feature extraction to obtain a second image feature; inputting the second image feature into a trained segmenter for image segmentation, determining the image segmentation result as a label image, wherein the pixel values ​​of all pixels in the label image are of the same category as the pixels in the reference image, and pixels of the same category have the same pixel value in the label image; inputting the first image feature and the second image feature into a trained association model for feature correlation calculation to obtain... The first image feature and the second image feature are correlated in a feature association matrix. The feature association matrix is ​​multiplied by the label image to determine the aggregated feature. The aggregated feature is then input into the trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed. The step of inputting the reference image into the trained convolutional model for feature extraction to obtain the second image feature includes: reconstructing the reference image using a trained variational autoencoder to obtain N reconstructed images, and acquiring the label image corresponding to each reconstructed image, where N is a positive integer. The trained variational autoencoder includes a trained variational encoder. The reference image is input into the trained variational encoder to obtain hidden layer features. Gaussian sampling is performed on the hidden layer features to obtain the Gaussian sampling result. The Gaussian sampling result is then input into the trained variational decoder to output the reconstructed image. The reference image and the N reconstructed images are respectively input into the trained convolutional model for feature extraction to obtain N+1 second image features. The first image features and the second image features are then input into a trained correlation model for feature correlation calculation to obtain a feature correlation matrix between the first image features and the second image features. The feature correlation matrix is ​​then compared with the... The process of multiplying the labeled images to determine the aggregated features includes: inputting the first image features and the N+1 second image features into the trained association model to obtain N+1 feature association matrices; multiplying the N+1 feature association matrices with their corresponding labeled images to obtain N+1 aggregated features; and inputting the aggregated features into a trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed includes: calculating the mean of the N+1 aggregated features to obtain the mean feature; and inputting the mean feature into the trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed.

2. The image segmentation method according to claim 1, characterized in that, The step of inputting the first image feature and the second image feature into the trained association model to calculate the feature correlation matrix between the first image feature and the second image feature includes: multiplying the transpose of the second image feature with the first image feature to obtain a multiplication result, the multiplication result including at least two rows of multiplied feature values; normalizing each row of multiplied feature values ​​in the multiplication result; and concatenating the normalization results of all rows to obtain the feature correlation matrix.

3. The image segmentation method according to claim 1, characterized in that, The step of multiplying the feature association matrix and the label image to determine the multiplication result as the aggregated feature includes: reducing the dimensionality of the label image to obtain a dimensionality-reduced image with the same feature dimension as the first image; and multiplying the dimensionality-reduced image and the feature association matrix to obtain the aggregated feature.

4. An image segmentation device based on artificial intelligence, characterized in that, The image segmentation device includes: a first convolution module, used to input the acquired image to be processed into a trained convolution model for feature extraction to obtain a first image feature; a second convolution module, used to acquire a reference image, input the reference image into the trained convolution model for feature extraction to obtain a second image feature, input the second image feature into a trained segmenter for image segmentation, and determine the image segmentation result as a label image, wherein the pixel values ​​of all pixels in the label image are the same as the pixel categories of the reference image, and pixels of the same pixel category have the same pixel value in the label image; a feature association module, used to input the first image feature and the second image feature into a trained association model for feature correlation calculation to obtain a feature association matrix between the first image feature and the second image feature; an image segmentation module, used to multiply the feature association matrix and the label image to determine the multiplication result as an aggregate feature, input the aggregate feature into the trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed; the second convolution module includes: an image reconstruction unit, used to reconstruct the reference image through a trained variational autoencoder to obtain N reconstructed images, and obtain Take the label image corresponding to each reconstructed image, where N is a positive integer. The trained variational autoencoder includes a trained variational encoder and a trained variational decoder. The reference image is input into the trained variational encoder to obtain hidden layer features. Gaussian sampling is performed on the hidden layer features to obtain the Gaussian sampling result. The Gaussian sampling result is input into the trained variational decoder to output the reconstructed image. A reconstructed image convolutional unit is used to input the reference image and the N reconstructed images into the trained convolutional model for feature extraction, obtaining N+1 second image features. The association module includes: a feature association unit, used to input the first image features and the N+1 second image features into the trained association model to obtain N+1 feature association matrices; and a feature aggregation unit, used to multiply the N+1 feature association matrices by their corresponding label images to obtain N+1 aggregated features. The image segmentation module includes: a mean calculation unit, used to calculate the mean of the N+1 aggregated features to obtain mean features; and an image segmentation unit, used to input the mean features into the trained segmenter for pixel category segmentation to obtain the image segmentation result corresponding to the image to be processed.

5. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image segmentation method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the image segmentation method as described in any one of claims 1 to 3.

Images