A method and device for fine segmentation of image sites based on deep learning

Through the deep learning-based image site fine segmentation method, the image enhancement model and Mask R-CNN model are used to solve the problem of poor image segmentation effect in the existing technology, and high-precision and automated site fine segmentation are achieved. .

CN118823040BActive Publication Date: 2024-11-26GUIZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411315008.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-20
Publication Date
2024-11-26
Estimated Expiration
2044-09-20

AI Technical Summary

Technical Problem

The prior art is difficult to achieve high precision and consistency in image site fine segmentation, especially in complex or blurred image areas, and requires manual participation in feature selection and parameter adjustment.

Method used

Using a deep learning-based method, the image enhancement model is constructed, the reconstruction loss function and perceived loss function are defined, the feature extraction is performed using the ResNeXt-50 and ResNeXt-101 network structures, and the segmentation is combined with the Mask R-CNN model to achieve automation and high quality. Accurate site refinement segmentation.

Benefits of technology

It improves the accuracy and robustness of image segmentation, reduces manual intervention, and can achieve efficient and accurate site fine segmentation in complex images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118823040B_ABST
    Figure CN118823040B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image processing, and a method and device for fine segmentation of image sites based on deep learning, comprising: constructing a ResNeXt-50 network structure and a ResNeXt-101 network structure for enhancing an image to be segmented, configuring 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure, training the ResNeXt-50 network structure and the ResNeXt-101 network structure to obtain a ResNeXt-50 model and a ResNeXt-101 model; establishing a feature ResNeXt extraction model for enhancing an image to be segmented, extracting features from the enhanced image to be segmented, and obtaining a segmentation feature map of the enhanced image to be segmented; establishing a Mask R-CNN model for enhancing an image to be segmented, calculating the intersection-over-union ratio of the Mask R-CNN model, and performing fine segmentation on the enhanced image to be segmented using the Mask R-CNN model to obtain a fine segmentation result of the enhanced image to be segmented. The present invention can improve the effect of fine segmentation of an image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and device for fine-grained segmentation of image sites based on deep learning. Background Art

[0002] Image site fine-grained segmentation refers to the process of accurately and meticulously dividing and identifying specific points of interest or areas (sites) in an image. Image site fine-grained segmentation can achieve accurate identification and contour depiction of every tiny detail in the image to be segmented, so that the segmentation result not only accurately divides the main objects in the image, but also can be refined to edge details, texture structures and internal tiny structures, thereby providing high-precision and high-reliability basic data for subsequent image analysis and understanding, greatly improving the practicality and application value of image processing.

[0003] Traditional methods for fine-grained segmentation of image sites usually rely on manual or semi-automatic image processing techniques, such as edge detection, region growing, watershed transform, etc. These methods often require manual participation in feature selection and parameter adjustment and cannot adaptively learn image features, resulting in poor segmentation effects in complex or blurred image areas, making it difficult to achieve high-precision and consistent site fine-grained segmentation. Summary of the invention

[0004] The present invention provides a method and device for fine-grained image segmentation based on deep learning, the main purpose of which is to improve the effect of fine-grained image segmentation.

[0005] To achieve the above object, the present invention provides a method for fine-grained image site segmentation based on deep learning, comprising:

[0006] Acquire an image enhancement training set of the image to be segmented, determine the enhancement model architecture of the image to be segmented, and define a reconstruction loss function and a perception loss function of the enhancement model architecture, and construct an image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set;

[0007] Based on the image enhancement model, image optimization is performed on the image to be segmented to obtain an enhanced image to be segmented, a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented are constructed, and 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure are configured;

[0008] According to the 50 structural hyperparameters and the 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained to obtain the ResNeXt-50 model and the ResNeXt-101 model;

[0009] Establishing a feature ResNeXt extraction model for the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and performing feature extraction on the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented;

[0010] Determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection-and-union ratio of the Mask R-CNN model, and when the intersection-and-union ratio meets the preset intersection-and-union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.

[0011] Optionally, constructing the image enhancement model of the image to be segmented by using the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set includes:

[0012] Calculate the image effective coefficient of the image enhancement training set;

[0013] Extracting a target enhanced training image of the image enhancement training set according to the image effective coefficient;

[0014] Initializing the enhanced model architecture to obtain an initial enhanced model;

[0015] Determine an Adam optimizer for the initial enhanced model, and define optimizer parameters of the Adam optimizer;

[0016] Outputting a reconstructed image of a target enhanced training image using the initial enhancement model;

[0017] Calculating the reconstruction loss of the reconstructed image using the reconstruction loss function;

[0018] Calculating the perceptual loss of the reconstructed image by the perceptual loss function;

[0019] According to the reconstruction loss and the perceptual loss, the initial enhancement model is optimized using the Adam optimizer and optimizer parameters to obtain an image enhancement model of the image to be segmented.

[0020] Optionally, calculating the perceptual loss of the reconstructed image by using the perceptual loss function includes:

[0021] Constructing a feature extractor of the perceptual loss function and defining an image feature layer of the feature extractor;

[0022] Based on the image feature layer, determining a feature vector of the image feature layer;

[0023] According to the feature extractor, the image feature layer and the feature vector, the perceptual loss of the reconstructed image is calculated using the following formula:

[0024]

[0025] in, represents the perceptual loss of the reconstructed image, represents the number of image feature layers of the reconstructed image, represents the feature extractor, Represents the reconstructed image The feature vector of the image feature layer, Indicates that the reconstructed image corresponds to the target enhanced training image The feature vector of the image feature layer, Represents the reconstructed image The feature vector of the image feature layer and the reconstructed image correspond to the target enhanced training image The L1 norm of the feature vector of the image feature layer.

[0026] Optionally, optimizing the initial enhancement model using the Adam optimizer and optimizer parameters according to the reconstruction loss and the perceptual loss to obtain the image enhancement model of the image to be segmented includes:

[0027] Extracting initial enhancement model parameters of the initial enhancement model;

[0028] Calculating the total image loss of the image to be segmented according to the reconstruction loss and the perception loss;

[0029] Calculating the loss gradient of the total image loss with respect to the initial enhancement model parameters;

[0030] Based on the loss gradient and optimizer parameters, the first-order moment estimate and the second-order moment estimate of the Adam optimizer are calculated using the following function:

[0031]

[0032]

[0033] in, Represents the Adam optimizer The first-order moment estimate of the iteration, Represents the Adam optimizer The second-order moment estimate of the iteration, represents the exponential decay rate of the first-order moment estimate of the optimizer parameters, represents the exponential decay rate of the optimizer parameter corresponding to the second-order moment estimate, Represents the Adam optimizer The first-order moment estimate of the iteration, Represents the Adam optimizer The second-order moment estimate of the iteration, represents the loss gradient;

[0034] The initial enhancement model is optimized according to the first-order moment estimation and the second-order moment estimation to obtain an image enhancement model of the image to be segmented.

[0035] Optionally, optimizing the initial enhancement model according to the first-order moment estimation and the second-order moment estimation to obtain the image enhancement model of the image to be segmented includes:

[0036] Calculating a corrected first-order moment estimate and a corrected second-order moment estimate of the first-order moment estimate and the second-order moment estimate;

[0037] Based on the corrected first-order moment estimation and the corrected second-order moment estimation, the model update parameters of the initial enhanced model are analyzed using the following formula:

[0038]

[0039] in, represents the initial enhancement model The model update parameters for the iteration, represents the initial enhancement model The model update parameters for the iteration, represents the numerical stability constant, represents the learning rate of the initial enhancement model, represents the corrected second-order moment estimate, represents the corrected first-order moment estimate, Indicates The corrected second-order moment estimate of the iteration corresponds to the exponential decay rate of the second-order moment estimate;

[0040] The initial enhancement model is optimized according to the model update parameters to obtain the image enhancement model of the image to be segmented.

[0041] Optionally, the training of the ResNeXt-50 network structure and the ResNeXt-101 network structure according to the 50 structure hyperparameters and the 101 structure hyperparameters to obtain a training ResNeXt-50 model and a training ResNeXt-101 model includes:

[0042] Constructing a meta-learning algorithm for the ResNeXt-50 network structure and the ResNeXt-101 network structure;

[0043] Analyze the image category of the image to be segmented corresponding to the ResNeXt-50 network structure;

[0044] Obtaining a single example training set and an unseen example test set of the image category;

[0045] According to the single example training set, 50 structural hyperparameters, and 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained using the meta-learning algorithm to obtain a training ResNeXt-50 model and a training ResNeXt-101 model;

[0046] Using the unseen example test set, calculating the 50 model test results and the 101 model test results of the trained ResNeXt-50 model and the trained ResNeXt-101 model;

[0047] Constructing a 50 model accuracy curve and a 101 model accuracy curve of the ResNeXt-50 model through the 50 model test results and the 101 model test results;

[0048] Based on the 50 model accuracy curve and the 101 model accuracy curve, analyze the 50 model performance and the 101 model performance of the training ResNeXt-50 model and the training ResNeXt-101 model;

[0049] When the 50 model performance and the 101 model performance meet the preset 50 model performance threshold and the 101 model performance threshold, the trained ResNeXt-50 model and the trained ResNeXt-101 model are used as the ResNeXt-50 model and the trained ResNeXt-101 model for enhancing the image to be segmented.

[0050] Optionally, the extracting features of the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented includes:

[0051] Based on the feature ResNeXt extraction model, extracting a 50 model feature map and a 101 model feature map of the enhanced image to be segmented;

[0052] Calculating the size difference between the 50-model feature map and the 101-model feature map;

[0053] Calculate the pooling layer parameters of the 50-model feature map and the 101-model feature map according to the size difference;

[0054] According to the pooling layer parameters, the 50 model feature map and the 101 model feature map are pooled to obtain a pooled 50 model feature map and a pooled 101 model feature map;

[0055] Identify the 50 model feature map channels and the 101 model feature map channels of the pooled 50 model feature map and the pooled 101 model feature map;

[0056] Based on the 50 model feature map channels and the 101 model feature map channels, the pooled 50 model feature map and the pooled 101 model feature map are element-by-element multiplied to obtain the segmentation feature map of the enhanced image to be segmented.

[0057] Optionally, calculating the intersection-over-union ratio of the Mask R-CNN model includes:

[0058] Collecting a test feature map of the Mask R-CNN model and defining a true mask of the test feature map;

[0059] Constructing an upsampling layer of the test feature map, and upsampling the test feature map based on the upsampling layer to obtain an upsampled feature map;

[0060] Adding a classification layer of the upsampled feature map, and predicting pixel categories of the upsampled feature map based on the classification layer;

[0061] Calculating the category score of the pixel category using a Sigmoid function;

[0062] Determining a prediction mask of the enhanced image to be segmented according to the category score and the upsampled feature map;

[0063] The intersection-over-union (IoU) ratio of the Mask R-CNN model is calculated based on the predicted mask and the true mask.

[0064] Optionally, calculating the intersection-over-union ratio of the Mask R-CNN model according to the predicted mask and the true mask includes:

[0065] Unifying the sizes of the predicted mask and the real mask to obtain a predicted mask and a real mask of the same size;

[0066] The mask intersection value of the predicted mask with the same size and the real mask with the same size is calculated using the following formula:

[0067]

[0068] in, Represents the mask intersection value of the predicted mask of the same size and the real mask of the same size, Represents the height of the prediction mask of the same size, represents the width of the prediction mask of the same size, represents the mask intersection function of the predicted mask of the same size and the real mask of the same size, Indicates the same size prediction mask at position The value of Indicates the same size prediction mask at position The value corresponding to the real mask of the same size;

[0069] The mask union value of the predicted mask with the same size and the real mask with the same size is calculated using the following formula:

[0070]

[0071] in, represents the mask union value of the predicted mask of the same size and the real mask of the same size, Represents the height of the prediction mask of the same size, represents the width of the prediction mask of the same size, represents the mask union function of the predicted mask of the same size and the real mask of the same size, Indicates the same size prediction mask at position The value of Indicates the same size prediction mask at position The value corresponding to the real mask of the same size;

[0072] Based on the mask intersection value and the mask union value, the intersection-over-union ratio of the Mask R-CNN model is calculated.

[0073] To achieve the above object, the present invention also provides an image site refined segmentation device based on deep learning, comprising:

[0074] An image enhancement model construction module is used to obtain an image enhancement training set of the image to be segmented, determine the enhancement model architecture of the image to be segmented, and define a reconstruction loss function and a perception loss function of the enhancement model architecture, and construct an image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set;

[0075] A structural hyperparameter determination module is used to perform image optimization on the image to be segmented based on the image enhancement model to obtain an enhanced image to be segmented, construct a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented, and configure 50 structural hyperparameters and 101 structural hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure;

[0076] A ResNeXt model construction module, used to train the ResNeXt-50 network structure and the ResNeXt-101 network structure according to the 50 structure hyperparameters and the 101 structure hyperparameters to obtain a ResNeXt-50 model and a ResNeXt-101 model;

[0077] A segmentation feature map acquisition module is used to establish a feature ResNeXt extraction model of the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and based on the feature ResNeXt extraction model, perform feature extraction on the enhanced image to be segmented to obtain a segmentation feature map of the enhanced image to be segmented;

[0078] A refined segmentation module is used to determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection and union ratio of the Mask R-CNN model, and when the intersection and union ratio meets the preset intersection and union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.

[0079] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0080] A memory storing at least one instruction;

[0081] A processor executes instructions stored in the memory to implement the above-mentioned image site refined segmentation method based on deep learning.

[0082] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned deep learning-based image site refined segmentation method.

[0083] The present invention defines the extracted high-level features to measure the difference between the reconstructed image and the original image. The reconstruction loss function and the perception loss function of the enhancement model architecture can enable the model to optimize the quality and perception characteristics of the reconstructed image during the training process. The present invention constructs the image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set, which can improve the quality and clarity of the image, improve the feature extraction process, and improve the accuracy and robustness of the segmentation through the optimization of the reconstruction loss and the perception loss. Furthermore, the present invention optimizes the ResNeXt-50 network structure and the ResNeXt-50 network structure according to the 50 structural hyperparameters and the 101 structural hyperparameters. The t-101 network structure is trained to obtain the training ResNeXt-50 model and the training ResNeXt-101 model, which can improve the model's adaptability to new tasks, thereby performing better in practical applications. Furthermore, the present invention establishes the feature ResNeXt extraction model of the enhanced image to be segmented based on the ResNeXt-50 model and the ResNeXt-101 model, which can make full use of the feature representations of the two models at different levels, and helps the model to more accurately identify and locate different objects and structures in the image in the refined segmentation task. Furthermore, based on the target detection framework and the mask branch, the present invention establishes the Mask R-CNN model of the enhanced image to be segmented, which integrates the capabilities of target detection and segmentation, and can simultaneously locate the objects in the feature map output by the feature ResNeXt extraction model and generate accurate segmentation masks. This integrated processing improves the accuracy and efficiency of segmentation. Finally, when the IOU meets the preset IOU threshold, the present invention uses the Mask R-CNN model to perform refined segmentation on the enhanced image to be segmented according to the segmentation feature map, and obtains the refined segmentation result of the enhanced image to be segmented, which can realize the refined segmentation of the image. Therefore, the present invention can improve the effect of image fine segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 A schematic diagram of a process for fine-grained image site segmentation based on deep learning provided by an embodiment of the present invention;

[0085] Figure 2 A functional module diagram of a device for fine segmentation of image sites based on deep learning provided by an embodiment of the present invention;

[0086] Figure 3 A schematic diagram of the structure of an electronic device for implementing the deep learning-based image site refined segmentation method provided in one embodiment of the present invention.

[0087] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0088] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0089] The embodiment of the present application provides a method for fine-grained segmentation of image sites based on deep learning. The execution subject of the method for fine-grained segmentation of image sites based on deep learning includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for fine-grained segmentation of image sites based on deep learning can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0090] Reference Figure 1 FIG. 1 is a flow chart of a method for fine-grained segmentation of image sites based on deep learning provided by an embodiment of the present invention. In this embodiment, the method for fine-grained segmentation of image sites based on deep learning includes:

[0091] S1. Obtain an image enhancement training set of the image to be segmented, determine the enhancement model architecture of the image to be segmented, and define the reconstruction loss function and the perception loss function of the enhancement model architecture. Through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set, construct the image enhancement model of the image to be segmented.

[0092] It should be explained that the image enhancement training set refers to a set of image data sets prepared for training the image enhancement model. The image enhancement training set includes images with different lighting, angles, occlusions, noise and corresponding annotation information. The enhancement model architecture refers to a deep learning model used to process the original image and generate an enhanced version. The enhancement model architecture usually includes an encoder-decoder structure, in which the encoder is used to extract low-level features of the image and the decoder is used to reconstruct the image.

[0093] The present invention defines the reconstruction loss function and the perceptual loss function of the model architecture to measure the difference between the reconstructed image and the original image by using the extracted high-level features, so that the model can simultaneously optimize the quality and perceptual characteristics of the reconstructed image during the training process. The reconstruction loss function refers to a function used to measure the difference between the model output (reconstructed image) and the original image, and the perceptual loss function refers to a function used to measure the difference between the reconstructed image and the original image by using the extracted high-level features.

[0094] The present invention constructs the image enhancement model of the image to be segmented through the enhancement model architecture, reconstruction loss function, perception loss function and image enhancement training set, which can improve the quality and clarity of the image, improve the feature extraction process, and improve the accuracy and robustness of segmentation by optimizing the reconstruction loss and the perception loss.

[0095] In detail, the image enhancement model of the image to be segmented is constructed by using the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set, including:

[0096] Calculate the image effective coefficient of the image enhancement training set;

[0097] Extracting a target enhanced training image of the image enhancement training set according to the image effective coefficient;

[0098] Initializing the enhanced model architecture to obtain an initial enhanced model;

[0099] Determine an Adam optimizer for the initial enhanced model, and define optimizer parameters of the Adam optimizer;

[0100] Outputting a reconstructed image of a target enhanced training image using the initial enhancement model;

[0101] Calculating the reconstruction loss of the reconstructed image using the reconstruction loss function;

[0102] Calculating the perceptual loss of the reconstructed image by the perceptual loss function;

[0103] According to the reconstruction loss and the perceptual loss, the initial enhancement model is optimized using the Adam optimizer and optimizer parameters to obtain an image enhancement model of the image to be segmented.

[0104] Among them, the image effective coefficient refers to a parameter used to evaluate the enhancement potential of each image in the image enhancement training set or its importance to the training process, the target enhanced training image refers to an image subset screened according to the image effective coefficient and used to train the image enhancement model, the initial enhancement model refers to a model obtained after initializing the enhancement model architecture before starting the training process, the Adam optimizer refers to an optimizer used to optimize the results of the deep learning model training process, the optimizer parameters refer to parameters related to the Adam optimizer, including the learning rate (learningrate), the exponential decay rate of the first-order moment estimate (beta1), the exponential decay rate of the second-order moment estimate (beta2) and a small constant (epsilon) for numerical stability, the reconstruction loss refers to an indicator that measures the difference between the reconstructed image and the original image, the perceptual loss refers to a measure of the quality loss of the reconstructed image at the perceptual level, and the image enhancement model refers to a model obtained after training, which can receive the original image as input and output a model of an enhanced image with higher quality.

[0105] Further, the calculating the perceptual loss of the reconstructed image by using the perceptual loss function includes:

[0106] Constructing a feature extractor of the perceptual loss function and defining an image feature layer of the feature extractor;

[0107] Based on the image feature layer, determining a feature vector of the image feature layer;

[0108] According to the feature extractor, the image feature layer and the feature vector, the perceptual loss of the reconstructed image is calculated using the following formula:

[0109]

[0110] in, represents the perceptual loss of the reconstructed image, represents the number of image feature layers of the reconstructed image, represents the feature extractor, Represents the reconstructed image The feature vector of the image feature layer, Indicates that the reconstructed image corresponds to the target enhanced training image The feature vector of the image feature layer, Represents the reconstructed image The feature vector of the image feature layer and the reconstructed image correspond to the target enhanced training image The L1 norm of the feature vector of the image feature layer.

[0111] Among them, the feature extractor refers to extracting high-level features from the input image, the image feature layer refers to a specific layer inside the feature extractor for extracting image features, the feature vector refers to a one-dimensional vector extracted from the image feature layer, and the L1 norm refers to an indicator of the difference between the feature vectors of the reconstructed image and the target enhanced training image.

[0112] Furthermore, the initial enhancement model is optimized using the Adam optimizer and optimizer parameters according to the reconstruction loss and the perception loss to obtain the image enhancement model of the image to be segmented, including:

[0113] Extracting initial enhancement model parameters of the initial enhancement model;

[0114] Calculating the total image loss of the image to be segmented according to the reconstruction loss and the perception loss;

[0115] Calculating the loss gradient of the total image loss with respect to the initial enhancement model parameters;

[0116] Based on the loss gradient and optimizer parameters, the first-order moment estimate and the second-order moment estimate of the Adam optimizer are calculated using the following function:

[0117]

[0118]

[0119] in, Represents the Adam optimizer The first-order moment estimate of the iteration, Represents the Adam optimizer The second-order moment estimate of the iteration, represents the exponential decay rate of the first-order moment estimate of the optimizer parameters, represents the exponential decay rate of the optimizer parameter corresponding to the second-order moment estimate, Represents the Adam optimizer The first-order moment estimate of the iteration, Represents the Adam optimizer The second-order moment estimate of the iteration, represents the loss gradient;

[0120] The initial enhancement model is optimized according to the first-order moment estimation and the second-order moment estimation to obtain an image enhancement model of the image to be segmented.

[0121] Among them, the initial enhancement model parameters refer to the model parameters set when initializing the enhancement model before starting training, the total image loss refers to the total loss obtained by adding the reconstruction loss and the perceptual loss during the training process, the loss gradient refers to the gradient of the loss function with respect to the model parameters, which indicates which parameters need to be adjusted to minimize the loss, the first-order moment estimate refers to the first-order moment estimate containing uncorrected gradient information during the update, and the second-order moment estimate refers to the second-order moment estimate containing uncorrected gradient information during the update.

[0122] Furthermore, the initial enhancement model is optimized according to the first-order moment estimation and the second-order moment estimation to obtain the image enhancement model of the image to be segmented, including:

[0123] Calculating a corrected first-order moment estimate and a corrected second-order moment estimate of the first-order moment estimate and the second-order moment estimate;

[0124] Based on the corrected first-order moment estimation and the corrected second-order moment estimation, the model update parameters of the initial enhanced model are analyzed using the following formula:

[0125]

[0126] in, represents the initial enhancement model The model update parameters for the iteration, represents the initial enhancement model The model update parameters for the iteration, represents the numerical stability constant, represents the learning rate of the initial enhancement model, represents the corrected second-order moment estimate, represents the corrected first-order moment estimate, Indicates The corrected second-order moment estimate of the iteration corresponds to the exponential decay rate of the second-order moment estimate;

[0127] The initial enhancement model is optimized according to the model update parameters to obtain the image enhancement model of the image to be segmented.

[0128] Among them, the corrected first-order moment estimate refers to the value obtained by performing bias correction on the first-order moment estimate, the corrected second-order moment estimate refers to the value obtained by performing bias correction on the second-order moment estimate, the model update parameter refers to the parameter after the model is updated, the numerical stability constant refers to the constant used to maintain the stability of the algorithm when the denominator is zero, the learning rate of the initial enhancement model refers to the parameter used to control the step size of updating the enhancement model parameters, the initial image enhancement model refers to the model obtained through the optimization process, the enhancement performance refers to the performance of the model in enhancing image quality, and the enhancement performance threshold refers to the threshold used to judge whether the enhancement performance has reached the expected level.

[0129] S2. Based on the image enhancement model, image optimization is performed on the image to be segmented to obtain an enhanced image to be segmented, a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented are constructed, and the 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure are configured.

[0130] It should be explained that the enhanced image to be segmented refers to the image to be segmented after being processed by the image enhancement model, the ResNeXt-50 network structure refers to a ResNeXt model with 50 layers, and the ResNeXt-101 network structure refers to a ResNeXt model with 101 layers.

[0131] The 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure configured by the present invention can provide a basis for the later feature extraction model training. Among them, the 50 structure hyperparameters and 101 structure hyperparameters both include parameters such as feature extraction model learning rate, batch size, and number of training rounds.

[0132] S3. According to the 50 structural hyperparameters and the 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained to obtain the ResNeXt-50 model and the ResNeXt-101 model.

[0133] The present invention trains the ResNeXt-50 network structure and the ResNeXt-101 network structure according to the 50 structural hyperparameters and the 101 structural hyperparameters, and obtains the training ResNeXt-50 model and the training ResNeXt-101 model, which can improve the model's adaptability to new tasks, thereby performing better in practical applications.

[0134] In detail, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained according to the 50 structure hyperparameters and the 101 structure hyperparameters to obtain a training ResNeXt-50 model and a training ResNeXt-101 model, including:

[0135] Constructing a meta-learning algorithm for the ResNeXt-50 network structure and the ResNeXt-101 network structure;

[0136] Analyze the image category of the image to be segmented corresponding to the ResNeXt-50 network structure;

[0137] Obtaining a single example training set and an unseen example test set of the image category;

[0138] According to the single example training set, 50 structural hyperparameters, and 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained using the meta-learning algorithm to obtain a training ResNeXt-50 model and a training ResNeXt-101 model;

[0139] Using the unseen example test set, calculating the 50 model test results and the 101 model test results of the trained ResNeXt-50 model and the trained ResNeXt-101 model;

[0140] Constructing a 50 model accuracy curve and a 101 model accuracy curve of the ResNeXt-50 model through the 50 model test results and the 101 model test results;

[0141] Based on the 50 model accuracy curve and the 101 model accuracy curve, analyze the 50 model performance and the 101 model performance of the training ResNeXt-50 model and the training ResNeXt-101 model;

[0142] When the 50 model performance and the 101 model performance meet the preset 50 model performance threshold and the 101 model performance threshold, the trained ResNeXt-50 model and the trained ResNeXt-101 model are used as the ResNeXt-50 model and the trained ResNeXt-101 model for enhancing the image to be segmented.

[0143] Among them, the meta-learning algorithm refers to a learning algorithm that can help the model learn new categories by observing only one or several examples, such as MAML, Prototypical Networks and other algorithms; the image category refers to different objects or structures existing in the image to be segmented; the single example training set refers to a training set with only one example image for each image category; the unseen example test set refers to a test set containing multiple example images of each category that have not been seen in the training set; the trained ResNeXt-50 model refers to the initial ResNeXt-50 model obtained by training with the meta-learning algorithm and the single sample training set; the 50 model test result; the trained ResNeXt-101 model refers to the initial ResNeXt-101 model obtained by training with the meta-learning algorithm and the single sample training set; the 101 model test result refers to the result obtained by testing the trained ResNeXt-50 model on the unseen example test set; the 50 model accuracy curve refers to the accuracy of the trained ResNeXt-50 model on the unseen example test set over time The 101 model accuracy curve refers to the accuracy curve of the training ResNeXt-101 model on the unseen example test set over time, the 50 model performance refers to the performance of the training ResNeXt-50 model on the unseen example test set, the 101 model performance refers to the performance of the training ResNeXt-101 model on the unseen example test set, the 50 model performance threshold refers to the value used to judge whether the performance of the training ResNeXt-50 model on the unseen example test set has reached the expected level, the 101 model performance threshold refers to the value used to judge whether the performance of the training ResNeXt-101 model on the unseen example test set has reached the expected level, the ResNeXt-50 model refers to a ResNeXt model with a 50-layer network structure, and the training ResNeXt-101 model refers to a ResNeXt model with a 101-layer network structure.

[0144] S4. Establish a feature ResNeXt extraction model for the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and perform feature extraction on the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented.

[0145] The present invention establishes the feature ResNeXt extraction model for enhancing the image to be segmented based on the ResNeXt-50 model and the ResNeXt-101 model, which can make full use of the feature representations of the two models at different levels, and help the model to more accurately identify and locate different objects and structures in the image in the refined segmentation task. In detail, the feature ResNeXt extraction model can be integrated by splicing the feature extraction layers of the ResNeXt-50 model and the ResNeXt-101 model through a deep learning framework.

[0146] In detail, the feature extraction of the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented includes:

[0147] Based on the feature ResNeXt extraction model, extracting the 50 model feature map and the 101 model feature map of the enhanced image to be segmented;

[0148] Calculating the size difference between the 50-model feature map and the 101-model feature map;

[0149] Calculate the pooling layer parameters of the 50-model feature map and the 101-model feature map according to the size difference;

[0150] According to the pooling layer parameters, the 50 model feature map and the 101 model feature map are pooled to obtain a pooled 50 model feature map and a pooled 101 model feature map;

[0151] Identify the 50 model feature map channels and the 101 model feature map channels of the pooled 50 model feature map and the pooled 101 model feature map;

[0152] Based on the 50 model feature map channels and the 101 model feature map channels, the pooled 50 model feature map and the pooled 101 model feature map are element-by-element multiplied to obtain the segmentation feature map of the enhanced image to be segmented.

[0153] Among them, the 50 model feature map refers to the features extracted from the input image by the ResNeXt-50 model, the 101 model feature map refers to the features extracted from the input image by the ResNeXt-101 model, the size difference refers to the size difference between the two feature maps in the height and width dimensions, the pooling layer parameters include the kernel size (kernelsize) and the stride (stride) of the pooling layer, the pooled 50 model feature map refers to the feature map whose size is adjusted to a preset size by applying the pooling layer, the pooled 101 model feature map refers to the feature map whose size is adjusted to a preset size by applying the pooling layer, the 50 model feature map channel refers to the channel dimension of the ResNeXt-50 model feature map, the 101 model feature map channel refers to the channel dimension of the ResNeXt-101 model feature map, and the segmentation feature map refers to the fused feature map.

[0154] S5. Determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection-and-union ratio of the Mask R-CNN model, and when the intersection-and-union ratio meets the preset intersection-and-union ratio threshold, segment the enhanced image to be segmented according to the segmentation feature map using the Mask R-CNN model to obtain a refined segmentation result of the enhanced image to be segmented.

[0155] The present invention is based on the target detection framework and the mask branch, and establishes the Mask R-CNN model for enhancing the image to be segmented, which integrates the capabilities of target detection and segmentation, can simultaneously locate the objects in the feature map output by the feature ResNeXt extraction model and generate accurate segmentation masks. This integrated processing improves the accuracy and efficiency of segmentation.

[0156] It should be explained that the target detection framework refers to a framework for detecting objects in an image, the mask branch refers to a branch for generating a pixel-level segmentation mask for each detected object, and the Mask R-CNN model refers to a model for generating refined segmentation of objects.

[0157] The present invention can determine whether the model performance meets the subsequent image segmentation task by calculating the intersection-over-union ratio of the Mask R-CNN model, thereby improving the effect of image fine segmentation.

[0158] In detail, the calculation of the intersection-over-union ratio of the Mask R-CNN model includes:

[0159] Collecting a test feature map of the Mask R-CNN model and defining a true mask of the test feature map;

[0160] Constructing an upsampling layer of the test feature map, and upsampling the test feature map based on the upsampling layer to obtain an upsampled feature map;

[0161] Adding a classification layer of the upsampled feature map, and predicting pixel categories of the upsampled feature map based on the classification layer;

[0162] Calculating the category score of the pixel category using a Sigmoid function;

[0163] Determining a prediction mask of the enhanced image to be segmented according to the category score and the upsampled feature map;

[0164] The intersection-over-union (IoU) ratio of the Mask R-CNN model is calculated based on the predicted mask and the true mask.

[0165] Among them, the predicted mask refers to the position and category value of the target in the image predicted by the model, the position and category value of the target actually existing in the real mask image, the same-size predicted mask refers to the predicted mask after size adjustment, the test feature map refers to the feature map used to test the intersection and union ratio of the Mask R-CNN model, the upsampling layer refers to the layer used to upsample the feature map to the original image size, the upsampled feature map refers to the feature map processed by the upsampling layer, the classification layer refers to the layer used to predict which category each pixel in the image belongs to, the pixel category refers to the category into which each pixel in the image is classified, the category score refers to the probability that the classification layer predicts that each pixel belongs to each category, the Sigmoid function refers to a nonlinear function used to convert the output of the classification layer into a probability value, and the intersection and union ratio refers to a measure of the degree of match between the predicted mask and the true mask.

[0166] Further, the calculating the intersection-over-union ratio of the Mask R-CNN model according to the predicted mask and the real mask includes:

[0167] Unifying the sizes of the predicted mask and the real mask to obtain a predicted mask and a real mask of the same size;

[0168] The mask intersection value of the predicted mask with the same size and the real mask with the same size is calculated using the following formula:

[0169]

[0170] in, Represents the mask intersection value of the predicted mask of the same size and the real mask of the same size, Represents the height of the prediction mask of the same size, represents the width of the prediction mask of the same size, represents the mask intersection function of the predicted mask of the same size and the real mask of the same size, Indicates the same size prediction mask at position The value of Indicates the same size prediction mask at position The value corresponding to the real mask of the same size;

[0171] The mask union value of the predicted mask with the same size and the real mask with the same size is calculated using the following formula:

[0172]

[0173] in, represents the mask union value of the predicted mask of the same size and the real mask of the same size, Represents the height of the prediction mask of the same size, represents the width of the prediction mask of the same size, represents the mask union function of the predicted mask of the same size and the real mask of the same size, Indicates the same size prediction mask at position The value of Indicates the same size prediction mask at position The value corresponding to the real mask of the same size;

[0174] Based on the mask intersection value and the mask union value, the intersection-over-union ratio of the Mask R-CNN model is calculated.

[0175] Among them, the size unification refers to adjusting the sizes of the two masks to the same size, which ensures that the predicted results and the true results are compared at the same scale, so that the performance of the model can be evaluated more accurately. The same-size true mask refers to the true mask after size adjustment. The mask intersection value refers to the number of pixels in the predicted mask and the true mask that are at least one marked as the same category. The mask union value refers to the number of pixels in the predicted mask and the true mask that are simultaneously marked as the same category.

[0176] When the IoU meets a preset IoU threshold, the Mask R-CNN model is used to segment the enhanced image to be segmented according to the segmentation feature map, and a refined segmentation result of the enhanced image to be segmented is obtained to achieve refined segmentation of the image.

[0177] In detail, the segmentation of the enhanced image to be segmented using the Mask R-CNN model according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented includes: using the Mask R-CNN model to generate a segmentation mask of the enhanced image to be segmented, extracting the object contour from the segmentation mask using contour extraction technology, and performing refined segmentation on the enhanced image to be segmented according to the object contour. The segmentation mask refers to a binary image generated by the Mask R-CNN model according to the segmentation feature map, the object contour is a set of object edges extracted from the segmentation mask, and the refined segmentation result includes a bounding box, a segmentation mask, a category label, a category score, and other content.

[0178] The present invention defines the extracted high-level features to measure the difference between the reconstructed image and the original image. The reconstruction loss function and the perception loss function of the enhancement model architecture can enable the model to optimize the quality and perception characteristics of the reconstructed image during the training process. The present invention constructs the image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set, which can improve the quality and clarity of the image, improve the feature extraction process, and improve the accuracy and robustness of the segmentation through the optimization of the reconstruction loss and the perception loss. Furthermore, the present invention optimizes the ResNeXt-50 network structure and the ResNeXt-50 network structure according to the 50 structural hyperparameters and the 101 structural hyperparameters. The t-101 network structure is trained to obtain the training ResNeXt-50 model and the training ResNeXt-101 model, which can improve the model's adaptability to new tasks, thereby performing better in practical applications. Furthermore, the present invention establishes the feature ResNeXt extraction model of the enhanced image to be segmented based on the ResNeXt-50 model and the ResNeXt-101 model, which can make full use of the feature representations of the two models at different levels, and helps the model to more accurately identify and locate different objects and structures in the image in the refined segmentation task. Furthermore, based on the target detection framework and the mask branch, the present invention establishes the Mask R-CNN model of the enhanced image to be segmented, which integrates the capabilities of target detection and segmentation, and can simultaneously locate the objects in the feature map output by the feature ResNeXt extraction model and generate accurate segmentation masks. This integrated processing improves the accuracy and efficiency of segmentation. Finally, when the IOU meets the preset IOU threshold, the present invention uses the Mask R-CNN model to perform refined segmentation on the enhanced image to be segmented according to the segmentation feature map, and obtains the refined segmentation result of the enhanced image to be segmented, which can realize the refined segmentation of the image. Therefore, the present invention can improve the effect of image fine segmentation.

[0179] like Figure 2, is a functional module diagram of an image site fine segmentation device based on deep learning provided by an embodiment of the present invention.

[0180] The image site fine segmentation device 100 based on deep learning of the present invention can be installed in an electronic device. According to the functions implemented, the image site fine segmentation device 100 based on deep learning can include an image enhancement model construction module 101, a structural hyperparameter determination module 102, a ResNeXt model construction module 103, a segmentation feature map acquisition module 104 and a fine segmentation module 105. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, which are stored in the memory of the electronic device.

[0181] The image enhancement model construction module 101 is used to obtain an image enhancement training set of the image to be segmented, determine the enhancement model architecture of the image to be segmented, and define a reconstruction loss function and a perception loss function of the enhancement model architecture, and construct an image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set;

[0182] The structural hyperparameter determination module 102 is used to optimize the image to be segmented based on the image enhancement model to obtain an enhanced image to be segmented, construct a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented, and configure the 50 structural hyperparameters and 101 structural hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure;

[0183] The ResNeXt model construction module 103 is used to train the ResNeXt-50 network structure and the ResNeXt-101 network structure according to the 50 structure hyperparameters and the 101 structure hyperparameters to obtain the ResNeXt-50 model and the ResNeXt-101 model;

[0184] The segmentation feature map acquisition module 104 is used to establish a feature ResNeXt extraction model of the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and perform feature extraction on the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented;

[0185] The refined segmentation module 105 is used to determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection and union ratio of the Mask R-CNN model, and when the intersection and union ratio meets the preset intersection and union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.

[0186] In detail, each module in the image site fine segmentation device 100 based on deep learning in the embodiment of the present invention is used in the same manner as above. Figure 1 The same technical means are used as the image site refined segmentation method based on deep learning described in , and can produce the same technical effects, so I will not go into details here.

[0187] like Figure 3 , is a schematic diagram of the structure of an electronic device for implementing a deep learning-based image site refinement segmentation method provided by an embodiment of the present invention.

[0188] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a deep learning-based image site refinement segmentation method program.

[0189] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 also includes an internal storage unit of the electronic device 1 and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the image site refinement segmentation method program based on deep learning, but also can be used to temporarily store data that has been output or is to be output.

[0190] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (such as image site refinement segmentation method programs based on deep learning, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0191] The bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 may be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0192] Figure 3 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0193] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0194] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0195] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0196] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0197] The image site refined segmentation method program based on deep learning stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0198] Acquire an image enhancement training set of the image to be segmented, determine the enhancement model architecture of the image to be segmented, and define a reconstruction loss function and a perception loss function of the enhancement model architecture, and construct an image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set;

[0199] Based on the image enhancement model, image optimization is performed on the image to be segmented to obtain an enhanced image to be segmented, a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented are constructed, and 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure are configured;

[0200] According to the 50 structural hyperparameters and the 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained to obtain the ResNeXt-50 model and the ResNeXt-101 model;

[0201] Establishing a feature ResNeXt extraction model for the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and performing feature extraction on the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented;

[0202] Determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection-and-union ratio of the Mask R-CNN model, and when the intersection-and-union ratio meets the preset intersection-and-union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.

[0203] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figures 1 to 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0204] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0205] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor of an electronic device, the computer program can implement:

[0206] Acquire an image enhancement training set of the image to be segmented, determine the enhancement model architecture of the image to be segmented, and define a reconstruction loss function and a perception loss function of the enhancement model architecture, and construct an image enhancement model of the image to be segmented through the enhancement model architecture, the reconstruction loss function, the perception loss function and the image enhancement training set;

[0207] Based on the image enhancement model, image optimization is performed on the image to be segmented to obtain an enhanced image to be segmented, a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented are constructed, and 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure are configured;

[0208] According to the 50 structural hyperparameters and the 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained to obtain the ResNeXt-50 model and the ResNeXt-101 model;

[0209] Establishing a feature ResNeXt extraction model for the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and performing feature extraction on the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented;

[0210] Determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection-and-union ratio of the Mask R-CNN model, and when the intersection-and-union ratio meets the preset intersection-and-union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.

[0211] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only illustrative, and actual implementation may have other division methods.

[0212] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0214] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0215] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0216] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for fine-grained image segmentation based on deep learning, characterized in that: The method comprises: Acquire an image enhancement training set of an image to be segmented, determine an enhancement model architecture of the image to be segmented, and define a reconstruction loss function and a perceptual loss function of the enhancement model architecture, calculate an image effective coefficient of the image enhancement training set, extract a target enhancement training image of the image enhancement training set according to the image effective coefficient, initialize the enhancement model architecture to obtain an initial enhancement model, determine an Adam optimizer of the initial enhancement model, and define optimizer parameters of the Adam optimizer, use the initial enhancement model to output a reconstructed image of the target enhancement training image, use the reconstruction loss function to calculate the reconstruction loss of the reconstructed image, calculate the perceptual loss of the reconstructed image through the perceptual loss function, and optimize the initial enhancement model according to the reconstruction loss and perceptual loss using the Adam optimizer and optimizer parameters to obtain an image enhancement model of the image to be segmented; Based on the image enhancement model, image optimization is performed on the image to be segmented to obtain an enhanced image to be segmented, a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented are constructed, and 50 structure hyperparameters and 101 structure hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure are configured; According to the 50 structural hyperparameters and the 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained to obtain the ResNeXt-50 model and the ResNeXt-101 model; Establishing a feature ResNeXt extraction model for the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and performing feature extraction on the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented; Determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection-and-union ratio of the Mask R-CNN model, and when the intersection-and-union ratio meets the preset intersection-and-union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.

2. The image site refined segmentation method based on deep learning according to claim 1, characterized in that: The calculating the perceptual loss of the reconstructed image by using the perceptual loss function comprises: Constructing a feature extractor of the perceptual loss function and defining an image feature layer of the feature extractor; Based on the image feature layer, determining a feature vector of the image feature layer; According to the feature extractor, the image feature layer and the feature vector, the perceptual loss of the reconstructed image is calculated using the following formula: Among them, ρ percrp represents the perceptual loss of the reconstructed image, Q represents the number of image feature layers of the reconstructed image, θ represents the feature extractor, (reconstructed) r Represents the feature vector of the rth image feature layer of the reconstructed image, (original) r Represents the feature vector of the rth image feature layer of the target enhanced training image corresponding to the reconstructed image, ||θ(reconstructed) r -θ(original)r||1 represents the L1 norm of the feature vector of the r-th image feature layer of the reconstructed image and the feature vector of the r-th image feature layer of the target enhanced training image corresponding to the reconstructed image.

3. The image site refined segmentation method based on deep learning as claimed in claim 2, characterized in that: The method of optimizing the initial enhancement model by using the Adam optimizer and optimizer parameters according to the reconstruction loss and the perception loss to obtain the image enhancement model of the image to be segmented includes: Extracting initial enhancement model parameters of the initial enhancement model; Calculating the total image loss of the image to be segmented according to the reconstruction loss and the perception loss; Calculating the loss gradient of the total image loss with respect to the initial enhancement model parameters; Based on the loss gradient and optimizer parameters, the first-order moment estimate and the second-order moment estimate of the Adam optimizer are calculated using the following function: t c =σ1·τ c-1 +S c (1-σ1) Among them, τ c represents the first-order moment estimate of the Adam optimizer at the cth iteration, ω c represents the second-order moment estimate of the Adam optimizer at the cth iteration, σ1 represents the exponential decay rate of the optimizer parameter corresponding to the first-order moment estimate, σ2 represents the exponential decay rate of the optimizer parameter corresponding to the second-order moment estimate, τ c-1 represents the first-order moment estimate of the Adam optimizer at the c-1th iteration, ω c-1 represents the second-order moment estimate of the Adam optimizer at the c-1th iteration, S c represents the loss gradient; The initial enhancement model is optimized according to the first-order moment estimation and the second-order moment estimation to obtain an image enhancement model of the image to be segmented.

4. The image site refined segmentation method based on deep learning as claimed in claim 3, characterized in that: The step of optimizing the initial enhancement model according to the first-order moment estimation and the second-order moment estimation to obtain the image enhancement model of the image to be segmented includes: Calculating a corrected first-order moment estimate and a corrected second-order moment estimate of the first-order moment estimate and the second-order moment estimate; Based on the corrected first-order moment estimation and the corrected second-order moment estimation, the model update parameters of the initial enhanced model are analyzed using the following formula: Among them, B c+1 represents the model update parameter of the initial enhanced model at the c+1th iteration, B c represents the model update parameter of the initial enhancement model at the cth iteration, ∈ represents the numerical stability constant, α represents the learning rate of the initial enhancement model, ω c,j represents the corrected second-order moment estimate, τ c,j represents the corrected first-order moment estimate, represents the exponential decay rate of the corrected second-order moment estimate corresponding to the second-order moment estimate of the c-th iteration; The initial enhancement model is optimized according to the model update parameters to obtain the image enhancement model of the image to be segmented.

5. The image site refined segmentation method based on deep learning as claimed in claim 4, characterized in that: The method of training the ResNeXt-50 network structure and the ResNeXt-101 network structure according to the 50 structural hyperparameters and the 101 structural hyperparameters to obtain a training ResNeXt-50 model and a training ResNeXt-101 model includes: Constructing a meta-learning algorithm for the ResNeXt-50 network structure and the ResNeXt-101 network structure; Analyze the image category of the image to be segmented corresponding to the ResNeXt-50 network structure; Obtaining a single example training set and an unseen example test set of the image category; According to the single example training set, 50 structural hyperparameters, and 101 structural hyperparameters, the ResNeXt-50 network structure and the ResNeXt-101 network structure are trained using the meta-learning algorithm to obtain a training ResNeXt-50 model and a training ResNeXt-101 model; Using the unseen example test set, calculating the 50 model test results and the 101 model test results of the trained ResNeXt-50 model and the trained ResNeXt-101 model; Constructing a 50 model accuracy curve and a 101 model accuracy curve of the ResNeXt-50 model through the 50 model test results and the 101 model test results; Based on the 50 model accuracy curve and the 101 model accuracy curve, analyze the 50 model performance and the 101 model performance of the training ResNeXt-50 model and the training ResNeXt-101 model; When the 50 model performance and the 101 model performance meet the preset 50 model performance threshold and the 101 model performance threshold, the trained ResNeXt-50 model and the trained ResNeXt-101 model are used as the ResNeXt-50 model and the trained ResNeXt-101 model for enhancing the image to be segmented.

6. The image site refined segmentation method based on deep learning as claimed in claim 5, characterized in that: The step of extracting features from the enhanced image to be segmented based on the feature ResNeXt extraction model to obtain a segmentation feature map of the enhanced image to be segmented includes: Based on the feature ResNeXt extraction model, extracting the 50 model feature map and the 101 model feature map of the enhanced image to be segmented; Calculating the size difference between the 50-model feature map and the 101-model feature map; Calculate the pooling layer parameters of the 50-model feature map and the 101-model feature map according to the size difference; According to the pooling layer parameters, the 50 model feature map and the 101 model feature map are pooled to obtain a pooled 50 model feature map and a pooled 101 model feature map; Identify the 50 model feature map channels and the 101 model feature map channels of the pooled 50 model feature map and the pooled 101 model feature map; Based on the 50 model feature map channels and the 101 model feature map channels, the pooled 50 model feature map and the pooled 101 model feature map are element-by-element multiplied to obtain the segmentation feature map of the enhanced image to be segmented.

7. The image site refined segmentation method based on deep learning according to claim 6, characterized in that: The calculating the intersection-over-union ratio of the Mask R-CNN model includes: Collecting a test feature map of the Mask R-CNN model and defining a true mask of the test feature map; Constructing an upsampling layer of the test feature map, and upsampling the test feature map based on the upsampling layer to obtain an upsampled feature map; Adding a classification layer of the upsampled feature map, and predicting pixel categories of the upsampled feature map based on the classification layer; Calculating the category score of the pixel category using a Sigmoid function; Determining a prediction mask of the enhanced image to be segmented according to the category score and the upsampled feature map; The intersection-over-union (IoU) ratio of the Mask R-CNN model is calculated based on the predicted mask and the true mask.

8. The image site refined segmentation method based on deep learning according to claim 7, characterized in that: The calculating the intersection-over-union ratio of the Mask R-CNN model according to the predicted mask and the true mask includes: Unifying the sizes of the predicted mask and the real mask to obtain a predicted mask and a real mask of the same size; The mask intersection value of the predicted mask with the same size and the real mask with the same size is calculated using the following formula: Among them, j represents the mask intersection value of the same-size prediction mask and the same-size true mask, F represents the height of the same-size prediction mask, E represents the width of the same-size prediction mask, m j represents the mask intersection function of the predicted mask of the same size and the real mask of the same size, pred ve Indicates the value of the same size prediction mask at position (v, e), true ve Indicates the value of the predicted mask of the same size at position (v, e) corresponding to the real mask of the same size; The mask union value of the predicted mask with the same size and the real mask with the same size is calculated using the following formula: Among them, b represents the mask union value of the same-size prediction mask and the same-size true mask, F represents the height of the same-size prediction mask, E represents the width of the same-size prediction mask, m b The mask union function representing the predicted mask of the same size and the real mask of the same size, pred ve Indicates the value of the same size prediction mask at position (v, e), true ve Indicates the value of the predicted mask of the same size at position (v, e) corresponding to the real mask of the same size; Based on the mask intersection value and the mask union value, the intersection-over-union ratio of the Mask R-CNN model is calculated.

9. A device for fine segmentation of image sites based on deep learning, characterized in that: The device comprises: An image enhancement model construction module is used to obtain an image enhancement training set of an image to be segmented, determine an enhancement model architecture of the image to be segmented, define a reconstruction loss function and a perceptual loss function of the enhancement model architecture, calculate an image effective coefficient of the image enhancement training set, extract a target enhancement training image of the image enhancement training set according to the image effective coefficient, initialize the enhancement model architecture to obtain an initial enhancement model, determine an Adam optimizer of the initial enhancement model, and define optimizer parameters of the Adam optimizer, use the initial enhancement model to output a reconstructed image of the target enhancement training image, use the reconstruction loss function to calculate the reconstruction loss of the reconstructed image, use the perceptual loss function to calculate the perceptual loss of the reconstructed image, and optimize the initial enhancement model according to the reconstruction loss and perceptual loss using the Adam optimizer and optimizer parameters to obtain an image enhancement model of the image to be segmented; A structural hyperparameter determination module is used to perform image optimization on the image to be segmented based on the image enhancement model to obtain an enhanced image to be segmented, construct a ResNeXt-50 network structure and a ResNeXt-101 network structure of the enhanced image to be segmented, and configure 50 structural hyperparameters and 101 structural hyperparameters of the ResNeXt-50 network structure and the ResNeXt-101 network structure; A ResNeXt model construction module, used to train the ResNeXt-50 network structure and the ResNeXt-101 network structure according to the 50 structure hyperparameters and the 101 structure hyperparameters to obtain a ResNeXt-50 model and a ResNeXt-101 model; A segmentation feature map acquisition module is used to establish a feature ResNeXt extraction model of the enhanced image to be segmented according to the ResNeXt-50 model and the ResNeXt-101 model, and based on the feature ResNeXt extraction model, perform feature extraction on the enhanced image to be segmented to obtain a segmentation feature map of the enhanced image to be segmented; A refined segmentation module is used to determine the target detection framework and mask branch of the enhanced image to be segmented, establish a Mask R-CNN model of the enhanced image to be segmented based on the target detection framework and the mask branch, calculate the intersection and union ratio of the Mask R-CNN model, and when the intersection and union ratio meets the preset intersection and union ratio threshold, use the Mask R-CNN model to segment the enhanced image to be segmented according to the segmentation feature map to obtain a refined segmentation result of the enhanced image to be segmented.