Method, segmentation method and device for generating image segmentation model based on search space

Abstract

The present application provides an image segmentation model generation method, a segmentation method and a device based on a search space. Based on the last control network, the current node parameters and the current weight parameters of each execution layer in the search space corresponding to the current segmentation network to be trained are determined, and the current segmentation network to be trained is constructed. At the same time, the constructed current segmentation network to be trained is trained based on a training image set, and the segmentation return rate of the current segmentation network to be trained that has been trained is determined based on a validation image set. The last control network is updated based on the obtained segmentation return rate to obtain the updated current control network. The current control network is used as the last control network, and the network parameters are continuously updated until the preset number of updates is reached, so as to generate an image segmentation model. In this way, the structure of each segmentation network can be continuously adjusted during the training of the model, so that the generated image segmentation model has the characteristics of being lightweight, which helps to improve the accuracy of the segmentation result.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular, to a method, a segmentation method and a device for generating an image segmentation model based on a search space. Background Art

[0002] Image semantic segmentation is one of the basic tasks in the field of computer vision and has a wide range of applications in various fields, such as medical images, autonomous driving, video monitoring, etc. Its goal is to segment an image into regions with different semantic information and label the corresponding semantic tags for each region.

[0003] Currently, usually, a technician pre-constructs a basic model for training a segmentation model and trains the constructed basic model with the obtained training image set to obtain a segmentation model for image segmentation. However, the structure of the basic model is relatively single and fixed, and the segmentation effects achieved in different technical fields are uneven, sometimes good and sometimes bad. Summary of the Invention

[0004] In view of this, the purpose of the present application is to provide a method, a segmentation method and a device for generating an image segmentation model based on a search space, which can continuously adjust the structure of the network during the training process of the model, train an image segmentation model with a better segmentation effect, and thus can more accurately segment the target object from the image.

[0005] The embodiment of the present application provides a method for generating an image segmentation model based on a search space. The method for generating the image segmentation model includes:

[0006] Based on the last control network that has been updated last time and obtained, determine the current node parameters and current weight parameters of each execution layer in the search space corresponding to the currently to-be-trained segmentation network to construct the currently to-be-trained segmentation network;

[0007] Train the currently to-be-trained segmentation network based on the training image set, and determine the segmentation return rate of the currently to-be-trained segmentation network that has been trained based on the obtained validation image set;

[0008] Update the last control network based on the segmentation return rate to obtain the currently updated control network;

[0009] Use the currently updated control network as the last control network and continue to update the network parameters until the preset number of updates is reached, and determine that the control network has been finally updated;

[0010] Determine an image segmentation model from multiple currently to-be-trained segmentation networks that have been trained during the network parameter update process.

[0011] Further, determining the image segmentation model from multiple currently to-be-trained segmentation networks that have completed the network parameter update process includes:

[0012] Determining the segmentation accuracy of each currently to-be-trained segmentation network based on the validation image set;

[0013] Determining the currently to-be-trained segmentation network with the highest segmentation accuracy among the multiple currently to-be-trained segmentation networks as the image segmentation model.

[0014] Further, training the currently to-be-trained segmentation network based on the training image set and determining the segmentation return rate of the currently to-be-trained segmentation network that has completed training based on the obtained validation image set includes:

[0015] Obtaining the training image set and the validation image set;

[0016] Inputting each to-be-segmented sample image in the training image set into the constructed currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to each to-be-segmented sample image;

[0017] Determining the loss value of the currently to-be-trained segmentation network based on the predicted segmentation label corresponding to each to-be-segmented sample image and the true segmentation label corresponding to each to-be-segmented sample image in the training image set;

[0018] Adjusting the first network parameters of the currently to-be-trained segmentation network based on the loss value until the number of training times of the currently to-be-trained segmentation network reaches the preset number of training times, and determining that the currently to-be-trained segmentation network has completed training;

[0019] Inputting the validation image set into the currently to-be-trained segmentation network that has completed training to determine the segmentation return rate of the currently to-be-trained segmentation network that has completed training.

[0020] Further, inputting each to-be-segmented sample image in the training image set into the constructed currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to each to-be-segmented sample image includes:

[0021] For each to-be-segmented sample image in the training image set, inputting the to-be-segmented sample image into the encoding layer in the currently to-be-trained segmentation network to obtain the encoded to-be-segmented sample image;

[0022] Inputting the encoded to-be-segmented sample image into the cross-scale layer in the currently to-be-trained segmentation network to extract the image feature maps of multiple dimensions of the to-be-segmented sample image from the encoded to-be-segmented sample image;

[0023] Perform feature fusion processing on the image feature maps of the multiple dimensions to obtain an image fusion feature map corresponding to the sample image to be segmented;

[0024] Input the image fusion feature map into the decoding layer in the current segmentation network to be trained to obtain a predicted segmentation label corresponding to the sample image to be segmented.

[0025] Further, the performing feature fusion processing on the image feature maps of the multiple dimensions to obtain an image fusion feature map corresponding to the sample image to be segmented includes:

[0026] Perform sampling processing on the image features of each dimension according to the output size of the cross-scale layer to obtain a target image feature map corresponding to the image feature map of each dimension;

[0027] Perform feature concatenation on the obtained multiple target image feature maps to determine an image fusion feature map corresponding to the sample image to be segmented.

[0028] Further, the updating the previous control network based on the segmentation return rate to obtain the current control network after updating includes:

[0029] Determine the optimal expected value of the previous control network based on the segmentation return rate;

[0030] Adjust the second network parameters of the previous control network based on the optimal expected value to obtain the current control network after updating.

[0031] An embodiment of the present application further provides an image segmentation method, and the image segmentation method includes:

[0032] Obtain an image to be segmented;

[0033] Input the image to be segmented into an image segmentation model obtained by the above-mentioned method for generating an image segmentation model based on a search space to obtain a segmentation result of the target object in the image to be segmented.

[0034] An embodiment of the present application further provides a device for generating an image segmentation model based on a search space, and the generating device includes:

[0035] A model construction module, configured to determine current node parameters and current weight parameters of each execution layer in the search space corresponding to the current segmentation network to be trained based on the obtained previous control network that has been updated last time, so as to construct the current segmentation network to be trained;

[0036] A calculation module, configured to train the current segmentation network to be trained based on a training image set, and determine the segmentation return rate of the trained current segmentation network to be trained based on an obtained validation image set;

[0037] A network update module, configured to update the previous control network based on the segmentation return rate to obtain an updated current control network;

[0038] A loop training module, configured to use the current control network as the previous control network and continue to update network parameters until a preset number of updates is reached, and determine that the control network is finally updated;

[0039] A model determination module, configured to determine an image segmentation model from multiple current segmentation networks to be trained that are trained during the network parameter update process.

[0040] Further, when the model determination module is configured to determine an image segmentation model from multiple current segmentation networks to be trained that are trained during the network parameter update process, the model determination module is configured to:

[0041] Determine the segmentation accuracy of each current segmentation network to be trained based on the validation image set;

[0042] Determine the current segmentation network to be trained with the highest segmentation accuracy among the multiple current segmentation networks to be trained as the image segmentation model.

[0043] Further, when the calculation module is configured to train the current segmentation network to be trained based on a training image set and determine the segmentation return rate of the trained current segmentation network to be trained based on an obtained validation image set, the calculation module is configured to:

[0044] Obtain a training image set and a validation image set;

[0045] Input each sample image to be segmented in the training image set into the constructed current segmentation network to be trained to obtain a predicted segmentation label corresponding to each sample image to be segmented;

[0046] Determine the loss value of the current segmentation network to be trained based on the predicted segmentation label corresponding to each sample image to be segmented and the true segmentation label corresponding to each sample image to be segmented in the training image set;

[0047] Adjust the first network parameters of the current segmentation network to be trained based on the loss value until the number of training times of the current segmentation network to be trained reaches a preset number of training times, and determine that the current segmentation network to be trained is trained;

[0048] Input the verification image set into the trained current segmentation network to be trained, and determine the segmentation return rate of the trained current segmentation network to be trained.

[0049] Further, when the calculation module is used to input each sample image to be segmented in the training image set into the constructed current segmentation network to be trained to obtain the predicted segmentation label corresponding to each sample image to be segmented, the calculation module is used for:

[0050] For each sample image to be segmented in the training image set, input the sample image to be segmented into the encoding layer in the current segmentation network to be trained to obtain the encoded sample image to be segmented;

[0051] Input the encoded sample image to be segmented into the cross-scale layer in the current segmentation network to be trained, and extract the image feature maps of multiple dimensions of the sample image to be segmented from the encoded sample image to be segmented;

[0052] Perform feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the sample image to be segmented;

[0053] Input the image fusion feature map into the decoding layer in the current segmentation network to be trained to obtain the predicted segmentation label corresponding to the sample image to be segmented.

[0054] Further, when the calculation module is used to perform feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the sample image to be segmented, the calculation module is used for:

[0055] Perform sampling processing on the image features of each dimension according to the output size of the cross-scale layer to obtain the target image feature map corresponding to the image feature map of each dimension;

[0056] Perform feature concatenation on the obtained multiple target image feature maps to determine the image fusion feature map corresponding to the sample image to be segmented.

[0057] Further, when the network update module is used to update the previous control network based on the segmentation return rate to obtain the updated current control network, the network update module is used for:

[0058] Based on the segmentation return rate, determine the optimal expected value of the previous control network;

[0059] Based on the optimal expected value, adjust the second network parameters of the previous control network to obtain the updated current control network.

[0060] The embodiments of the present application also provide an image segmentation device, which includes:

[0061] An image acquisition module, configured to acquire an image to be segmented;

[0062] An image segmentation module, configured to input the image to be segmented into an image segmentation model obtained by the above-mentioned method for generating an image segmentation model based on a search space, so as to obtain a segmentation result of the target object in the image to be segmented.

[0063] The embodiments of the present application also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the memory through the bus. When the machine-readable instructions are executed by the processor, the steps of the above-mentioned method for generating an image segmentation model based on a search space or the steps of the above-mentioned image segmentation method are executed.

[0064] The embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored. When the computer program is run by a processor, the steps of the above-mentioned method for generating an image segmentation model based on a search space or the steps of the above-mentioned image segmentation method are executed.

[0065] In this way, the present application determines the current node parameters and current weight parameters of each execution layer in the search space for constructing the current segmentation network to be trained based on obtaining the previous control network that has been updated last time, and thus constructs the current segmentation network to be trained based on the current node parameters and current weight parameters of each execution layer; at the same time, trains the constructed current segmentation network to be trained based on the obtained training image set, and determines the segmentation return rate of the current segmentation network to be trained that has been trained based on the obtained verification image set; then updates the previous control network based on the calculated segmentation return rate of the current segmentation network to be trained to obtain the updated current control network; uses the updated current control network as the previous control network, and continues to update the network parameters until the preset number of updates is reached, and determines that the control network parameters are finally updated; determines an image segmentation model that can be used for subsequent image segmentation from multiple current training segmentation networks that have been trained during the network parameter update process. In this way, the structure of each segmentation network can be continuously adjusted during the training process of the model, and an image segmentation model with better segmentation effect can be trained, so that the target object can be more accurately segmented from the image, which helps to improve the accuracy of the segmentation result.

[0066] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically gives preferred embodiments and, in conjunction with the accompanying drawings, makes a detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] To more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for the embodiments will be briefly introduced below. It should be understood that the following accompanying drawings only show some embodiments of the present application and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related accompanying drawings can be obtained based on these drawings without creative efforts.

[0068] Figure 1 It is a flowchart of a method for generating an image segmentation model provided by an embodiment of the present application;

[0069] Figure 2 It is a schematic diagram of the hypernetwork model structure;

[0070] Figure 3 It is a schematic diagram of the control network structure;

[0071] Figure 4 It is a flowchart of a method for generating an image segmentation model based on a search space provided by another embodiment of the present application;

[0072] Figure 5 It is a schematic diagram of the feature fusion process;

[0073] Figure 6 It is a flowchart of an image segmentation method provided by an embodiment of the present application;

[0074] Figure 7 It is a schematic diagram of the structure of an apparatus for generating an image segmentation model based on a search space provided by an embodiment of the present application;

[0075] Figure 8 It is a schematic diagram of the structure of an image segmentation apparatus provided by an embodiment of the present application;

[0076] Figure 9 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0077] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part rather than all of the embodiments of this application. Components of the embodiments of this application described and illustrated in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without creative efforts falls within the scope of protection of this application.

[0078] First, the applicable application scenarios of this application are introduced. This application can be applied to the field of image processing technology. Based on the previously updated last control network obtained, the current node parameters and current weight parameters of each execution layer in the search space corresponding to the currently to-be-trained segmentation network are determined, and thus the currently to-be-trained segmentation network is constructed based on the current node parameters and current weight parameters of each execution layer. At the same time, the constructed currently to-be-trained segmentation network is trained based on the obtained training image set, and the segmentation return rate of the currently to-be-trained segmentation network after training is determined based on the obtained validation image set. Then, the last control network is updated based on the calculated segmentation return rate of the currently to-be-trained segmentation network to obtain the updated current control network. The updated current control network is used as the last control network, and the network parameter update continues until the preset update times are reached, determining that the control network parameters are finally updated. From the multiple currently trained segmentation networks that are trained during the network parameter update process, an image segmentation model that can be used for subsequent image segmentation is determined. In this way, the structure of each segmentation network can be continuously adjusted during the model training process, and an image segmentation model with a better segmentation effect can be trained, so that the target object can be more accurately segmented from the image, which helps to improve the accuracy of the segmentation result.

[0079] Through research, it is found that currently, usually, technicians pre-construct a basic model for training a segmentation model, and the constructed basic model is trained with the obtained training image set to obtain a segmentation model for image segmentation. However, the structure of the basic model is relatively single and fixed, and the segmentation effects achieved in different technical fields are uneven, sometimes good and sometimes bad.

[0080] Based on this, the embodiments of this application provide a method for generating an image segmentation model, which can continuously adjust the structure of the network during the model training process and train an image segmentation model with a better segmentation effect, so that the target object can be more accurately segmented from the image.

[0081] Please refer to Figure 1 , Figure 1 , which is a flowchart of a method for generating an image segmentation model based on a search space provided by an embodiment of the present application. As Figure 1 shown in

[0082] S101. Based on the last control network that was updated successfully last time, determine the current node parameters and current weight parameters of each execution layer in the search space corresponding to the current segmentation network to be trained, so as to construct the current segmentation network to be trained.

[0083] In this step, obtain the last control network that was updated successfully last time. Based on the obtained last control network that was updated successfully last time, determine the current node parameters and current weight parameters of each execution layer in the search space corresponding to the current segmentation network to be trained, and construct the current segmentation network to be trained based on the determined current node parameters and current weight parameters of each execution layer.

[0084] Here, the search space corresponding to the current segmentation network to be trained may include multiple execution layers. Specifically, it may include an encoding layer (Encoder), an intermediate cross-scale layer (M), a decoding layer (Decoder), etc. For each execution layer, each execution layer includes its own current node parameters. Specifically, the node parameters of the encoding layer may include the number of nodes in the encoding layer; the node parameters of the decoding layer may include the number of nodes in the decoding layer; the intermediate cross-scale layer may include input nodes, scale nodes, skip connection nodes, etc. Furthermore, by constructing the search space corresponding to the current segmentation network to be trained, the trade-off problem between semantic information and spatial position information and the lightweight problem are solved, so that the generated image segmentation model has the characteristics of being lightweight.

[0085] Among them, the image features extracted from the sample image to be segmented are input into the intermediate cross-scale layer through the input nodes, and the scale nodes represent the size and number of channels of the output image fusion features.

[0086] Exemplarily, the meanings and values represented by the current node parameters of each execution layer are shown in Table 1:

[0087] Table 1 Node Parameters

[0088]

[0089] Here, we perform skip connections on all nodes of the model in the Decoder module. There are two benefits to doing this: on the one hand, performing skip connections on all previous nodes can make full use of all the previous feature maps, achieve the reuse of features, avoid repeated calculations, improve the effectiveness of parameters, and at the same time share the information obtained in each layer. Therefore, it can greatly reduce the number of channels of the feature map and reduce the feature redundancy between different channels of the feature map; on the other hand, the skip connections in the Decoder can achieve an implicit deep supervision, thereby slowing down the overfitting of the network and improving the gradient backpropagation of the shallow neural network.

[0090] In addition, the images in the training image set in the embodiments of the present application can be medical images, autonomous driving images, video monitoring images, etc.

[0091] S102. Train the current to-be-trained segmentation network based on the training image set, and determine the segmentation return rate of the current to-be-trained segmentation network that has been trained based on the obtained validation image set.

[0092] In this step, after constructing the current to-be-trained segmentation network, use the obtained training image set to train the constructed current to-be-trained segmentation network, and after the end of this training process, use the obtained validation image set to verify the current to-be-trained segmentation network after this training, and determine the segmentation return rate obtained after the current to-be-trained segmentation network segments the images in the validation image set after the end of this training process.

[0093] Specifically, use each to-be-segmented sample image in the training image set as the input feature, and use the true segmentation label corresponding to each to-be-segmented sample image in the training image set as the output feature to train the constructed current to-be-trained segmentation network to obtain the current to-be-trained segmentation network that has been trained.

[0094] Among them, the current to-be-trained segmentation network can be a hypernetwork model, such as Figure 2 shown, Figure 2 is a schematic diagram of the hypernetwork model structure. The hypernetwork model includes an encoding layer 2a, a cross-scale layer 2b, and a decoding layer 2c.

[0095] Here, before using the to-be-segmented sample images in the training image set to train the current to-be-trained segmentation network, perform data preprocessing and data loading on the to-be-segmented sample images in the training image set. Specifically, first adjust the brightness, contrast, and gamma value of each to-be-segmented sample image in the training image set to enhance the to-be-segmented sample images in the training image set. At the same time, use the data prefetching technology (Prefetch) to accelerate data loading.

[0096] In addition, the validation image set is not subject to data augmentation. However, the images in the validation image set have the same resolution as the sample images to be segmented in the training image set, and are used to evaluate the segmentation performance of the current segmentation network to be trained.

[0097] S103. Update the last control network based on the segmentation return rate to obtain the current control network after the update.

[0098] In this step, based on the segmentation return rate of the current segmentation network to be trained determined through the validation image set, the last control network is updated to obtain the current control network after the update.

[0099] Among them, the control network is constructed based on the long short-term memory network. Similarly, corresponding to the current segmentation network to be trained, the control network also includes corresponding execution layers: an encoding layer, a cross-scale layer, and a decoding layer, and the number of nodes in each execution layer is the same as the node parameters in each execution layer of the current segmentation network to be trained. For example Figure 3 as shown Figure 3 is a schematic diagram of the control network structure. Corresponding to the initial segmentation model, the control network includes an encoding layer 3a, a cross-scale layer 3b, and a decoding layer 3c.

[0100] However, different from the current segmentation network to be trained, each intermediate cross-scale layer in the control network, compared with each intermediate cross-scale layer in the current segmentation network to be trained, in addition to including input nodes, scale nodes, and skip connection nodes, also includes an output unit anchor for output as the input of subsequent nodes.

[0101] S104. Use the current control network as the last control network and continue to update the network parameters until the preset number of updates is reached, and determine that the control network is finally updated.

[0102] In this step, use the updated current control network as the last control network and continue to update the network parameters, that is, through the last control network obtained again, reconstruct a current segmentation network to be trained, train the currently constructed current segmentation network to be trained again using the training image set, and use the validation image set to determine the segmentation return rate of the currently constructed current segmentation network to be trained. Repeat the above network parameter update process until the number of updates reaches the preset number of updates, or until the currently trained current segmentation network to be trained converges when updating the network parameters, and determine that the control network is finally updated.

[0103] S105. Determine the image segmentation model from multiple currently trained segmentation networks to be trained during the network parameter update process.

[0104] In this step, during the process of updating network parameters in a loop, multiple currently trained segmentation networks to be trained will be obtained, and an image segmentation model that can ultimately be used for image segmentation is determined from the multiple currently trained segmentation networks.

[0105] In this way, based on obtaining the previously updated previous control network, the current node parameters and current weight parameters of each execution layer in the search space corresponding to the currently trained segmentation network to be constructed are determined, so as to construct the currently trained segmentation network based on the current node parameters and current weight parameters of each execution layer; at the same time, the constructed currently trained segmentation network is trained based on the obtained training image set, and the segmentation return rate of the currently trained segmentation network after training is determined based on the obtained validation image set; then the previous control network is updated based on the calculated segmentation return rate of the currently trained segmentation network to obtain the updated current control network; the updated current control network is used as the previous control network, and the network parameters are continuously updated until the preset number of updates is reached, and it is determined that the control network parameters are finally updated; an image segmentation model that can be used for subsequent image segmentation is determined from the multiple currently trained segmentation networks that are trained during the network parameter update process. In this way, the structure of each segmentation network can be continuously adjusted during the training process of the model, and an image segmentation model with better segmentation effect can be trained, so that the target object can be more accurately segmented from the image, which helps to improve the accuracy of the segmentation result.

[0106] Please refer to Figure 4 , Figure 4 which is a flowchart of a method for generating an image segmentation model based on a search space provided in another embodiment of the present application. As shown in Figure 4 , the method for generating an image segmentation model based on a search space provided in the embodiment of the present application includes:

[0107] S401. Based on the previously updated previous control network obtained, determine the current node parameters and current weight parameters of each execution layer in the search space corresponding to the currently trained segmentation network to be constructed, so as to construct the currently trained segmentation network.

[0108] S402. Train the currently trained segmentation network based on the training image set, and determine the segmentation return rate of the currently trained segmentation network after training based on the obtained validation image set.

[0109] S403. Update the previous control network based on the segmentation return rate to obtain the updated current control network.

[0110] S404. Use the current control network as the previous control network, and continue to update the network parameters until the preset number of updates is reached, and determine that the control network update is finally completed.

[0111] S405. Determine the segmentation accuracy of each current segmentation network to be trained based on the verification image set.

[0112] In this step, each current segmentation network to be trained is verified based on the obtained verification image set, and the segmentation accuracy of each current segmentation network to be trained is determined.

[0113] S406. Determine the current segmentation network with the highest segmentation accuracy among the multiple current segmentation networks to be trained as the image segmentation model.

[0114] In this step, after obtaining the segmentation accuracy of each current segmentation network to be trained, the current segmentation network with the highest segmentation accuracy among the multiple current segmentation networks to be trained is determined as the image segmentation model for subsequent image segmentation processes.

[0115] Among them, the descriptions of S401 to S404 can refer to the descriptions of S101 to S104 and can achieve the same technical effects, which will not be elaborated here.

[0116] Further, step S402 includes: obtaining a training image set and a verification image set; inputting each sample image to be segmented in the training image set into the constructed current segmentation network to be trained to obtain a predicted segmentation label corresponding to each sample image to be segmented; determining the loss value of the current segmentation network to be trained based on the predicted segmentation label corresponding to each sample image to be segmented and the true segmentation label corresponding to each sample image to be segmented in the training image set; adjusting the first network parameters of the current segmentation network to be trained based on the loss value until the number of training times of the current segmentation network to be trained reaches the preset number of training times, and determining that the current segmentation network to be trained is trained; inputting the verification image set into the trained current segmentation network to be trained, and determining the segmentation return rate of the trained current segmentation network to be trained.

[0117] In this step, a training image set and a validation image set for training the current segmentation network to be trained are obtained. Each sample image to be segmented in the training image set is input into the constructed current segmentation network to be trained. The predicted segmentation label corresponding to each sample image to be segmented is determined through the constructed current segmentation network to be trained. Based on the predicted segmentation label of each sample image to be segmented and the true segmentation label corresponding to each sample image to be segmented, the loss value corresponding to the current segmentation network to be trained is determined. The first network parameters of the current segmentation network to be trained are adjusted through the calculated loss value until the number of training times of the current segmentation network to be trained reaches the preset number of training times, and it is determined that the training of the current segmentation network to be trained is completed.

[0118] The validation image is input into the trained current segmentation network to be trained, and the segmentation return rate of the current segmentation network to be trained after the end of this round of training is determined.

[0119] Specifically, the loss value of the initial segmentation model is calculated through the following formula:

[0120]

[0121] Where, is the loss value of the current segmentation network to be trained, B is the number of sample images to be segmented in the training image set, and Y c is the predicted segmentation label of the c-th sample image to be segmented, is the true segmentation label of the c-th sample image to be segmented.

[0122] Further, the step of inputting each sample image to be segmented in the training image set into the constructed current segmentation network to be trained to obtain the predicted segmentation label corresponding to each sample image to be segmented includes: for each sample image to be segmented in the training image set, inputting the sample image to be segmented into the encoding layer in the current segmentation network to be trained to obtain the encoded sample image to be segmented; inputting the encoded sample image to be segmented into the cross-scale layer in the current segmentation network to be trained, and extracting image feature maps of multiple dimensions of the sample image to be segmented from the encoded sample image to be segmented; performing feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the sample image to be segmented; inputting the image fusion feature map into the decoding layer in the current segmentation network to be trained to obtain the predicted segmentation label corresponding to the sample image to be segmented.

[0123] In this step, for each sample image to be segmented in the training image set, the sample image to be segmented is input into the encoding layer in the current segmentation network to be trained, and the encoded sample image to be segmented is determined.

[0124] Then, input the encoded sample image to be segmented into the cross-scale layer of the current segmentation network to be trained, and extract the image feature maps of multiple dimensions of the sample image to be segmented from the encoded sample image to be segmented. Perform feature fusion processing on the image feature maps of multiple dimensions of the sample image to be segmented through the cross-scale layer to obtain the image fusion feature map corresponding to the sample image to be segmented.

[0125] Finally, input the obtained image fusion feature map corresponding to the sample image to be segmented into the decoding layer of the current segmentation network to be trained to obtain the predicted segmentation label corresponding to the sample image to be segmented.

[0126] Further, the feature fusion processing of the image features of multiple dimensions to obtain the image fusion feature map corresponding to the sample image to be segmented includes: sampling the image feature map of each dimension according to the output size of the cross-scale layer to obtain the target image feature map corresponding to the image feature of each dimension; performing feature concatenation on the obtained multiple target image feature maps to determine the image fusion feature map corresponding to the sample image to be segmented.

[0127] In this step, as Figure 5 shown, Figure 5 is a schematic diagram of the feature fusion process. According to the output size of the cross-scale layer determined in advance, sample the image feature map of each dimension of the sample image to be segmented. Among them, the sampling process includes any one of upsampling and downsampling to obtain the target image feature map corresponding to the image feature map of each dimension. The size of the obtained target image feature map is the same as the output size of the cross-scale layer determined in advance.

[0128] Perform feature concatenation processing on the obtained multiple target image feature maps, associate the multiple target image feature maps, and perform convolution operations on the multiple target image feature maps after feature concatenation processing to fuse the multiple target image feature maps to obtain the image fusion feature map corresponding to the sample image to be segmented.

[0129] Among them, the purpose of upsampling is to enlarge the size of the original image feature map so that the image feature map retains more details. The purpose of downsampling is to reduce the size of the original image feature map so that the image feature map is more abstract and more suitable for image segmentation.

[0130] The principle of downsampling is as follows: For an image I with a size of M*N, perform s-fold downsampling on it, that is, obtain a resolution image with a size of (M / s)*(N / s). Of course, s should be a common divisor of M and N. If considering the image in matrix form, it is to turn the image within the s*s window of the original image into a single pixel, and the value of this pixel is the average value of all pixels within the window.

[0131] The principle of upsampling is as follows: the interpolation method is adopted, that is, on the basis of the original image pixels, appropriate interpolation algorithms are used to insert new elements between pixel points.

[0132] Feature cascading processing can associate multiple target image feature maps obtained after sampling processing through a trained cascaded classifier.

[0133] In this way, by performing feature fusion processing on the sample image to be segmented, all image features of the sample image to be segmented can be fully utilized, realizing the reuse of image features, avoiding repeated calculations, improving the effectiveness of model parameters. At the same time, since each image feature is shared, the number of channels of image features can be greatly reduced, reducing feature redundancy.

[0134] In addition, we introduce the concept of cross-scale, that is, different scale features are selected for fusion in the intermediate module and then output to the Decoder module. The intermediate cross-scale module can have two advantages: on the one hand, it can fuse multi-scale features to generate new features, increasing the richness of features; on the other hand, it can dynamically adjust the receptive field according to different tasks. The cross-scale operation is equivalent to stacking 3x3 convolutions on the basis of the original network, which is equivalent to deepening the depth of the network and increasing the receptive field.

[0135] Further, updating the last control network based on the segmentation return rate to obtain the updated current control network includes: determining the optimal expected value of the last control network based on the segmentation return rate; adjusting the second network parameters of the last control network based on the optimal expected value to obtain the updated current control network.

[0136] In this step, according to the segmentation return rate of the current segmentation network to be trained, the optimal expected value of the last control network is calculated; using the obtained optimal expected value, the expected return rate of the last control network is determined, so as to adjust the second network parameters in the last control network to obtain the updated current control network.

[0137] Specifically, the expected return rate of the last control network is calculated by the following formula:

[0138]

[0139] where, is the expected return rate of the last control network, is the optimal expected value, R(τ) is the segmentation return rate of the current segmentation network to be trained, and p θ (τ) is the sampling probability of the last control network.

[0140] In this way, the present application determines the current node parameters and current weight parameters for constructing each execution layer in the search space corresponding to the current segmentation network to be trained based on obtaining the previous control network that was updated last time, and thus constructs the current segmentation network to be trained based on the current node parameters and current weight parameters of each execution layer. At the same time, the constructed current segmentation network to be trained is trained based on the obtained training image set, and the segmentation return rate of the current segmentation network to be trained that has been trained is determined based on the obtained validation image set. Then, the previous control network is updated based on the calculated segmentation return rate of the current segmentation network to be trained to obtain the updated current control network. The updated current control network is used as the previous control network, and the network parameter update continues until the preset update times are reached, determining that the control network parameters are finally updated. Among the multiple current segmentation networks to be trained that have been trained during the network parameter update process, the segmentation accuracy of the determined multiple current segmentation networks to be trained is verified, and the current segmentation network with the best segmentation effect is selected as the image segmentation model for subsequent image segmentation. In summary, the present application can continuously adjust the structure of the network during the training process of the model, and train an image segmentation model with a better segmentation effect, so that the target object can be more accurately segmented from the image.

[0141] Please refer to Figure 6 , Figure 6 which is a flowchart of an image segmentation method provided by an embodiment of the present application. As Figure 6 shown in

[0142] S601. Obtain the image to be segmented.

[0143] S602. Input the image to be segmented into the image segmentation model obtained by the method for generating an image segmentation model based on a search space provided by an embodiment of the present application, and obtain the segmentation result of the target object in the image to be segmented.

[0144] In this step, the image to be segmented that needs to be segmented is obtained, and the image to be segmented is input into the target segmentation model to obtain the segmentation result of the target object in the image to be segmented.

[0145] Exemplarily, when the image to be segmented is a medical image, the target object may be a blood vessel, and the medical image is input into the image segmentation model to obtain the segmented image of the blood vessel; for another example, when the image to be segmented is a video surveillance image, the target object may be a person in the image, and the video surveillance image is input into the target segmentation model to obtain the segmented image of the person.

[0146] Please refer to Figure 7 , Figure 8 , Figure 7Schematic structural diagram of an image segmentation model generation device provided by an embodiment of the present application Figure 8 Schematic structural diagram of an image segmentation device provided by an embodiment of the present application. As Figure 7 shown in the figure, the image segmentation model generation device 700 includes:

[0147] A model construction module 710, configured to determine current node parameters and current weight parameters of each execution layer in the search space for constructing the current segmentation network to be trained based on the previously updated last control network obtained, so as to construct the current segmentation network to be trained;

[0148] A calculation module 720, configured to train the current segmentation network to be trained based on a training image set, and determine the segmentation return rate of the current segmentation network to be trained that has been trained based on the obtained validation image set;

[0149] A network update module 730, configured to update the last control network based on the segmentation return rate to obtain an updated current control network;

[0150] A loop training module 740, configured to use the current control network as the last control network, and continue to update network parameters until a preset number of updates is reached, and determine that the control network is finally updated;

[0151] A model determination module 750, configured to determine an image segmentation model from multiple current segmentation networks to be trained that have been trained during the network parameter update process.

[0152] Further, when the model determination module 750 is used to determine an image segmentation model from multiple current segmentation networks to be trained that have been trained during the network parameter process, the model determination module 750 is configured to:

[0153] Determine the segmentation accuracy of each current segmentation network to be trained based on the validation image set;

[0154] Determine the current segmentation network to be trained with the highest segmentation accuracy among the multiple current segmentation networks to be trained as the image segmentation model.

[0155] Further, when the calculation module 720 is used to train the current segmentation network to be trained based on a training image set and determine the segmentation return rate of the current segmentation network to be trained that has been trained based on the obtained validation image set, the calculation module 720 is configured to:

[0156] Obtain a training image set and a validation image set;

[0157] Input each sample image to be segmented in the training image set into the currently to-be-trained segmentation network that has been constructed, and obtain the predicted segmentation label corresponding to each sample image to be segmented;

[0158] Based on the predicted segmentation label corresponding to each sample image to be segmented and the true segmentation label corresponding to each sample image to be segmented in the training image set, determine the loss value of the currently to-be-trained segmentation network;

[0159] Based on the loss value, adjust the first network parameters of the currently to-be-trained segmentation network until the number of training times of the currently to-be-trained segmentation network reaches the preset number of training times, and determine that the training of the currently to-be-trained segmentation network is completed;

[0160] Input the validation image set into the currently to-be-trained segmentation network that has been trained, and determine the segmentation return rate of the currently to-be-trained segmentation network that has been trained.

[0161] Further, when the calculation module 720 is used to input each sample image to be segmented in the training image set into the currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to each sample image to be segmented, the calculation module 720 is used for:

[0162] For each sample image to be segmented in the training image set, input the sample image to be segmented into the encoding layer in the currently to-be-trained segmentation network to obtain the encoded sample image to be segmented;

[0163] Input the encoded sample image to be segmented into the cross-scale layer in the currently to-be-trained segmentation network, and extract the image feature maps of multiple dimensions of the sample image to be segmented from the encoded sample image to be segmented;

[0164] Perform feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the sample image to be segmented;

[0165] Input the image fusion feature map into the decoding layer in the currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to the sample image to be segmented.

[0166] Further, when the calculation module 720 is used to perform feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the sample image to be segmented, the calculation module 720 is used for:

[0167] According to the output size of the cross-scale layer, perform sampling processing on the image features of each dimension to obtain the target image feature map corresponding to the image feature map of each dimension;

[0168] Cascade the obtained multiple target image feature maps to determine the image fusion feature map corresponding to the sample image to be segmented.

[0169] Further, when the network update module 730 is used to update the last control network based on the segmentation return rate to obtain the current control network after the update, the network update module 730 is used to:

[0170] Determine the optimal expected value of the last control network based on the segmentation return rate;

[0171] Adjust the second network parameters of the last control network based on the optimal expected value to obtain the current control network after the update.

[0172] Further, as Figure 8 shown in

[0173] An image acquisition module 810, configured to acquire an image to be segmented;

[0174] An image segmentation module 820, configured to input the image to be segmented into an image segmentation model obtained by the above-mentioned method for generating an image segmentation model based on a search space, and obtain a segmentation result of a target object in the image to be segmented.

[0175] In this way, in this application, by obtaining the last control network that has been updated last time, the current node parameters and current weight parameters of each execution layer in the search space corresponding to the current segmentation network to be trained are determined, and then the current segmentation network to be trained is constructed based on the current node parameters and current weight parameters of each execution layer; at the same time, the constructed current segmentation network to be trained is trained based on the obtained training image set, and the segmentation return rate of the current segmentation network to be trained after training is determined based on the obtained verification image set; then the last control network is updated based on the calculated segmentation return rate of the current segmentation network to be trained to obtain the current control network after the update; the current control network after the update is used as the last control network, and the network parameters are continuously updated until the preset number of updates is reached, and it is determined that the control network parameters are finally updated; an image segmentation model that can be used for subsequent image segmentation is determined from the multiple current segmentation networks that have been trained during the network parameter update process. In this way, the structure of each segmentation network can be continuously adjusted during the training process of the model, and an image segmentation model with better segmentation effect can be trained, so that the target object can be more accurately segmented from the image, which helps to improve the accuracy of the segmentation result.

[0176] Please refer to Figure 9 , Figure 9 which is a schematic structural diagram of an electronic device provided by an embodiment of this application. AsFigure 9 As shown in Figure 9 , the electronic device 900 includes a processor 910, a memory 920, and a bus 930.

[0177] The memory 920 stores machine-readable instructions executable by the processor 910. When the electronic device 900 runs, the processor 910 communicates with the memory 920 via the bus 930. When the machine-readable instructions are executed by the processor 910, the steps of the method for generating an image segmentation model based on a search space in the method embodiments as described above Figure 1 and Figure 4 the steps of the image segmentation method in the method embodiments as shown in Figure 4 can be executed. For the specific implementation manners, reference can be made to the method embodiments and will not be elaborated herein. Figure 6 The steps of the image segmentation method in the method embodiments as shown in Figure 6 can be executed. For the specific implementation manners, reference can be made to the method embodiments and will not be elaborated herein.

[0178] The embodiments of the present application further provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is run by a processor, the steps of the method for generating an image segmentation model based on a search space in the method embodiments as described above Figure 1 and Figure 4 the steps of the image segmentation method in the method embodiments as shown in Figure 4 can be executed. For the specific implementation manners, reference can be made to the method embodiments and will not be elaborated herein. Figure 6 The steps of the image segmentation method in the method embodiments as shown in Figure 6 can be executed. For the specific implementation manners, reference can be made to the method embodiments and will not be elaborated herein.

[0179] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0180] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division manners in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some communication interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, in each embodiment of the present application, each functional unit may be integrated into one processing unit, may exist separately as individual physical units, or two or more units may be integrated into one unit.

[0183] If the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0184] Finally, it should be noted that the above-mentioned embodiments are only specific implementation manners of the present application, used to illustrate the technical solutions of the present application, rather than limiting them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present application can still modify the technical solutions recorded in the foregoing embodiments, or can easily think of changes, or perform equivalent replacements for some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating an image segmentation model based on a search space, characterized in that The method for generating the image segmentation model includes: Based on the previously updated last control network obtained, determine the current node parameters and current weight parameters for each execution layer in the search space corresponding to the currently to-be-trained segmentation network, so as to construct the currently to-be-trained segmentation network; Train the currently to-be-trained segmentation network based on the training image set, and determine the segmentation return rate of the currently to-be-trained segmentation network that has been trained based on the obtained validation image set; Update the last control network based on the segmentation return rate to obtain the currently updated current control network; Use the current control network as the last control network, and continue to update the network parameters until the preset number of updates is reached, and determine that the control network is finally updated; Determine the image segmentation model from multiple currently to-be-trained segmentation networks that have been trained during the network parameter update process.

2. The method for generating an image segmentation model according to claim 1, wherein The determining the image segmentation model from multiple currently to-be-trained segmentation networks that have been trained during the network parameter update process includes: Determine the segmentation accuracy of each currently to-be-trained segmentation network based on the validation image set; Determine the currently to-be-trained segmentation network with the highest segmentation accuracy among the multiple currently to-be-trained segmentation networks as the image segmentation model.

3. The method for generating an image segmentation model according to claim 1, wherein The training the currently to-be-trained segmentation network based on the training image set and determining the segmentation return rate of the currently to-be-trained segmentation network that has been trained based on the obtained validation image set includes: Obtain the training image set and the validation image set; Input each to-be-segmented sample image in the training image set into the constructed currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to each to-be-segmented sample image; Determine the loss value of the currently to-be-trained segmentation network based on the predicted segmentation label corresponding to each to-be-segmented sample image and the true segmentation label corresponding to each to-be-segmented sample image in the training image set; Adjust the first network parameters of the currently to-be-trained segmentation network based on the loss value until the number of training times of the currently to-be-trained segmentation network reaches the preset number of training times, and determine that the currently to-be-trained segmentation network is trained; Input the validation image set into the currently to-be-trained segmentation network that has been trained, and determine the segmentation return rate of the currently to-be-trained segmentation network that has been trained.

4. The method for generating an image segmentation model according to claim 3, wherein The inputting each to-be-segmented sample image in the training image set into the constructed currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to each to-be-segmented sample image includes: For each to-be-segmented sample image in the training image set, input the to-be-segmented sample image into the encoding layer in the currently to-be-trained segmentation network to obtain the encoded to-be-segmented sample image; Input the encoded to-be-segmented sample image into the cross-scale layer in the currently to-be-trained segmentation network, and extract the image feature maps of multiple dimensions of the to-be-segmented sample image from the encoded to-be-segmented sample image; Perform feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the to-be-segmented sample image. Input the image fusion feature map into the decoding layer in the currently to-be-trained segmentation network to obtain the predicted segmentation label corresponding to the to-be-segmented sample image.

5. The method for generating an image segmentation model according to claim 4, wherein The performing feature fusion processing on the image feature maps of multiple dimensions to obtain the image fusion feature map corresponding to the to-be-segmented sample image includes: Performing sampling processing on the image features of each dimension according to the output size of the cross-scale layer to obtain the target image feature map corresponding to the image feature map of each dimension; Performing feature concatenation on the obtained multiple target image feature maps to determine the image fusion feature map corresponding to the to-be-segmented sample image.

6. The method for generating an image segmentation model according to claim 1, wherein The updating the last control network based on the segmentation return rate to obtain the updated current control network includes: Determining the optimal expected value of the last control network based on the segmentation return rate; Adjusting the second network parameters of the last control network based on the optimal expected value to obtain the updated current control network.

7. An image segmentation method, characterized in that, The image segmentation method includes: Obtaining an image to be segmented; Inputting the image to be segmented into an image segmentation model obtained by the method for generating an image segmentation model based on a search space according to any one of claims 1-6 to obtain the segmentation result of the target object in the image to be segmented.

8. An image segmentation model generation device based on a search space, characterized in that, The image segmentation model generating device includes: A model construction module, configured to determine the current node parameters and current weight parameters of each execution layer in the search space corresponding to the currently to-be-trained segmentation network based on the last control network that was updated last time and obtained, so as to construct the currently to-be-trained segmentation network; A calculation module, configured to train the currently to-be-trained segmentation network based on a training image set, and determine the segmentation return rate of the currently to-be-trained segmentation network that has been trained based on the obtained validation image set; A network update module, configured to update the last control network based on the segmentation return rate to obtain the updated current control network; A loop training module, configured to use the current control network as the last control network and continue to update network parameters until a preset number of updates is reached, and determine that the control network has been finally updated; A model determination module, configured to determine an image segmentation model from multiple currently to-be-trained segmentation networks that have been trained during the network parameter update process.

9. An electronic device, characterized in that, including: A processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device runs, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to execute the steps of the method for generating an image segmentation model based on a search space according to any one of claims 1 to 6 and / or execute the steps of the image segmentation method according to claim 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the steps of the method for generating an image segmentation model based on a search space according to any one of claims 1 to 6 and / or executes the steps of the image segmentation method according to claim 7.

Images