Model training, image segmentation method and device

Abstract

The application discloses a model training method and device, and relates to the technical field of image processing. A specific implementation of the method comprises the following steps: obtaining a training sample set, improving a preset neural network to obtain a target neural network by the following manner: in a preset neural network encoder, an attention map generation module is arranged between each two adjacent sub-modules of at least two adjacent sub-modules; a target loss function is determined based on a first loss function; and the target neural network is trained based on the target loss function and a labeled sample image set to obtain a segmentation model. The implementation effectively improves the segmentation precision of the segmentation model obtained through training.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical health, in particular to the technical field of image processing, and especially to a model training method and device and an image segmentation method and device. BACKGROUND

[0002] By taking advantage of the efficiency and large data capacity of artificial intelligence technology, different symptoms of children's lung diseases can be quickly standardized and processed, a robust and generalizable children's pneumonia detection model can be trained in a smaller average training period, which is not restricted by space and time constraints for replication and deployment, reduces the cost of artificial labor and the cost of training full-time film readers, makes up for the shortage of doctors, improves the diagnosis and treatment efficiency of children's lung disease detection hospitals, and has broad application prospects.

[0003] Currently, common algorithms mainly include image segmentation algorithms and knowledge distillation. The image segmentation algorithm performs pixel-level segmentation on the uniform weight pixel array in the image without any discrimination, which may cause the algorithm to pay more attention to global defects, and give up searching for the difficult areas that really need attention under the action of the loss function, resulting in poor model performance. The knowledge distillation method focuses more on the transfer of knowledge from a large model to a small model, and ignores the importance of context information on the input data. The relationship between classes is not modeled, and the risk of pixel misclassification is high. SUMMARY

[0004] The embodiments of the present application provide a model training method and device, equipment and a storage medium.

[0005] According to a first aspect, the embodiments of the present application provide a model training method, which comprises: obtaining a training sample set, and improving a preset neural network to obtain a target neural network by the following way: setting an attention map generation module between each two adjacent sub-modules in a preset neural network encoder; determining a target loss function based on a first loss function; training the target neural network based on the target loss function and a labeled sample image set to obtain a segmentation model.

[0006] According to a second aspect, the embodiments of the present application provide an image segmentation method, which comprises: obtaining an image to be segmented; and inputting the image to be segmented into a segmentation model to obtain a semantic segmentation result of the image to be segmented, wherein the segmentation model is obtained by the method described in any of the implementations of the first aspect.

[0007] According to a third aspect, embodiments of the present application provide a model training apparatus, the apparatus comprising: an obtaining module configured to obtain a training sample set; an improving module configured to improve a preset neural network to obtain a target neural network by: setting an attention map generation module between each two adjacent sub-modules in a preset neural network encoder; determining a target loss function based on a first loss function; and training the target neural network based on the target loss function and a set of labeled sample images to obtain a segmentation model.

[0008] According to a fourth aspect, embodiments of the present application provide an image segmentation apparatus, the apparatus comprising: an obtaining module configured to obtain an image to be segmented; and a segmentation module configured to input the image to be segmented into a segmentation model to obtain a semantic segmentation result of the image to be segmented, wherein the segmentation model is obtained by the method according to any implementation manner of the first aspect.

[0009] According to a fifth aspect, embodiments of the present application provide an electronic device, comprising one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any implementation manner of the first aspect or the second aspect.

[0010] According to a sixth aspect, embodiments of the present application provide a computer readable medium having a computer program stored thereon, when the program is executed by a processor, the method according to any implementation manner of the first aspect or the second aspect is implemented.

[0011] The present application obtains a training sample set, improves a preset neural network to obtain a target neural network by: setting an attention map generation module between each two adjacent sub-modules in a preset neural network encoder; determining a target loss function based on a first loss function; and training the target neural network based on the target loss function and a set of labeled sample images to obtain a segmentation model, thereby improving the segmentation accuracy of the obtained segmentation model.

[0012] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;

[0014] Figure 2ais a flowchart of one embodiment of the model training method according to the present application;

[0015] Figure 2b is a schematic diagram of one embodiment of the model training method according to the present application;

[0016] Figure 2c is a schematic diagram of another embodiment of the model training method according to the present application;

[0017] Figure 2d is a schematic diagram of another embodiment of the model training method according to the present application;

[0018] Figure 3 is a schematic diagram of one application scenario of the model training method according to the present application;

[0019] Figure 4 is a flowchart of another embodiment of the image segmentation method according to the present application;

[0020] Figure 5 is a schematic diagram of one embodiment of the model training apparatus according to the present application;

[0021] Figure 6 is a schematic diagram of one embodiment of the image segmentation apparatus according to the present application;

[0022] Figure 7 is a structural schematic diagram of a computer system of a server suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present application are described herein with reference to the accompanying drawings, which are meant to be exemplary and not limiting. Therefore, it should be recognized that many modifications and variations can be made to the embodiments described herein without departing from the spirit and scope of the application. Also, for the purpose of clarity and a concise description, descriptions of well-known functions and constructions can be omitted.

[0024] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] Figure 1 An exemplary system architecture 100 to which the embodiments of the model training method of the present application can be applied is shown.

[0026] As Figure 1As shown, the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0027] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as image segmentation applications, communication applications, etc.

[0028] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to mobile phones and laptops. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (e.g., to provide a model training service), or as a single software or software module. No specific limitation is made herein.

[0029] The server 105 can be a server that provides various services, such as obtaining a training sample set, improving a preset neural network to obtain a target neural network by: setting an attention map generation module between each two adjacent sub-modules in a preset neural network encoder; determining a target loss function based on a first loss function; and training the target neural network based on the target loss function and a set of labeled sample images to obtain a segmentation model.

[0030] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., to provide a training model service), or as a single software or software module. No specific limitation is made herein.

[0031] It should be noted that the model training method provided by the embodiments of the present disclosure can be executed by the server 105, or can be executed by the terminal device 101, 102, or 103, or can be executed by the server 105 and the terminal device 101, 102, or 103 in cooperation with each other. Accordingly, the device for model training includes various parts (for example, various units, sub-units, modules, and sub-modules), which can be all arranged in the server 105, or can be all arranged in the terminal device 101, 102, or 103, or can be arranged in the server 105 and the terminal device 101, 102, or 103 respectively.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above embodiments is only illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0033] FIG. 2 shows a flowchart 200 that can be applied to the embodiments of the model training method of the present application. In the present embodiment, the model training method includes the following steps:

[0034] Step 201: Obtain a training sample set.

[0035] In the present embodiment, the execution subject (such as the server 105 or the terminal device 101, 102, or 103 shown in FIG. 1) can obtain the training sample set from a server locally or remotely storing the training sample set. Figure 1

[0036] The training sample set includes a set of labeled sample images, and the set of labeled sample images is composed of a plurality of sample images containing target objects and labels corresponding to each sample image, wherein the label is used to indicate the class information of each pixel point of the sample image.

[0037] Here, the training sample set can be a training sample set of medical images, and the target object can be a human body part, such as the lung, heart, brain, etc., which is not limited in the present application.

[0038] Specifically, the sample image is a child lung X-ray image, and the class information of each pixel point (i, j) in the sample image is l ij ∈{1,...,N c}, where N c is the number of preset lung diseases.

[0039] Step 202: Improve the preset neural network to obtain a target neural network by the following method.

[0040] ​In this embodiment, the execution subject can improve the preset neural network in the following manner to obtain the target neural network. The preset neural network is an encoder-decoder structure, and the encoder includes multiple levels of sub-modules, each of which includes a convolutional layer, such as an FCN network, a U-Net network, and the like.

[0041] The execution subject can improve the preset neural network in the following manner to obtain the target neural network: in the encoder of the preset neural network, an attention map generation module is arranged between each two adjacent sub-modules of at least two adjacent sub-modules, the attention map generation module is configured to generate an attention map according to a feature map output by a lower-level sub-module of the two sub-modules, and input the attention map to a higher-level sub-module of the two sub-modules to supervise the generation of the attention map of the higher-level sub-module.

[0042] In some optional manners, the preset neural network is an end-to-end trained neural network U-Net.

[0043] In this implementation, the preset neural network is an end-to-end trained neural network U-Net. As known from the prior art, the U-Net network includes an encoder and a decoder, the encoder can have four sub-modules, each of which includes at least two convolutional layers, and each sub-module is followed by a down-sampling layer implemented by max pool. The decoder includes four sub-modules, and the resolution is sequentially increased by up-sampling operation until it is consistent with the resolution of the input image. The network also uses a skip connection to connect the up-sampling result with the output of the sub-module in the encoder having the same resolution, as the input of the next sub-module in the decoder, as shown in Figure 2b .

[0044] Specifically, as shown in Figure 2c , for the five sub-modules in the encoder of the U-Net network, the five sub-modules correspond to the five stages indicated by E1-E5 respectively, the levels of E1-E5 gradually decrease, E1 has the highest level and is configured to receive the input image, and E5 has the lowest level and is configured to extract feature maps output by each of the five sub-modules, each of the feature maps is denoted as f1-f5. For at least two adjacent sub-modules of the five sub-modules, such as two sub-modules (E2 stage and E3 stage), an attention map generation module is arranged between the two adjacent sub-modules to generate an attention map according to a feature map output by a lower-level sub-module of the two sub-modules. The attention map generation module can include an attention feature generator The attention feature output of the mth (such as E3) stage is

[0045]

[0046] wherein, Cm , H m , W m respectively represent the number of channels, height, and width of the attention feature.

[0047] Further, the attention feature A m is generated through a mapping function to generate an attention map. Here, p can be set according to actual requirements, such as p = 2.

[0048] In addition, in order to ensure that the output attention map is consistent with the size of the feature map of the previous stage, the attention map generation module further includes a spatial classifier (softmax) and an up-sampler, and the up-sampler is used to perform an up-sampling operation on the attention map The spatial classifier is used to perform a spatial classification operation Φ(·) on the up-sampled attention map, and the self-attention mapping function can be represented by the following formula:

[0049]

[0050] Since the U-Net network relieves the problem of loss of geometric detail information in the convolution and down-sampling process to a certain extent by connecting features of different levels through a step connection, using the U-Net network helps to improve the segmentation accuracy of the segmentation model obtained by training.

[0051] In step 203, a target loss function is determined based on the first loss function.

[0052] In this embodiment, the execution subject can construct the target loss function based on the first loss function. The first loss function can be constructed based on the output result, i.e., the prediction result, of the preset neural network for the sample image and the label of the sample image.

[0053] Here, the first loss function can include one or more loss functions that can further improve the performance of the model in existing technologies or future technologies, such as cross-entropy Loss, IOU Loss, Dice Loss, ExponentialLogarithmic Loss, etc., which are not limited in the present application.

[0054] In some optional manners, the target loss function is determined based on the first loss function, including: determining the target loss function based on the first loss function and a second loss function.

[0055] In this implementation, the execution subject can determine the target loss function based on the first loss function and the second loss function.

[0056] The second loss function, i.e., the self-attention distillation loss, can be determined based on the attention map output by the attention map generation module and the attention map output by the high-level sub-module of the two sub-modules corresponding to the attention map generation module.

[0057] Here, the second loss function Specifically, the first loss function can be represented by the following formula:

[0058]

[0059] wherein, is the sub-loss function corresponding to the two sub-modules corresponding to each attention map generation module, and K-1 is the same as the number of attention map generation modules.

[0060] Specifically, the first loss function includes a cross-entropy loss function, and the target loss L can be represented by the following formula:

[0061]

[0062] wherein, is the cross-entropy loss function, s is a label, i.e., an expected output result, is a prediction result, i.e., a predicted segmentation image, and λ is a preset weight, λ ∈ (0, 1).

[0063] The implementation determines the target loss function based on the first loss function and the second loss function, and then trains the target neural network based on the target loss function and the training sample to obtain the trained target neural network, i.e., the segmentation model, which helps to further improve the segmentation accuracy of the determined segmentation model.

[0064] In some optional manners, the target loss function is determined based on the first loss function and the second loss function, including: determining the target loss function based on the cross-entropy loss function, the intersection over union loss function, and the second loss function.

[0065] In the implementation, the first loss function can include a cross-entropy loss function and an intersection over union loss function, and the execution subject can construct the target loss function based on the cross-entropy loss function, the intersection over union loss function, and the second loss function.

[0066] wherein, the intersection over union loss function aims to improve the intersection over union between s (the expected output result) and (the prediction result), and can be represented by the following formula:

[0067]

[0068] Here, N p is the pixel data in the prediction result, and N gis the number of pixels in the expected output result, i.e., the true value segmentation map, and N o is the number of pixels where the two intersect.

[0069] Specifically, the first loss function includes a cross-entropy loss function and an intersection-over-union loss function, and the target loss L can be represented by the following formula:

[0070]

[0071] wherein, is the cross-entropy loss function, is the intersection-over-union loss function, s is a label, i.e., an expected output result, is a prediction result, i.e., a predicted segmentation image, and λ is a preset weight, λ ∈ (0, 1).

[0072] The implementation determines the target loss function based on the cross-entropy loss function, the intersection-over-union loss function, and the second loss function, and then trains the target neural network based on the target loss function and the training samples to obtain the trained target neural network, i.e., the segmentation model, which helps to further improve the segmentation accuracy of the determined segmentation model.

[0073] In step 204, the target neural network is trained based on the target loss function and the set of labeled sample images to obtain the segmentation model.

[0074] In the present embodiment, the execution subject can train the target neural network based on the target loss function and the set of labeled sample images, and end the training to obtain the trained target neural network, i.e., the segmentation model, when a training end condition is met.

[0075] The training end condition can include, but is not limited to, at least one of the following: the training time exceeds a preset length of time; the training frequency exceeds a preset number of times; and the loss value calculated using the target loss function is less than a preset loss value threshold.

[0076] Here, it should be noted that, in the training process, for the feature output of each stage of the encoder, the feature needs to be upsampled and feature spliced cat(·) to obtain the output of the decoding stage, which is specifically shown in the following formula:

[0077]

[0078] The last stage feature f'1 is calculated, and the prediction result of the training sample can be obtained through the detection head.

[0079] In some alternative approaches, in a preset neural network encoder, an attention map generation module is set between every two adjacent sub-modules of at least two adjacent sub-modules, including: setting an attention map generation module between every two adjacent sub-modules of at least three sub-modules in the encoder.

[0080] In this implementation, the execution entity can improve the preset neural network to obtain the target neural network in the following way: between every two adjacent sub-modules in all sub-modules of the preset neural network encoder, an attention map generation module is set. The attention map generation module is used to generate an attention map based on the feature map output by the lower-level sub-module in the two sub-modules, and input the attention map into the higher-level sub-module in the two sub-modules to supervise the generation of the attention map of the higher-level sub-module.

[0081] Specifically, such as Figure 2d As shown, the U-Net network encoder has five sub-modules, each corresponding to one of the five stages indicated by E1-E5. The levels of E1-E5 gradually decrease, with E1 being the highest level and used to receive the input image, and E5 being the lowest level. Feature maps output by each of the five sub-modules are extracted, and each feature map is represented as f1-f5. For every two adjacent sub-modules (e.g., the sub-modules corresponding to stages E1 and E2, E2 and E3, E3 and E4, and E4 and E5), an attention map generation module is set between each pair of adjacent sub-modules to generate an attention map based on the feature map output by the lower-level sub-module in the corresponding two sub-modules. Each attention map generation module can include an attention feature generator.

[0082] This implementation improves the segmentation accuracy of the trained segmentation model by setting an attention map generation module between every two adjacent sub-modules in all sub-modules of the encoder.

[0083] In some alternative approaches, the method further includes: inputting unlabeled sample images into the segmentation model to obtain a set of pseudo-labeled sample images corresponding to the unlabeled sample images; merging the set of pseudo-labeled sample images and the set of labeled sample images, and retraining the segmentation model to obtain a new segmentation model.

[0084] In this implementation, the training sample set includes an unlabeled sample image set. After obtaining the segmentation model, the execution subject can input the unlabeled sample images into the trained target neural network, i.e., the segmentation model, to obtain a pseudo-labeled sample image set corresponding to the unlabeled sample images. The pseudo-labeled sample image set and the labeled sample image set are merged, and the segmentation model is retrained to obtain a new segmentation model.

[0085] The implementation inputs the unlabeled sample image into the segmentation model to obtain a set of pseudo-labeled sample images corresponding to the unlabeled sample image; the pseudo-labeled sample image set and the labeled sample image set are merged to retrain the segmentation model, and a new segmentation model is obtained, that is, the trained target neural network is further trained and updated in a semi-supervised manner, effectively improving the generalization ability of the segmentation model.

[0086] Continuing to refer to Figure 3 , Figure 3 is one of the application scenarios of the model training method according to the embodiment.

[0087] In Figure 3 application scenarios, the execution subject 301 can obtain a training sample set from a local or remote server storing the training sample set, and the training sample set includes a labeled sample image set 302, which is composed of a plurality of sample images containing target objects, such as children's lungs, and corresponding labels of each sample image. The label is used to indicate the class information of each pixel point of the sample image, such as lung disease categories. The U-Net network is improved in the following way to obtain an improved U-Net network 303, i.e., a target neural network: in the U-Net network encoder, at least two adjacent sub-modules, such as the sub-modules corresponding to the E2 stage and the E3 stage, are set between each two adjacent sub-modules, and the attention map generation module is used to generate an attention map according to the feature map (semantic features of the children's lung image) output by the low-level sub-module of the corresponding two sub-modules. Here, the attention map encodes rich context information of the lung lesion area. Further, the attention map is input into the high-level sub-module of the two sub-modules to supervise the generation of the attention map of the high-level sub-module, i.e., the texture of the lesion area in the children's lung image is learned in a refined manner using self-attention knowledge distillation.

[0088] Further, based on the output result of the improved U-Net network for the sample image, i.e., the prediction result, and the label of the sample image (the class information of each pixel point of the children's lung image), a first loss function 304 is constructed, based on the attention map output by the attention map generation module and the attention map output by the high-level sub-module of the two sub-modules corresponding to the attention map generation module, a second loss function 305 is determined, based on the first loss function 304 and the second loss function 305, a target loss function 306 is constructed, based on the target loss function 306, the labeled sample image set 302, the improved U-Net network 303 is trained to obtain a segmentation model 307.

[0089] The method for training the model of the present disclosure, by obtaining a training sample set, improving the U-Net network in the following manner to obtain an improved U-Net network: setting an attention map generation module between each two adjacent sub-modules in the U-Net network encoder, the attention map generation module being used to generate an attention map according to the feature map output by the low-level sub-module of the two sub-modules, and inputting the attention map into the high-level sub-module of the two sub-modules to supervise the generation of the attention map of the high-level sub-module, determining a target loss function based on a first loss function and a second loss function, training the improved U-Net network based on the target loss function and the labeled sample image set to obtain a segmentation model, effectively improving the segmentation accuracy and robustness of the trained segmentation model.

[0090] With reference to Figure 4 , a flow 400 of one embodiment of the image segmentation method according to the present application is shown. The image segmentation method comprises the following steps:

[0091] Step 401, obtaining an image to be segmented.

[0092] In this embodiment, the execution subject can obtain the image to be segmented containing the target object, i.e. the image to be segmented, through wired or wireless connection.

[0093] Here, the target object can be a human body part, for example, the lung, heart, brain, etc. The target object can have different lesion areas.

[0094] Among them, the wireless connection mode can include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection modes.

[0095] Step 402, inputting the image to be segmented into a segmentation model to obtain a semantic segmentation result of the image to be segmented.

[0096] In this embodiment, after obtaining the image to be segmented, the execution subject can input the image to be segmented into the trained improved U-Net network, i.e. the segmentation model, to obtain the semantic segmentation result of the image to be segmented. The segmentation model is the segmentation model obtained by the method described in the corresponding embodiment of FIG. 2, which is not described here.

[0097] Specifically, the execution subject can obtain the image to be segmented containing the child lung image with the lesion area through wired or wireless manner, input the image to be segmented into the segmentation model, and obtain the semantic segmentation result of the lesion area in the child lung image.

[0098] The image segmentation method provided in the embodiments of the present disclosure comprises: obtaining a to-be-segmented image containing a target object; inputting the to-be-segmented image into a segmentation model to obtain a semantic segmentation result of the to-be-segmented image, wherein the segmentation model is a segmentation model obtained by the method described in the embodiment of FIG. 2, that is, for input image data (such as X image data of a child's lung disease), an attention map is extracted at a proper layer of the model, the attention map encodes rich context information of a lesion area, the context information of the lesion area is transformed as a supervision signal to guide the model learning, and self-attention knowledge distillation is performed in a hierarchical form in the model, so that the model learns more refined pathological features, thereby effectively improving the segmentation accuracy of the to-be-segmented image.

[0099] Further referring to Figure 5 , as an implementation of the method shown in the above figures, the present application provides an embodiment of a model training device, which corresponds to the method embodiment shown in FIG. 2, and the device can be specifically applied to various electronic devices.

[0100] As shown in Figure 5 , the model training device 500 of the present embodiment comprises an obtaining module 501, an improving module 502, a determining module 503 and a training module 504.

[0101] The obtaining module 501 can be configured to obtain a training sample set.

[0102] The improving module 502 can be configured to improve the U-Net network to obtain an improved U-Net network by: in the U-Net network encoder, setting an attention map generation module between each two adjacent sub-modules of at least two adjacent sub-modules.

[0103] The determining module 503 can be configured to determine a target loss function based on a first loss function.

[0104] The training module 504 can be configured to train the improved U-Net network based on the target loss function and a labeled sample image set to obtain a segmentation model.

[0105] In some optional manners of the present embodiment, the determining module is further configured to determine the target loss function based on the first loss function and a second loss function.

[0106] In some optional manners of the present embodiment, the determining module is further configured to determine the target loss function based on a cross-entropy loss function, an intersection-over-union loss function and the second loss function.

[0107] In some optional manners of the embodiment, in the preset neural network encoder, an attention map generation module is arranged between each two adjacent sub-modules in the preset neural network encoder.

[0108] In some optional manners of the embodiment, the apparatus further comprises an updating module configured to input the unlabeled sample image into the segmentation model to obtain a pseudo-labeled sample image set corresponding to the unlabeled sample image; and merge the pseudo-labeled sample image set and the labeled sample image set to retrain the segmentation model to obtain a new segmentation model.

[0109] In some optional manners of the embodiment, the preset neural network is an end-to-end trained neural network U-Net.

[0110] Further referring to Figure 6 , as an implementation of the method shown in the above figures, the present disclosure provides an embodiment of an image segmentation apparatus, which corresponds to the method embodiment shown in Figure 4 , and the apparatus can be applied in various electronic devices.

[0111] As shown in Figure 6 , the image segmentation apparatus 600 of the embodiment comprises an obtaining module 601 and a segmentation module 602.

[0112] The obtaining module 601 can be configured to obtain an image to be segmented.

[0113] The segmentation module 602 can be configured to input the image to be segmented into a segmentation model to obtain a semantic segmentation result of the image to be segmented.

[0114] According to the embodiments of the present application, the present application further provides an electronic device and a readable storage medium.

[0115] As shown in Figure 7 , is a block diagram of an electronic device according to the method of model training of the embodiments of the present application.

[0116] 700 is a block diagram of an electronic device according to the method of model training of the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed in this document.

[0117] As shown in Figure 7 The electronic device includes one or more processors 701, memory 702, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected utilizing different buses, and can be mounted on a common motherboard or in other manners, as desired. The processor can process instructions for execution within the electronic device, including instructions stored in the memory or on the memory to display graphical information for a GUI on an external input / output device, such as a display device coupled to the interface. In other implementations, multiple processors and / or multiple buses can be employed as desired to implement these functions, and multiple memories and types of memory can be used. Also, various Figure 7 The processor 701 is taken as an example in the method.

[0118] The memory 702 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the model training method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the model training method provided by the present application.

[0119] The memory 702 is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules of the model training method in the embodiments of the present application (for example, the acquisition module 501, the improvement module 502, the determination module 503 and the training module 504 shown in the figure). The processor 701 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 702, that is, implements the model training method in the method embodiments. Figure 5

[0120] ​The memory 702 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs, and / or data required by at least one function. The data storage area can store data created by the electronic device for model training, etc. In addition, the memory 702 can include a high-speed random access memory, and can also include a non-transitory memory such as at least one disk storage device, a flash memory device, or other non-transitory solid state memory device. In some embodiments, the memory 702 can optionally include a memory that is remotely located from the processor 701, and can be connected to the electronic device for model training through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0121] The electronic device for model training method can further include an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703, and the output device 704 can be connected through a bus or other means, Figure 7 The bus connection is taken as an example.

[0122] The input device 703 can receive input digital or character information, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 704 can include a display device, an auxiliary lighting device (e.g., an LED), a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0123] Various embodiments of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0124] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" "computer-readable medium" refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0125] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0126] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0127] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0128] According to the technical scheme of the embodiment of the present application, the segmentation precision of the segmentation model obtained by training is effectively improved.

[0129] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in this application can be performed in parallel, in series, in a different order, or any combination thereof, as long as the desired results of the technology disclosed in this application are achieved, which is not limited herein.

[0130] The specific embodiments described above are not to be taken as limitations on the scope of the application. It is to be understood that various modifications, combinations, sub-combinations, and alternatives can be employed, as indicated by the spirit and principles of the application. Any and all modifications, equivalents, or alternatives falling within the spirit and principles of the application are intended to be included within the scope of the application.

Claims

1. A model training method, the method comprising: Obtain a training sample set, which includes a set of labeled sample images, wherein the labels are used to indicate the category information of each pixel in the sample images; The target neural network is obtained by improving the preset neural network in the following way: In the preset neural network encoder, an attention map generation module is set between every two adjacent sub-modules in at least two adjacent sub-modules. The attention map generation module is used to generate an attention map based on the feature map output by the lower-level sub-module in the corresponding two sub-modules, and input the attention map into the higher-level sub-module in the two sub-modules to supervise the generation of the attention map of the higher-level sub-module. The sub-module includes a convolutional layer. The preset neural network is an encoder-decoder structure. The decoder is used to upsample and concatenate the features of the encoder output. Based on the first loss function, a target loss function is determined, wherein the first loss function is constructed based on the output of the target neural network for the sample image and the label of the sample image; Based on the target loss function and the labeled sample image set, the target neural network is trained to obtain a segmentation model.

2. The method according to claim 1, wherein, The step of determining the target loss function based on the first loss function includes: Based on the first loss function and the second loss function, a target loss function is determined, wherein the second loss function is determined based on the attention map output by the attention map generation module and the attention map output by the higher-level sub-module among the two sub-modules corresponding to the attention map generation module.

3. The method according to claim 2, wherein, The first loss function includes a cross-entropy loss function and a cross-union ratio (CUI) loss function, and the determination of the target loss function based on the first and second loss functions includes: The target loss function is determined based on the cross-entropy loss function, the cross-union ratio loss function, and the second loss function.

4. The method according to claim 1, wherein, In the preset neural network encoder, an attention map generation module is set between every two adjacent sub-modules of at least two adjacent sub-modules, including: An attention map generation module is set between every two adjacent sub-modules in at least three sub-modules of the encoder.

5. The method according to claim 1, wherein, The training sample set further includes: an unlabeled sample image set, and the method further includes: Input the unlabeled sample image into the segmentation model to obtain the pseudo-labeled sample image set corresponding to the unlabeled sample image; The pseudo-labeled sample image set and the labeled sample image set are merged, and the segmentation model is retrained to obtain a new segmentation model.

6. The method according to any one of claims 1-5, wherein, The preset neural network is an end-to-end trained neural network U-Net.

7. An image segmentation method, the method comprising: Obtain the image to be segmented; The image to be segmented is input into a segmentation model to obtain a semantic segmentation result of the image to be segmented, wherein the segmentation model is a segmentation model obtained by the method described in any one of claims 1-6.

8. A model training device, comprising: The acquisition module is configured to acquire a training sample set, which includes a set of labeled sample images, wherein the labels are used to indicate the category information of each pixel in the sample images; An improvement module is configured to improve a preset neural network to obtain a target neural network in the following way: In the preset neural network encoder, an attention map generation module is set between every two adjacent sub-modules of at least two adjacent sub-modules. The attention map generation module is used to generate an attention map based on the feature map output by the lower-level sub-module in the corresponding two sub-modules, and input the attention map into the higher-level sub-module in the two sub-modules to supervise the generation of the attention map of the higher-level sub-module. The sub-module includes a convolutional layer. The preset neural network is an encoder-decoder structure. The decoder is used to upsample and concatenate the features of the encoder output. The determination module is configured to determine a target loss function based on a first loss function, wherein the first loss function is constructed based on the output of the target neural network for the sample image and the label of the sample image; The training module is configured to train the target neural network based on the target loss function and the labeled sample image set to obtain a segmentation model.

9. The apparatus according to claim 8, wherein, The determining module is further configured to: Based on the first loss function and the second loss function, a target loss function is determined, wherein the second loss function is determined based on the attention map output by the attention map generation module and the attention map output by the higher-level sub-module among the two sub-modules corresponding to the attention map generation module.

10. The apparatus according to claim 9, wherein, The first loss function includes a cross-entropy loss function and a cross-union ratio (CUI) loss function, and the determining module is further configured to: The target loss function is determined based on the cross-entropy loss function, the cross-union ratio loss function, and the second loss function.

11. The apparatus according to claim 8, wherein, In the preset neural network encoder, an attention map generation module is set between every two adjacent sub-modules of at least two adjacent sub-modules, including: An attention map generation module is set between every two adjacent sub-modules in at least three sub-modules of the encoder.

12. The apparatus according to claim 8, wherein, The training sample set further includes: an unlabeled sample image set, and the device further includes: The update module is configured to input unlabeled sample images into the segmentation model to obtain a set of pseudo-labeled sample images corresponding to the unlabeled sample images; merge the set of pseudo-labeled sample images and the set of labeled sample images, and retrain the segmentation model to obtain a new segmentation model.

13. The apparatus according to any one of claims 8-12, wherein, The preset neural network is an end-to-end trained neural network U-Net.

14. An image segmentation apparatus, comprising: The acquisition module is configured to acquire the image to be segmented. The segmentation module is configured to input the image to be segmented into a segmentation model to obtain a semantic segmentation result of the image to be segmented, wherein the segmentation model is a segmentation model obtained by the method described in any one of claims 1-6.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores information that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images