Network training method, image detection method and medium

Abstract

The present application relates to a network training method, an image detection method, an apparatus, a computer device and a storage medium. The method comprises: obtaining a set of training medical images; inputting each training medical image into an initial network, determining the predicted position of the region of interest in each training medical image through the backbone network and the first network in the initial network, and determining the predicted category corresponding to each training medical image through the backbone network and the second network in the initial network; calculating the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and calculating the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images; training the initial network according to the category loss and the position loss, and determining the trained network. The use of this method can reduce the cost of manual labeling.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a network training method, an image detection method, an apparatus, a computer device and a storage medium. Background Art

[0002] With the continuous development of medical imaging technology, more and more patients will be asked to take a medical image when they go to the hospital for examination. The doctor will then analyze the taken medical image to obtain the image analysis result of the patient.

[0003] In related technologies, when doctors analyze medical images, they usually combine deep learning technology to analyze medical images. That is, doctors will manually annotate multiple sample medical images in advance, annotate the location and category of all lesions on each sample medical image, and train the deep learning network through each annotated sample medical image to obtain a trained network. After that, doctors can analyze the patient's medical images through the well-trained network to obtain the patient's medical image analysis results.

[0004] However, using the above technology to train deep learning networks requires doctors to spend a lot of time and energy to label sample medical images during the training process, which will result in high manual labeling costs. Summary of the invention

[0005] Based on this, it is necessary to provide a network training method, image detection method, device, computer equipment and storage medium that can reduce the cost of manual labeling in response to the above technical problems.

[0006] A network training method, the network comprising a backbone network, a first network and a second network, the method comprising:

[0007] Obtaining a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and a location label corresponding to some training medical images;

[0008] Input each training medical image into the initial network, determine the predicted position of the region of interest in each training medical image through the backbone network and the first network in the initial network, and determine the predicted category corresponding to each training medical image through the backbone network and the second network in the initial network;

[0009] Calculating the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and calculating the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images;

[0010] The initial network is trained according to the category loss and the position loss to determine the trained network.

[0011] In one embodiment, the category label includes an image category label of a training medical image and / or a positive and negative sample label of the training medical image, the predicted category corresponding to the training medical image includes a predicted sample category of the training medical image and a predicted image category of the training medical image, and the calculating of the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image includes:

[0012] Calculate the first category loss between the predicted sample category and the corresponding positive and negative sample labels corresponding to each training medical image;

[0013] Calculate the second category loss between the image category label of each training medical image and the corresponding predicted image category;

[0014] Determine the category loss based on the first category loss and the second category loss.

[0015] In one embodiment, determining the category loss according to the first category loss and the second category loss includes:

[0016] Obtaining a positive sample heat map corresponding to each training medical image output by the second network and a category heat map corresponding to each training medical image output by the first network; the positive sample heat map includes predicted sample categories corresponding to the positive sample training medical images in each training medical image, and the category heat map includes predicted categories and predicted positions of regions of interest in the training medical images;

[0017] Calculate the heatmap loss between the positive sample heatmap and the category heatmap;

[0018] The class loss is determined based on the first class loss, the second class loss, and the heatmap loss.

[0019] In one embodiment, determining the category loss according to the first category loss, the second category loss and the heat map loss includes:

[0020] The first category loss, second category loss, and heatmap loss are summed to determine the category loss.

[0021] In one embodiment, the position labels include position labels of some or all regions of interest in the training medical images, and the calculating of the position loss between the predicted positions of the regions of interest in each training medical image and the position labels corresponding to some of the training medical images includes:

[0022] The loss between the predicted position of the region of interest in each training medical image and the position label of the region of interest in the corresponding training medical image is calculated to obtain the position loss.

[0023] In one embodiment, the initial network is trained according to the category loss and the position loss to determine the trained network, including:

[0024] Sum the category loss and position loss, and use the sum as the value of the loss function;

[0025] The initial network is trained using the value of the loss function to determine the trained network.

[0026] In one of the embodiments, the first network includes a first convolution module and a first output channel, and the first output channel is used to output a category heat map.

[0027] In one embodiment, the second network includes a second convolution module and a second output channel, a first fully connected layer and a third output channel, a second fully connected layer and a fourth output channel;

[0028] The second output channel is used to output the positive sample heat map, the third output channel is used to output the predicted sample category of the training medical image, and the fourth output channel is used to output the predicted image category of the training medical image.

[0029] An image detection method, the method comprising:

[0030] Acquire a medical image to be tested; the medical image to be tested includes at least one region of interest;

[0031] The medical image to be tested is input into the trained network for detection processing to determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

[0032] A network training device, the network comprising a backbone network, a first network and a second network, the device comprising:

[0033] A sample image acquisition module is used to acquire a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and a position label corresponding to some training medical images;

[0034] A prediction module, used to input each training medical image into the initial network, determine the predicted position of the region of interest in each training medical image through the backbone network and the first network in the initial network, and determine the prediction category corresponding to each training medical image through the backbone network and the second network in the initial network;

[0035] A loss calculation module, used to calculate the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and to calculate the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images;

[0036] The training module is used to train the initial network according to the category loss and the position loss to determine the trained network.

[0037] An image detection device, comprising:

[0038] A test image acquisition module is used to acquire a medical image to be tested; the medical image to be tested includes at least one region of interest;

[0039] The detection module is used to input the medical image to be tested into the trained network for detection processing, and determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the category and position of all areas of interest on the medical image to be tested.

[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] Obtaining a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and a location label corresponding to some training medical images;

[0042] Input each training medical image into the initial network, determine the predicted position of the region of interest in each training medical image through the backbone network and the first network in the initial network, and determine the predicted category corresponding to each training medical image through the backbone network and the second network in the initial network;

[0043] Calculating the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and calculating the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images;

[0044] The initial network is trained according to the category loss and the position loss to determine the trained network.

[0045] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0046] Acquire a medical image to be tested; the medical image to be tested includes at least one region of interest;

[0047] The medical image to be tested is input into the trained network for detection processing to determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

[0048] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0049] Obtaining a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and a location label corresponding to some training medical images;

[0050] Input each training medical image into the initial network, determine the predicted position of the region of interest in each training medical image through the backbone network and the first network in the initial network, and determine the predicted category corresponding to each training medical image through the backbone network and the second network in the initial network;

[0051] Calculating the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and calculating the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images;

[0052] The initial network is trained according to the category loss and the position loss to determine the trained network.

[0053] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0054] Acquire a medical image to be tested; the medical image to be tested includes at least one region of interest;

[0055] The medical image to be tested is input into the trained network for detection processing to determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

[0056] The above network training method, image detection method, device, computer equipment and storage medium, the network includes a backbone network, a first network and a second network, by obtaining a training medical image set, each training medical image in the training medical image set is input into the initial network, the predicted position of the region of interest in each training medical image is obtained through the backbone network and the first network, the predicted category corresponding to each training medical image is obtained through the backbone network and the second network, the category loss between the predicted category and the category label of each training medical image is calculated, and the position loss between the predicted position of the region of interest in each training medical image and the corresponding position label is calculated, and the initial network is trained according to the category loss and the position loss, and the network trained by Hu Dongen; wherein the training medical image set includes multiple training medical images, the category label corresponding to each training medical image, and the position label corresponding to some training medical images. In this method, since the training samples used in training the network include the category labels of each training medical image and the position labels of some training medical images, that is, the network can be trained without fully annotating the position labels of each training medical image, then it is not necessary to spend a lot of time and energy on the doctor to annotate the samples, so the use of this method can save the cost of manual annotation of samples. In addition, since the network in this method includes a backbone network, a first network and a second network, and the network can be trained through the prediction outputs of these three networks to obtain a trained network, it is possible to use samples with a small number of position labels to train the network and obtain a trained network. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is an internal structure diagram of a computer device in one embodiment;

[0058] Figure 2 A schematic diagram of a flow chart of a network training method in one embodiment;

[0059] Figure 2a A diagram showing an example of a network structure in one embodiment;

[0060] Figure 3 A schematic diagram of a flow chart of a network training step in another embodiment;

[0061] Figure 4 is a flowchart of a network training method in another embodiment;

[0062] Figure 4a This is a diagram showing a specific structure example of the first network and the second network in another embodiment;

[0063] Figure 5 1 is a schematic diagram of a flow chart of an image detection method in an embodiment;

[0064] Figure 6is a structural block diagram of a network training device in one embodiment;

[0065] Figure 7 FIG. 4 is a structural block diagram of an image detection device in one embodiment. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] The network training method provided in the embodiment of the present application can be applied to medical equipment, which includes a scanning device and a computer device that are interconnected. The scanning device can scan the detection object and send the obtained scanning data to the computer device, so that the computer device performs data processing, image reconstruction and other processes based on the obtained scanning data.

[0068] The computer device may be a terminal device or a server. Taking the computer device as a terminal device as an example, its internal structure diagram may be as follows: Figure 1 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a network training method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0069] Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0070] It should be noted that the execution subject of the embodiments of the present application can be a medical device, a computer device in a medical device, a network training device, or an image detection device. The network training device and the image detection device can be implemented as part or all of the computer device through software, hardware, or a combination of software and hardware. The following method embodiments are described by taking the execution subject as a computer device as an example.

[0071] The following embodiment first describes the network training process.

[0072] In one embodiment, a network training method is provided, the network comprising a backbone network, a first network and a second network, such as Figure 2 As shown, the method may include the following steps:

[0073] S202, obtaining a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and position labels corresponding to some training medical images.

[0074] Among them, each training medical image in the training medical image set can be a medical image of a different object, and each training medical image is marked with a category label, and some training medical images are marked with position labels. This can reduce the amount of time and energy that doctors need to spend on marking the position labels of all training medical images.

[0075] The category label here can be a positive or negative sample label of whether there is a region of interest in the training medical image, or an image category label of which region of interest is included in the training medical image, or both. The location label here can be the location of the region of interest in the training medical image.

[0076] It should be noted that if the training medical image includes regions of interest, the included regions of interest may be regions of interest of the same type or may be regions of interest of multiple types.

[0077] Specifically, before training the network, the regions of interest of different objects may be scanned in advance to obtain medical images of each object, and the medical images of each object may be labeled with categories and / or positions to obtain a training medical image set.

[0078] S204, input each training medical image into the initial network, determine the predicted position of the region of interest in each training medical image through the backbone network in the initial network and the first network, and determine the predicted category corresponding to each training medical image through the backbone network in the initial network and the second network.

[0079] The initial network refers to a network in which the network to be trained in this embodiment is in an initial untrained state. The network includes a backbone network, a first network, and a second network, and can also be called a weakly supervised learning network, wherein the backbone network is connected to the first network and the second network respectively. The backbone network can be a Backbone CNN network, and the first network and the second network can be convolutional neural networks (for example, the first network or the second network can be composed of a CNN convolution module and a fully connected layer). The specific structure of the network can be found in Figure 2a As shown, Backbone CNN is the backbone network, CNN convolution module and fully connected layer are the second network, and there can also be a network composed of CNN convolution module and / or fully connected layer before the lesion location information, which is the first network (not shown in the figure). Among them, bbox label is the location label, and class label is the category label.

[0080] It should be noted that Figure 2a It is just an example and does not affect the essential content of the embodiments of the present application.

[0081] The network structure composed of the above-mentioned backbone network and the first network is mainly used to identify the position of the region of interest in the training medical image, and the network structure composed of the backbone network and the second network is mainly used to identify the category of the training medical image. Then, after obtaining each training medical image, each training medical image can be input into the backbone network of the initial network respectively, and convolution operation processing and the like are performed in the backbone network. The result output by the backbone network can be input into the first network and the second network, and the result output by the backbone network is processed by convolution, pooling, activation, full connection and the like respectively through the first network and the second network. Finally, the predicted position of the region of interest in each training medical image can be output through the first network (i.e., the position of the region of interest in each training medical image is predicted), and the predicted category corresponding to each training medical image can be output through the second network (i.e., the category of each training medical image is predicted).

[0082] S206, calculating the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and calculating the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images.

[0083] In this step, after obtaining the predicted category corresponding to each training medical image and the predicted position of the region of interest in each training medical image, loss calculation can be performed with the corresponding label, that is, category loss calculation and position loss calculation can be performed.

[0084] Specifically, the category loss between the predicted category of each training medical image and the corresponding category label can be calculated. At the same time, the position loss between the position label of the training medical image marked with the position label and the predicted position of the region of interest in the corresponding training medical image can be calculated to obtain the category loss and the position loss.

[0085] S208, training the initial network according to the category loss and the position loss to determine a trained network.

[0086] In this step, after obtaining the category loss and the position loss, the category loss and the position loss can be used to adjust the parameters of the initial network respectively to obtain a trained network. Alternatively, the sum of the category loss and the position loss can be used to adjust the parameters of the initial network to obtain a trained network. Of course, the sum of the category losses of all training medical images and the sum of the position losses of all training medical images can be used to adjust the parameters of the initial network respectively to obtain a trained network. Of course, other training methods can also be used, which are not specifically limited here.

[0087] In the above network training method, the network includes a backbone network, a first network, and a second network. By obtaining a training medical image set, each training medical image in the training medical image set is input into the initial network, the predicted position of the region of interest in each training medical image is obtained through the backbone network and the first network, the predicted category corresponding to each training medical image is obtained through the backbone network and the second network, the category loss between the predicted category and the category label of each training medical image is calculated, and the position loss between the predicted position of the region of interest in each training medical image and the corresponding position label is calculated, and the initial network is trained according to the category loss and the position loss to obtain a trained network; wherein the training medical image set includes multiple training medical images, the category label corresponding to each training medical image, and the position label corresponding to some training medical images. In this method, since the training samples used in training the network include the category labels of each training medical image and the position labels of some training medical images, that is, the network can be trained without fully annotating the position labels of each training medical image, then it is not necessary to spend a lot of time and energy on doctors to annotate samples, so the use of this method can save the cost of manual annotation of samples. In addition, since the network in this method includes a backbone network, a first network and a second network, and the network can be trained through the prediction outputs of these three networks to obtain a trained network, it is possible to use samples with a small number of position labels to train the network and obtain a trained network.

[0088] In another embodiment, another network training method is provided. Based on the above embodiment, the above category labels include image category labels of training medical images and / or positive and negative sample labels of training medical images, and the prediction categories corresponding to the above training medical images include prediction sample categories of training medical images and prediction image categories of training medical images, such as Figure 3 As shown, the calculation process of the category loss in the above S206 may include the following steps:

[0089] S302, calculating the first category loss between the predicted sample category corresponding to each training medical image and the corresponding positive and negative sample labels.

[0090] The positive and negative sample labels in this embodiment refer to the labels of whether the training medical image is a positive sample or a negative sample, which represents whether the training medical image includes a region of interest. For example, if the training medical image includes a region of interest, the label of the training medical image is a positive sample label, otherwise it is a negative sample label.

[0091] The predicted sample category here, that is, the result of the training medical image predicted by the initial network as a positive sample or a negative sample, can be a binary classification prediction result.

[0092] Specifically, after obtaining the predicted sample category of each training medical image through the second network of the initial network, the loss between the predicted sample category of each training medical image and its corresponding positive and negative sample labels can be calculated, which is recorded as the first category loss. The first category loss here can be a binary cross entropy loss, etc.

[0093] S304, calculating the second category loss between the image category label of each training medical image and the corresponding predicted image category.

[0094] The image category label here refers to the category label of the region of interest included in the training medical image. For example, there are three types of regions of interest, with labels 1, 2, and 3 respectively. If a training medical image includes a region of interest, then the image category label of the training medical image can be 12, 23, 13, 123, etc., that is, the training medical image can include at least one type of region of interest.

[0095] The predicted image category refers to the category of the region of interest included in the training medical image predicted by the initial network.

[0096] Specifically, after obtaining the predicted image category of each training medical image through the second network of the initial network, the loss between the predicted image category of each training medical image and its corresponding image category label can be calculated, which is recorded as the second category loss. The second category loss here can be a multi-classification cross entropy loss, etc.

[0097] S306, determining the category loss according to the first category loss and the second category loss.

[0098] In this step, after obtaining the first category loss and the second category loss corresponding to each training medical image, the total category loss can be calculated. Optionally, this step may include the following steps A1-A3:

[0099] Step A1, obtaining a positive sample heat map corresponding to each training medical image output by the second network and a category heat map corresponding to each training medical image output by the first network; the positive sample heat map includes predicted sample categories corresponding to the positive sample training medical images in each training medical image, and the category heat map includes predicted categories and predicted positions of regions of interest in the training medical images.

[0100] The positive sample heat map here can also be called a positive sample feature map, which includes the category information of the training medical image as a positive sample, that is, the predicted sample category of the training medical image. The category heat map can also be called a category feature map, which includes the predicted category and predicted position of the region of interest in the training medical image. The predicted category of the region of interest refers to the predicted category of each region of interest, not the predicted image category of the training medical image. For example, the training medical image includes two types of regions of interest, and its image category is 12 or 23, or 13. The two types of regions of interest included therein will have categories of each region of interest, such as a type 1 region of interest 1, or a type 2 region of interest 2, or a type 3 region of interest 3, and so on.

[0101] Step A2, calculate the heatmap loss between the positive sample heatmap and the category heatmap.

[0102] Specifically, the following formula (1) can be used to calculate the heat map loss, as shown below:

[0103]

[0104] Where N is the number of regions of interest in the training medical images; M p is the positive sample heat map; M ak is the category heat map of the k-th region of interest; i and j are the pixel positions of the pixels on the heat map in the horizontal and vertical directions respectively; k is the k-th region of interest in the training medical image; y k What is calculated is the loss of the kth region of interest in the training medical image; T is the activation function.

[0105] Among them, M u To train the largest region of interest in a medical image, the calculation method is as shown in the following formula (2):

[0106]

[0107] The heat map loss between the positive sample heat map and the category heat map can be calculated by the above formulas (1) and (2).

[0108] Step A3, determining the category loss according to the first category loss, the second category loss and the heat map loss.

[0109] After the heat map loss is calculated, optionally, step A3 may include: performing a sum operation on the first category loss, the second category loss, and the heat map loss to determine the category loss.

[0110] The summation operation here can be a weighted summation or a direct summation, and of course other summation methods can also be used. In short, the sum of the first category loss, the second category loss and the heat map loss of each training medical image can be obtained, and the sum is used as the total category loss of the training medical image.

[0111] In this embodiment, by calculating the first category loss between the predicted sample category and the positive and negative sample labels of each training medical image, and calculating the second category loss between the image category label and the predicted image category, the total category loss is determined by these two category losses, so that the category loss can be refined and the calculated category loss is more accurate. In addition, by calculating the heat map loss between the positive sample heat map and the category heat map, and determining the total category loss by the heat map loss and the first two category losses, not only the image loss can be combined, but also the classification information loss can be combined, so that the final determined category loss is a combination of multiple losses, and thus more refined and accurate. Further, by summing the heat map loss and the first two category losses to obtain the total category loss, the total category loss can be obtained more quickly, the efficiency of obtaining the total category loss is improved, and the efficiency of training the network can be improved.

[0112] In another embodiment, another network training method is provided. Based on the above embodiment, the above position labels include position labels of some or all regions of interest in the training medical image. The calculation process of the position loss in the above S206 may include the following steps B:

[0113] Step B, calculating the loss between the predicted position of the region of interest in each training medical image and the position label of the region of interest in the corresponding training medical image to obtain the position loss.

[0114] In this step, when the position labels are annotated for each training medical image before the network is trained, some training medical images are annotated with position labels, while some training medical images are not annotated with position labels; in addition, among the training medical images annotated with position labels, some training medical images may be annotated with position labels of all the regions of interest included therein, or may be annotated with position labels of some of the regions of interest included therein, and by not annotating position labels for all training medical images in this way, the labeling cost is reduced. For example, the training medical images include three types of regions of interest, namely, 1, 2, and 3, and position labels may be annotated for all the three types of regions of interest, or may be annotated for only two or one of the types of regions of interest.

[0115] After obtaining the position labels, the network can be trained. During the network training process, the predicted positions of the regions of interest on each training medical image can be obtained. Since some training medical images do not have position labels or the marked position labels are incomplete, the position loss is calculated here. The position loss between the predicted position of each region of interest and the corresponding position label is calculated when each region of interest has both a predicted position and a position label.

[0116] For example, there is no position label on training medical image 1, and the network output of training medical image 1 has the predicted position of region of interest 1. Position labels of regions of interest 2 and 3 are marked on training medical image 2, and the network output of training medical image 2 has the predicted positions of regions of interest 2 and 3. When calculating the position loss here, what is calculated is the loss between the predicted position of region of interest 2 and the corresponding position label, and the loss between the predicted position of region of interest 3 and the corresponding position label.

[0117] In addition, the predicted position here can be obtained through the above-mentioned category heat map. Generally, the size of the category heat map may be smaller than the size of the original training medical image. Therefore, when calculating the position loss here, the category heat map can be interpolated to the size of the original training medical image by double interpolation, and then the loss between the predicted position and the position label is calculated. At the same time, the position loss here can use MSE mean square error loss or other types of losses, such as MSE loss. The calculation formula of the position loss can be as shown in the following formula (3):

[0118]

[0119] The parameters that are the same as those in formula (1) or (2) have the same meanings and are not described here. k , represents a binary image (or binarized image) of the same size as the original training medical image.

[0120] In this embodiment, the position loss is obtained by calculating the loss between the predicted position of the region of interest in each training medical image and the position label of the region of interest in the corresponding training medical image. In this way, only the position loss of the region of interest marked with the position label is calculated, which can make the calculated position loss more accurate and ensure the feasibility of training the network with only a small number of position labels.

[0121] In another embodiment, another network training method is provided. Based on the above embodiment, Figure 4 As shown, the above S208 may include the following steps:

[0122] S402, performing a sum operation on the category loss and the position loss, and using the sum as the value of the loss function.

[0123] In this step, the category loss here is the total category loss mentioned above, and the summation operation here can be a weighted summation or a direct summation, or of course other summation methods. In short, the sum of the total category loss and position loss of each training medical image can be obtained, and the sum is used as the value of the loss function. The loss function here can be set according to the actual situation, for example, it can be MSE mean square error loss, etc.

[0124] S404, training the initial network using the value of the loss function to determine a trained network.

[0125] In this step, after obtaining the value of the loss function, the parameters of the initial network can be adjusted using the value of the loss function, and this process is repeated. When the value of the loss function is less than the preset loss function threshold, or the value of the loss function remains unchanged, it can be determined that the initial network has been trained. At this time, the parameters of the initial network can be fixed to obtain a trained network.

[0126] In this embodiment, the initial network is trained by summing the category loss and position loss of the training medical images, and using the calculated sum as the value of the loss function to obtain a trained network. In this way, the value of the loss function can be obtained relatively quickly, thereby speeding up the adjustment of network parameters and further improving the efficiency of network training.

[0127] In another embodiment, another network training method is provided. Based on the above embodiment, the specific structures of the first network and the second network are described in the following embodiment.

[0128] The first network includes a first convolution module and a first output channel, and the first output channel is used to output a category heat map. The second network includes a second convolution module and a second output channel, a first fully connected layer and a third output channel, a second fully connected layer and a fourth output channel; the second output channel is used to output a positive sample heat map, the third output channel is used to output a predicted sample category of a training medical image, and the fourth output channel is used to output a predicted image category of a training medical image.

[0129] See here Figure 4a As shown, the first network is a network that outputs a category heat map, which includes a first convolution module (a convolution connected to the category heat map in the figure), an activation layer, and a first output channel. After the activation layer, the first output channel can output a category heat map. The category heat map includes the predicted position of the region of interest in the training medical image, and the position loss can be calculated with the position label of the region of interest, such as Loss2 in the figure.

[0130] The second network is a network that outputs positive sample heat maps, predicted sample categories, and predicted image categories. It includes a second convolution module (such as the convolution connected to the positive sample heat map in the figure), an activation layer, two fully connected layers, and three output channels. Among them, the second convolution module and the activation layer in the figure are followed by the second output channel, which can output positive sample heat maps and calculate heat map losses with category heat maps, i.e., Loss1. The third output channel is followed by the pooling and the first fully connected layer in the figure, which can output predicted sample categories (i.e., positive and negative sample information in the figure), and can calculate the first category loss with positive and negative sample labels, i.e., Loss3. The fourth output channel is followed by the pooling and the second fully connected layer in the figure (the bottom-most fully connected layer in the figure), which can output predicted image categories (i.e., category classification information in the figure), and can calculate the second category loss with image category labels, i.e., Loss4.

[0131] It should be noted that Figure 4a It is just an example and does not affect the essential content of the embodiments of the present application.

[0132] In this embodiment, by refining the structures of the first network and the second network and clarifying the information output by each output channel, the loss corresponding to each output channel can be quickly calculated during the network training process, thereby improving the efficiency of network training.

[0133] After the network is trained, the trained network can be used for image detection. The following embodiment will illustrate the testing process after the network is trained.

[0134] In one embodiment, an image detection method is provided. Based on the above embodiment, Figure 5 As shown, the method may include the following steps:

[0135] S502, obtaining a medical image to be tested; the medical image to be tested includes at least one region of interest.

[0136] In this step, the medical image to be tested may be a medical image of a certain body part of the subject to be tested, for example, it may be a medical image of the abdomen of the subject to be tested, and may be a two-dimensional image, a three-dimensional image, etc. The medical image to be tested may include one or more regions of interest. For example, taking an abdominal image as an example, the abdominal image may include multiple regions of interest such as intestinal obstruction, free gas, and stones. The number of each region of interest may be one or more. The medical image to be tested here and the medical images in the above-mentioned training medical image set may be images of the same body part.

[0137] S504, input the medical image to be tested into the trained network for detection processing, and determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

[0138] After the above network is trained, the medical image to be tested is input into the trained network, and then the position of each region of interest on the medical image to be tested and the category of each region of interest can be output through the network, so that the doctor can further analyze and process the image according to the position and category of each region of interest.

[0139] In this embodiment, the medical image to be tested is obtained and input into the trained network to obtain the position and category of each region of interest to be tested. Since the medical image to be tested is detected by the trained network, the obtained detection result must be more accurate. That is, the use of this method can improve the accuracy of the detection result obtained by detecting the medical image to be tested.

[0140] It should be understood that although Figure 2 , 3 The steps in the flowcharts of , 4, and 5 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 3 At least part of the steps in , 4, and 5 may include multiple steps or multiple stages. These steps or stages do not necessarily have to be performed at the same time, but can be performed at different times. The execution order of these steps or stages does not necessarily have to be sequentially, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0141] In one embodiment, Figure 6 As shown, a network training device is provided, the network includes a backbone network, a first network and a second network, the network training device may include: a sample image acquisition module 10, a prediction module 11, a loss calculation module 12 and a training module 13, wherein:

[0142] The sample image acquisition module 10 is used to acquire a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and a position label corresponding to some training medical images;

[0143] A prediction module 11 is used to input each training medical image into an initial network, determine the predicted position of the region of interest in each training medical image through the backbone network and the first network in the initial network, and determine the prediction category corresponding to each training medical image through the backbone network and the second network in the initial network;

[0144] A loss calculation module 12 is used to calculate the category loss between the predicted category corresponding to each training medical image and the category label corresponding to each training medical image, and to calculate the position loss between the predicted position of the region of interest in each training medical image and the position label corresponding to some training medical images;

[0145] The training module 13 is used to train the initial network according to the category loss and the position loss to determine the trained network.

[0146] For the specific limitations on the network training device, please refer to the limitations on the network training method above, which will not be repeated here.

[0147] In another embodiment, another network training device is provided. Based on the above embodiment, the above category label includes an image category label of a training medical image and / or a positive and negative sample label of a training medical image. The prediction category corresponding to the above training medical image includes a prediction sample category of the training medical image and a prediction image category of the training medical image. The above loss calculation module 12 may include:

[0148] A first category loss calculation unit, used to calculate the first category loss between the predicted sample category and the corresponding positive and negative sample labels corresponding to each training medical image;

[0149] A second category loss calculation unit, used to calculate the second category loss between the image category label of each training medical image and the corresponding predicted image category;

[0150] The category loss calculation unit is used to determine the category loss according to the first category loss and the second category loss.

[0151] Optionally, the above-mentioned category loss calculation unit may include:

[0152] A heat map acquisition subunit is used to obtain a positive sample heat map corresponding to each training medical image output by the second network and a category heat map corresponding to each training medical image output by the first network; the positive sample heat map includes the predicted sample category corresponding to the positive sample training medical image in each training medical image, and the category heat map includes the predicted category and predicted position of the region of interest in the training medical image;

[0153] The heat map loss calculation subunit is used to calculate the heat map loss between the positive sample heat map and the category heat map;

[0154] The category loss calculation subunit is used to determine the category loss according to the first category loss, the second category loss and the heat map loss.

[0155] Optionally, the above-mentioned category loss calculation subunit is specifically used to sum the first category loss, the second category loss and the heat map loss to determine the category loss.

[0156] In another embodiment, another network training device is provided. Based on the above embodiment, the above position labels include position labels of some or all regions of interest in the training medical image, and the above loss calculation module 12 may include:

[0157] The position loss calculation unit is used to calculate the loss between the predicted position of the region of interest in each training medical image and the position label of the region of interest in the corresponding training medical image to obtain the position loss.

[0158] In another embodiment, another network training device is provided. Based on the above embodiment, the training module 13 may include:

[0159] A calculation unit, used to sum the category loss and the position loss, and use the sum as the value of the loss function;

[0160] The training unit is used to train the initial network using the value of the loss function and determine the trained network.

[0161] In another embodiment, based on the above embodiment, the first network includes a first convolution module and a first output channel, and the first output channel is used to output a category heat map. The second network includes a second convolution module and a second output channel, a first fully connected layer and a third output channel, a second fully connected layer and a fourth output channel; the second output channel is used to output a positive sample heat map, the third output channel is used to output a predicted sample category of a training medical image, and the fourth output channel is used to output a predicted image category of a training medical image.

[0162] For the specific limitations on the network training device, please refer to the limitations on the network training method above, which will not be repeated here.

[0163] In one embodiment, Figure 7 As shown, an image detection device is provided, which may include: a test image acquisition module 20 and a detection module 21, wherein:

[0164] The test image acquisition module 20 is used to acquire a medical image to be tested; the medical image to be tested includes at least one region of interest;

[0165] The detection module 21 is used to input the medical image to be tested into the trained network for detection processing to determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the category and position of all regions of interest on the medical image to be tested.

[0166] For the specific definition of the image detection device, please refer to the definition of the image detection method above, which will not be repeated here.

[0167] Each module in the above network training device and image detection device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0168] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0169] A training medical image set is obtained; the training medical image set includes multiple training medical images, a category label corresponding to each training medical image, and a position label corresponding to some training medical images; each training medical image is input into an initial network, and a predicted position of a region of interest in each training medical image is determined through a backbone network and a first network in the initial network, and a predicted category corresponding to each training medical image is determined through a backbone network and a second network in the initial network; a category loss is calculated between a predicted category corresponding to each training medical image and a category label corresponding to each training medical image, and a position loss is calculated between a predicted position of a region of interest in each training medical image and a position label corresponding to some training medical images; the initial network is trained according to the category loss and the position loss to determine a trained network.

[0170] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0171] Calculate the first category loss between the predicted sample category and the corresponding positive and negative sample labels of each training medical image; calculate the second category loss between the image category label of each training medical image and the corresponding predicted image category; determine the category loss based on the first category loss and the second category loss.

[0172] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0173] A positive sample heat map corresponding to each training medical image output by the second network and a category heat map corresponding to each training medical image output by the first network are obtained; the positive sample heat map includes the predicted sample category corresponding to the positive sample training medical image in each training medical image, and the category heat map includes the predicted category and predicted position of the region of interest in the training medical image; the heat map loss between the positive sample heat map and the category heat map is calculated; and the category loss is determined according to the first category loss, the second category loss and the heat map loss.

[0174] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0175] The first category loss, second category loss, and heatmap loss are summed to determine the category loss.

[0176] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0177] The loss between the predicted position of the region of interest in each training medical image and the position label of the region of interest in the corresponding training medical image is calculated to obtain the position loss.

[0178] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0179] The category loss and the position loss are summed, and the sum is used as the value of the loss function; the initial network is trained using the value of the loss function to determine the trained network.

[0180] In one embodiment, the first network includes a first convolution module and a first output channel, and the first output channel is used to output a category heat map.

[0181] In one embodiment, the second network includes a second convolution module and a second output channel, a first fully connected layer and a third output channel, a second fully connected layer and a fourth output channel; the second output channel is used to output a positive sample heat map, the third output channel is used to output a predicted sample category of a training medical image, and the fourth output channel is used to output a predicted image category of a training medical image.

[0182] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0183] Acquire a medical image to be tested; the medical image to be tested includes at least one region of interest; input the medical image to be tested into a trained network for detection processing, and determine a detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

[0184] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0185] A training medical image set is obtained; the training medical image set includes multiple training medical images, a category label corresponding to each training medical image, and a position label corresponding to some training medical images; each training medical image is input into an initial network, and a predicted position of a region of interest in each training medical image is determined through a backbone network and a first network in the initial network, and a predicted category corresponding to each training medical image is determined through a backbone network and a second network in the initial network; a category loss is calculated between a predicted category corresponding to each training medical image and a category label corresponding to each training medical image, and a position loss is calculated between a predicted position of a region of interest in each training medical image and a position label corresponding to some training medical images; the initial network is trained according to the category loss and the position loss to determine a trained network.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0187] Calculate the first category loss between the predicted sample category and the corresponding positive and negative sample labels of each training medical image; calculate the second category loss between the image category label of each training medical image and the corresponding predicted image category; determine the category loss based on the first category loss and the second category loss.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] A positive sample heat map corresponding to each training medical image output by the second network and a category heat map corresponding to each training medical image output by the first network are obtained; the positive sample heat map includes the predicted sample category corresponding to the positive sample training medical image in each training medical image, and the category heat map includes the predicted category and predicted position of the region of interest in the training medical image; the heat map loss between the positive sample heat map and the category heat map is calculated; and the category loss is determined according to the first category loss, the second category loss and the heat map loss.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] The first category loss, second category loss, and heatmap loss are summed to determine the category loss.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] The loss between the predicted position of the region of interest in each training medical image and the position label of the region of interest in the corresponding training medical image is calculated to obtain the position loss.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0195] The category loss and the position loss are summed, and the sum is used as the value of the loss function; the initial network is trained using the value of the loss function to determine the trained network.

[0196] In one embodiment, the first network includes a first convolution module and a first output channel, and the first output channel is used to output a category heat map.

[0197] In one embodiment, the second network includes a second convolution module and a second output channel, a first fully connected layer and a third output channel, a second fully connected layer and a fourth output channel; the second output channel is used to output a positive sample heat map, the third output channel is used to output a predicted sample category of a training medical image, and the fourth output channel is used to output a predicted image category of a training medical image.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] Acquire a medical image to be tested; the medical image to be tested includes at least one region of interest; input the medical image to be tested into a trained network for detection processing, and determine a detection result corresponding to the medical image to be tested; the network is a network model trained according to the above-mentioned network training method, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

[0200] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0201] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0202] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A network training method, characterized in that: The network includes a backbone network, a first network, and a second network, and the method includes: Acquire a training medical image set; the training medical image set includes a plurality of training medical images, a category label corresponding to each training medical image, and position labels corresponding to some training medical images; the category label includes an image category label of the training medical image and positive and negative sample labels of the training medical image; Input each of the training medical images into the initial network, determine the category heat map corresponding to each of the training medical images through the backbone network and the first network in the initial network, and determine the positive sample heat map and predicted category corresponding to each of the training medical images through the backbone network and the second network in the initial network; the category heat map includes the predicted position of the region of interest in the training medical image, and the predicted category includes the predicted sample category and predicted image category of the training medical image; the positive sample heat map includes the predicted sample category corresponding to the positive sample training medical image in each of the training medical images; Calculating a first category loss between the predicted sample category and the corresponding positive and negative sample labels corresponding to each of the training medical images, calculating a second category loss between the image category label and the corresponding predicted image category of each of the training medical images, calculating a heat map loss between the positive sample heat map and the category heat map, determining a category loss based on the first category loss, the second category loss, and the heat map loss, and calculating a position loss between the predicted position of the region of interest in each of the training medical images and a position label corresponding to a portion of the training medical images; The initial network is trained according to the category loss and the position loss to determine a trained network.

2. The method according to claim 1, characterized in that: The determining the category loss according to the first category loss, the second category loss and the heat map loss comprises: The first category loss, the second category loss, and the heat map loss are summed to determine the category loss.

3. The method according to claim 1, characterized in that The position labels include position labels of some or all regions of interest in the training medical images, and the calculating of the position loss between the predicted positions of the regions of interest in the training medical images and the position labels corresponding to some of the training medical images includes: The loss between the predicted position of the region of interest in each of the training medical images and the position label of the region of interest in the corresponding training medical image is calculated to obtain the position loss.

4. The method according to any one of claims 1 to 3, characterized in that: The step of training the initial network according to the category loss and the position loss to determine a trained network includes: Performing a sum operation on the category loss and the position loss, and using the sum as the value of the loss function; The initial network is trained using the value of the loss function to determine a trained network.

5. The method according to claim 1, characterized in that The first network includes a first convolution module and a first output channel, and the first output channel is used to output the category heat map.

6. The method according to claim 1, characterized in that The second network includes a second convolutional module and a second output channel, a first fully connected layer and a third output channel, a second fully connected layer and a fourth output channel; The second output channel is used to output the positive sample heat map, the third output channel is used to output the predicted sample category of the training medical image, and the fourth output channel is used to output the predicted image category of the training medical image.

7. An image detection method, characterized in that: The method comprises: Acquire a medical image to be tested; the medical image to be tested includes at least one region of interest; The medical image to be tested is input into a trained network for detection processing to determine the detection result corresponding to the medical image to be tested; the network is a network model trained according to the method described in any one of claims 1 to 6, and the detection result includes the categories and positions of all regions of interest on the medical image to be tested.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images