Target detection network adversarial training method and electronic equipment

Abstract

The embodiment of the invention provides a target detection network adversarial training method and electronic equipment, and relates to the technical field of target detection. The method comprises the following steps: inputting a source domain image sample into a source domain detection network for supervised training to obtain a trained source domain detection network, and transferring a parameter matrix of the trained source domain detection network to a target domain detection network; and finally, performing adversarial training on the target domain detection network and a decoding discriminator by using the source domain image sample and the target domain image sample to obtain a trained target domain detection network. Therefore, the target domain detection network with high target detection accuracy can be trained only by labeling the source domain image samples.

Description

Technical Field

[0001] The present invention relates to the technical field of object detection, and in particular, to an object detection network adversarial training method and an electronic device. Background Art

[0002] Pancreatic ultrasound images are an important means for diagnosing pancreatic diseases in medical images. Due to the imaging characteristics of ultrasound imaging technology and the differences in the operation process, there are significant differences in the quality, resolution, and noise of pancreatic ultrasound images. Traditional methods for analyzing pancreatic ultrasound images mainly rely on manual analysis and judgment. However, the workload is large and misdiagnosis is prone to occur. Currently, deep learning object detection models can help doctors with detection to reduce the workload and the misdiagnosis rate.

[0003] However, the object detection model still faces difficulties in cross-domain generalization. Since there are significant differences in the image quality captured by different ultrasound devices, for example, when a model trained by supervised training with data from one model of device is directly used on ultrasound images from other models of devices, the detection accuracy will decrease. If supervised training is also performed on other models of devices, large-scale accurate data annotation by professional doctors is required, resulting in time consumption and an increase in human cost. Summary of the Invention

[0004] In view of this, the purpose of the embodiments of the present invention is to provide an object detection network adversarial training method and an electronic device to at least partially improve the above problems.

[0005] To achieve the above purpose, the technical solutions adopted in the embodiments of the present invention are as follows:

[0006] In a first aspect, an object detection network adversarial training method provided by an embodiment of the present invention includes:

[0007] Inputting source domain image samples into a source domain detection network for supervised training to obtain a trained source domain detection network; the source domain image samples include labeled tags;

[0008] Transferring the parameter matrix of the trained source domain detection network to a target domain detection network; the target domain detection network has the same structure as the source domain detection network;

[0009] Performing adversarial training on the target domain detection network and a decoding discriminator using the source domain image samples and target domain image samples to obtain a trained target domain detection network; the decoding discriminator is used to discriminate the probability that its input comes from the source domain.

[0010] Optionally, the transferring the parameter matrix of the trained source domain detection network to a target domain detection network includes:

[0011] Transfer the parameter matrix of the decoder of the trained source domain detection network to the decoder of the target domain detection network;

[0012] Assign the source domain parameter matrix of the source domain encoder of the trained source domain detection network to the transition encoder; the transition encoder has the same structure as the source domain encoder;

[0013] Perform adversarial training on the transition encoder and the encoding discriminator using the source domain image samples and the target domain image samples, and record the cumulative gradient of the parameter matrix of the transition encoder in each iteration batch; the encoding discriminator is used to discriminate the probability that its input comes from the source domain;

[0014] According to the cumulative gradient, select the parameters that meet the preset conditions from the source domain parameter matrix and transfer them to the target domain encoder of the target domain detection network.

[0015] Optionally, the step of selecting the parameters that meet the preset conditions from the source domain parameter matrix according to the cumulative gradient and transferring them to the target domain encoder of the target domain detection network includes:

[0016] Perform a dot product of the cumulative gradient and the source domain parameter matrix to obtain a dot product matrix;

[0017] Set the elements in the dot product matrix that are greater than the preset threshold to 1, and set the elements in the dot product matrix that are less than or equal to the preset threshold to 0 to obtain a selection matrix;

[0018] Perform a dot product of the selection matrix and the source domain parameter matrix to obtain a transfer parameter matrix;

[0019] Transfer the transfer parameter matrix to the target domain encoder of the target domain detection network according to the preset rule.

[0020] Optionally, the preset rule is:

[0021]

[0022] Wherein, are respectively the parameter matrix of the source domain encoder in the k-th iteration, the parameter matrix of the target domain encoder before adversarial training, is the parameter matrix of the target domain encoder after the (k - 1)-th round of adversarial training, α is the learning rate, γ is the parameter guidance smoothing coefficient, is the gradient of the parameter matrix before training in the (k - 1)-th iteration.

[0023] Optionally, the step of performing adversarial training on the target domain detection network and the decoding discriminator using the source domain image samples and the target domain image samples to obtain the trained target domain detection network includes:

[0024] Fix the parameter matrix of the target domain detection network, and input the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the decoding discriminator;

[0025] Fix the parameter matrix of the decoding discriminator, and input the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the target domain detection network;

[0026] Repeat and alternately input the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network, and train the decoding discriminator and the target domain detection network until a preset end condition is reached.

[0027] Optionally, the step of inputting the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the decoding discriminator includes:

[0028] Input the source domain image samples into the trained source domain detection network to obtain source domain detection results, and input the target domain image samples into the target domain detection network to obtain target domain detection results;

[0029] Input the source domain detection results and the target domain detection results into the decoding discriminator to obtain corresponding source domain discrimination results and target domain discrimination results;

[0030] Based on the decoding discriminator loss function, calculate the decoding discriminator loss according to the source domain discrimination results and the target domain discrimination results, and update the parameters of the decoding discriminator according to the decoding discriminator loss.

[0031] Optionally, both the source domain detection results and the target domain detection results include anatomical structures of multiple categories; the decoding discriminator loss function is:

[0032]

[0033] Where, is the decoding discriminator loss, X s and X t are the anatomical structures detected by the source domain detection network and the target domain detection network respectively. When the same category of X s and X t are detected from the source domain and the target domain at the same time, is 1, otherwise is 0, Dis de (X s )、Disde (X t ) is X s and X t is the probability from the source domain. z = 1 indicates that the input is the source domain image sample, and z = 0 indicates that the input is the target domain image sample.

[0034] Optionally, inputting the source domain image sample and the target domain image sample into the source domain detection network and the target domain detection network respectively to train the target domain detection network includes:

[0035] Input the source domain image sample into the trained source domain detection network to obtain the source domain detection result, and input the target domain image sample into the target domain detection network to obtain the target domain detection result;

[0036] Input the source domain detection result and the target domain detection result into the decoding discriminator to obtain the corresponding source domain discrimination result and target domain discrimination result;

[0037] Based on the target domain loss function, calculate the target domain loss according to the source domain discrimination result and the target domain discrimination result, and update the parameters of the target domain detection network according to the target domain loss.

[0038] Optionally, both the source domain detection result and the target domain detection result include detection results of multiple categories; the target domain loss function is:

[0039]

[0040] where is the target domain loss, X s , X t are the anatomical structures detected by the source domain detection network and the target domain detection network respectively. When the same category of X s and X t are detected from the source domain and the target domain at the same time, is 1, otherwise is 0, Dis de (X t ) is the probability that X t comes from the source domain.

[0041] In a second aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the program, the method described in any one of the above is implemented.

[0042] A target detection network adversarial training method and an electronic device provided by an embodiment of the present invention, after supervising and training the source domain detection network, then performing adversarial training on the target domain detection network and the decoding discriminator, so that only the source domain image samples need to be labeled to train a target domain detection network with high target detection accuracy.

[0043] Furthermore, in a parameter-selective guiding manner, the source domain network is used to teach the target domain network how to perform target detection of key pancreatic structures, and the source domain encoder transmits parameters to the target domain encoder to control the target domain encoder to update parameters in the correct direction.

[0044] To make the above objects, features, and advantages of the present application more obvious and understandable, the following specifically gives preferred embodiments and, in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0046] Figure 1 A schematic structural block diagram of the electronic device provided by the embodiment of the present invention;

[0047] Figure 2 A schematic flowchart of a target detection network adversarial training method provided by the embodiment of the present invention;

[0048] Figure 3 Another schematic flowchart of a target detection network adversarial training method provided by the embodiment of the present invention;

[0049] Figure 4 A schematic flowchart of step S224 provided by the embodiment of the present invention;

[0050] Figure 5 Another schematic flowchart of a target detection network adversarial training method provided by the embodiment of the present invention;

[0051] Figure 6 A data processing process diagram of a target detection network adversarial training method provided by the embodiment of the present invention.

[0052] Reference numerals: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - communication bus. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations.

[0054] Therefore, the detailed description of the embodiments of the present invention provided in the drawings below is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] It should be noted that: like reference numerals and letters denote like items in the following drawings, and thus, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of the present invention, terms such as "first", "second", etc. are only used for descriptive distinction and cannot be construed as indicating or implying relative importance.

[0056] It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0057] As described in the background art, currently, deep learning object detection models can assist doctors in detection to reduce the workload and the misdiagnosis rate. However, the object detection model still faces the difficulty of cross-domain generalization. A model trained by supervised training with data from one type of device often fails to detect accurately when detecting ultrasonic images of other types of devices. In related technologies, generally, ultrasonic images of each type of device are labeled and supervised training is carried out separately. However, this method requires professional doctors to perform large-scale accurate data labeling, which is extremely laborious, consumes a large amount of time, and increases a large amount of labor costs.

[0058] Based on the above situation, an embodiment of the present invention provides a method for adversarial training of a target detection network and an electronic device. After supervised training of the source domain detection network, adversarial training is then performed on the target domain detection network and the decoding discriminator, so that only the source domain image samples need to be labeled to train a target domain detection network with high target detection accuracy.

[0059] To implement the process steps and functions of various examples of the present invention, please refer to Figure 1 , Figure 1 FIG. is a schematic structural block diagram of an electronic device provided by an embodiment of the present invention. The electronic device 100 includes a memory 101 and a processor 103, and the memory 101 and the processor 103 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0060] The electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a distributed computer, and so on. It can be understood that the electronic device 100 is not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide the same functions as the server or virtual machine. The operating system of the electronic device 100 can be, but is not limited to, the Windows system, the Linux system, and so on.

[0061] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and so on.

[0062] The communication connection between the electronic device 100 and an external device is realized through at least one communication interface 102 (which can be wired or wireless).

[0063] The processor 103 may be an integrated circuit chip with the ability to process signals. In the implementation process, the steps of the embodiments of the present invention can be completed by the integrated logic circuit in the hardware of the processor 103 or the instructions in the form of software. The processor 103 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0064] It can be understood that Figure 1 The structure shown is only schematic, and the electronic device 100 may also include more or fewer components than those shown Figure 1 in the figure, or have a different configuration from that shown Figure 1 in the figure. Figure 1 Each component shown in the figure can be implemented by hardware, software, or a combination thereof.

[0065] Next, an exemplary description will be given of the target detection network adversarial training method provided by the present invention. Figure 2 It is a schematic flowchart of a target detection network adversarial training method provided by an embodiment of the present invention. Refer to Figure 2 , the execution subject of this method can be the above-mentioned Figure 1 shown electronic device 100, and this method includes the following steps as Figure 2 described:

[0066] S210: Input the source domain image samples into the source domain detection network for supervised training to obtain the trained source domain detection network; wherein, the source domain image samples include labeled tags.

[0067] S220: Transfer the parameter matrix of the trained source domain detection network to the target domain detection network; wherein, the target domain detection network has the same structure as the source domain detection network.

[0068] S230: Use the source domain image samples and the target domain image samples to perform adversarial training on the target domain detection network and the decoding discriminator to obtain the trained target domain detection network; wherein, the decoding discriminator is used to discriminate the probability that its input comes from the source domain.

[0069] In this embodiment, pancreatic ultrasound images will be used as the source domain image samples and target domain image samples for illustration. The source domain image samples and target domain image samples can come from different models of ultrasound devices. Take the ultrasound images of one model as the source domain image samples and label them, and take the ultrasound images of other models as the target domain image samples. It can be understood that during training, the ultrasound images of two models of ultrasound devices are included. One is used as the source domain image samples and is labeled, and the other is used as the target domain image samples and is not labeled.

[0070] In one round of training of the object detection network, first, use the labeled source domain image samples to perform supervised training on the source domain object detection network. After training the source domain object detection network, transfer the parameter matrix of the trained source domain object detection network to the target domain detection network. Since there are significant differences in the image quality captured by different ultrasound devices, this parameter matrix cannot be directly used and needs to be further optimized. A decoding discriminator can be constructed in the output spaces of the source domain detection network and the target domain detection network. This decoding discriminator is used to discriminate the probability that its input comes from the source domain. Then, use the decoding discriminator to perform adversarial training on the detection results of the target domain detection network for the output results of the source domain detection network and the target domain detection network, and obtain the trained target domain detection network.

[0071] This method first performs supervised training on the source domain detection network, and then performs adversarial training on the target domain detection network and the decoding discriminator. Thus, only by labeling the source domain image samples can a target domain detection network with a relatively high object detection accuracy be trained.

[0072] In step S210, for the training of the source domain detection network, a supervised learning method is adopted. The structure of this source domain detection network can be various, such as YOLO, R-CNN series, etc. In this embodiment, the structure of Faster-RCNN is adopted, including a backbone network for feature extraction, an RPN network for generating candidate boxes, and RCNN for classification and regression.

[0073] In step S220, there are various ways to transfer the parameter matrix of the source domain detection network to the target domain detection network. For example, directly transfer all the parameters, or transfer some parameters. Exemplarily, in this embodiment, only some parameters are transferred. Since not all the parameters in the source domain detection network are effective for the target domain, what the target domain detection network needs are the parameters that are beneficial to object detection, and only the parameters that are beneficial to object detection can be transferred. See Figure 3 The above step S220 may include the following steps:

[0074] S221: Transfer the parameter matrix of the decoder of the trained source domain detection network to the decoder of the target domain detection network.

[0075] S222: Assign the source domain parameter matrix of the source domain encoder of the trained source domain detection network to the transition encoder; wherein, the transition encoder has the same structure as the source domain encoder.

[0076] S223: Use the source domain image samples and the target domain image samples to perform adversarial training on the transition encoder and the encoding discriminator, and record the cumulative gradient of the parameter matrix of the transition encoder in each iteration batch; wherein, the encoding discriminator is used to discriminate the probability that its input comes from the source domain.

[0077] S224: According to the cumulative gradient, select the parameters that meet the preset conditions from the source domain parameter matrix and transfer them to the target domain encoder of the target domain detection network.

[0078] The structures of the source domain detection network and the target domain detection network both include an encoder and a decoder. Among them, the parameters of the decoder are shared by both, and directly transfer the parameter matrix of the decoder of the trained source domain detection network to the decoder of the target domain detection network.

[0079] For the parameter matrix of the source domain encoder, through a certain selection strategy, some parameters are screened and transferred to the target domain encoder. First, construct a transition encoder E c used to minimize the high-order feature space distance between the source domain and the target domain. The transition encoder has the same structure as the source domain encoder. Before the start of each iteration cycle, assign the source domain encoder parameter matrix θ s to the transition encoder E c , and at the same time input the source domain image samples and the target domain image samples, and then send the high-order feature maps of the two domains after encoding into the encoding discriminator for adversarial training. There are multiple iteration batches in each training round. For each transition encoder parameter matrix θ c updated in each iteration batch, record the gradient and accumulate the absolute value. The parameters with larger gradients are more likely to be selected because the larger the gradient, the greater the degree of change, indicating that these parameters are more variable and more sensitive to adaptation.

[0080] Among them, a single gradient is denoted as:

[0081]

[0082] The calculation formula for gradient accumulation is:

[0083]

[0084] Among them, is the target domain loss function of the encoder, and are the current gradient and the cumulative gradient of the transition encoder parameter θ c in the nth iteration batch respectively.

[0085] Then, according to the cumulative gradient, parameters that meet the preset conditions can be selected from the source domain parameter matrix and transferred to the target domain encoder of the target domain detection network. There can be various ways to select parameters. For example, in the cumulative gradient, the value at the position of the source domain parameter matrix corresponding to the larger element value can be selected as the transfer parameter. Because the cumulative gradient has a one-to-one correspondence with the elements at the same positions in the source domain encoder parameter matrix θ s , the parameter selection strategy in this embodiment can be: calculate the dot product matrix with θ s and sparsify it. Set the elements with larger values in this matrix to 1, indicating that the parameters at these positions are selected, and set the elements with smaller values to 0, indicating that the parameters at these positions are discarded. Subsequently, use this selection matrix to screen the source domain encoder parameter matrix. In a possible implementation, referring to Figure 4 , step S224 may include the following steps:

[0086] S2241: Perform a dot product of the cumulative gradient and the source domain parameter matrix to obtain a dot product matrix.

[0087] S2242: Set the elements greater than the preset threshold in the dot product matrix to 1, and set the elements less than or equal to the preset threshold in the dot product matrix to 0 to obtain a selection matrix.

[0088] S2243: Perform a dot product of the selection matrix and the source domain parameter matrix to obtain a transfer parameter matrix.

[0089] S2244: Transfer the transfer parameter matrix to the target domain encoder of the target domain detection network according to the preset rules.

[0090] This processing process can be represented by the following formula:

[0091]

[0092]

[0093] where M is the selection matrix, F is matrix sparsification, is the parameter finally selected from the source domain encoder for transfer to the target domain encoder, is the cumulative gradient, and θ s is the source domain parameter matrix.

[0094] Exemplarily, if the cumulative gradient is and the source domain parameter matrix is perform a dot product of the two to obtain a dot product matrix Suppose the preset threshold is 2.5. After comparison and selection, obtain the selection matrix Finally, from the source domain parameter matrix The selected transfer parameter matrix is Finally, transfer the transfer parameter matrix to the target domain encoder of the target domain detection network. All of these parameters can be transferred, or they can be transferred according to certain rules.

[0095] Since the source domain encoder can obtain supervised learning for object detection training, it can effectively update the parameters. The way the target domain encoder updates the parameters through backpropagation comes from the process of adversarial adaptation. The adversarial process is generally regarded as minimizing the Jensen-Shannon divergence (JS divergence) of two distributions. However, in high-dimensional feature distributions, the JS divergence is difficult to calculate. The target domain encoder that is completely disconnected from the source domain encoder is likely to lose the direction of adaptation during model training, resulting in the model not converging. Therefore, it is necessary to let the source domain encoder transfer parameters to the target domain encoder to control the target domain encoder to update the parameters in the correct direction. In one possible implementation, the parameter transfer in S2244 can be performed according to the following formula:

[0096]

[0097] where, are the parameter matrices of the source domain encoder at the k-th iteration, the parameter matrix of the target domain encoder before adversarial training, is the parameter matrix of the target domain encoder after the (k - 1)-th round of adversarial training, α is the learning rate, γ is the parameter guidance smoothing coefficient, is the gradient of the parameter matrix before training at the (k - 1)-th iteration.

[0098] It can be understood that if starting from the first round of training, after the source domain detection network is supervised trained, directly assign the parameter matrix of the source domain encoder to the target domain encoder, that is After performing adversarial training, the parameter matrix of the target domain encoder after the first round of adversarial training can be obtained After training. In the second round of training, after the source domain detection network is supervised trained, the parameter matrix of the source domain encoder is obtained The parameter transferred to the target domain encoder this time is and And so on, the parameter matrix transferred to the target domain encoder in each round of training.

[0099] In step S223, the way of performing adversarial training can be:

[0100] First, fix the parameter matrix of the over - encoder. Input the source - domain image samples and the target - domain image samples into the over - encoder to obtain the high - order feature maps of the two domains. Input each high - order feature map into the encoding discriminator to get the discrimination results. Calculate the loss of the encoding discriminator according to the discrimination results, and perform backpropagation update on the encoding discriminator. Among them, the calculation formula for the loss of the encoding discriminator is:

[0101]

[0102] Among them, is the loss of the decoding discriminator, \(X\) s and \(X\) t are the high - order feature maps encoded from the source - domain image samples and the target - domain image samples respectively. \(z = 1\) indicates that the input is the source - domain image sample. \(Dis\) en \((X\) s ) and \(Dis\) en \((X\) t ) are the probabilities that \(X\) s and \(X\) t come from the source domain.

[0103] It can be understood that for this calculation formula of the loss of the encoding discriminator, if the image sample is from the source domain, it is calculated using \(\log(Dis\) en \((X\) s ). If the image sample is from the target domain, it is calculated using \(1-\log(Dis\) en \((X\) t ). Finally, the losses of all image samples are added up to obtain the loss of the encoding discriminator.

[0104] Then, fix the parameter matrix of the encoding discriminator. Similarly, input the source - domain image samples and the target - domain image samples into the over - encoder to obtain the high - order feature maps of the two domains. Input each high - order feature map into the encoding discriminator to get the discrimination results. Calculate the loss of the over - encoder according to the discrimination results, and perform backpropagation update on the over - encoder. Among them, the calculation formula for the loss of the over - encoder is:

[0105]

[0106] Among them, is the loss of the over - encoder, \(X\) t is the high - order feature map encoded from the target - domain image sample. \(Dis\) en \((X\) t ) is the probability that \(X\) t comes from the source domain.

[0107] There are various implementation methods for adversarial training of the target - domain detection network and the decoding discriminator. In one possible implementation method, referring to Figure 5 , step S230 may include the following steps:

[0108] S231: Fix the parameter matrix of the target domain detection network, and input the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the decoding discriminator.

[0109] S232: Fix the parameter matrix of the decoding discriminator, and input the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the target domain detection network.

[0110] S233: Repeatedly and alternately input the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network to train the decoding discriminator and the target domain detection network until a preset end condition is reached.

[0111] First, fix the parameter matrix of the target domain detection network, and input the source domain and target domain image samples into the source domain detection network and the target domain detection network respectively to preliminarily train the decoding discriminator. Then, fix the parameter matrix of the decoding discriminator, and again use the source domain and target domain image samples to input into the source domain detection network and the target domain detection network respectively to train the target domain detection network. By repeating the above two operations, the performance of both is continuously optimized. The training process continues until a preset end condition (such as model convergence, the loss function reaches a threshold, or a predetermined number of iterations is completed) is reached.

[0112] Step S231 may include: Input the source domain image samples into the trained source domain detection network to obtain the source domain detection results, input the target domain image samples into the target domain detection network to obtain the target domain detection results; input the source domain detection results and the target domain detection results into the decoding discriminator to obtain the corresponding source domain discrimination results and target domain discrimination results; based on the decoding discriminator loss function, calculate the decoding discriminator loss according to the source domain discrimination results and the target domain discrimination results, and update the parameters of the decoding discriminator according to the decoding discriminator loss.

[0113] Input the source domain image samples and the target domain image samples into the corresponding source domain detection network and target domain detection network to obtain the source domain detection results and the target domain detection results, and then input the detection results into the decoding discriminator to obtain the source domain discrimination results and the target domain discrimination results. The decoding discriminator can be a binary classification network for distinguishing the detection results of the source domain and the target domain. According to the source domain discrimination results and the target domain discrimination results, the decoding discriminator loss can be calculated, and then the parameters of the decoding discriminator can be updated.

[0114] In image object detection, multiple categories of objects may be detected. For example, when performing object detection on pancreatic ultrasound images, multiple anatomical structures are often detected, such as the head of the pancreas, the body of the pancreas, and the tail of the pancreas. Since in the object detection process of the source domain and the target domain, due to the different images of the source domain and the target domain input simultaneously, not all the same anatomical structures may be present at the same time. Therefore, during adversarial optimization, only the anatomical structures detected simultaneously by the target domain detection network and the source domain detection network in the same batch are paired and sent into the discriminator for true / false discrimination. Therefore, optionally, the source domain detection results and the target domain detection results may both include multiple categories of anatomical structures; the above decoding discriminator loss function may be:

[0115]

[0116] wherein, is the decoding discriminator loss, X s and X t are the anatomical structures detected by the source domain detection network and the target domain detection network respectively. When the same category of X s and X t are detected simultaneously from the source domain and the target domain, is 1, otherwise is 0, Dis de (X s ) and Dis de (X t ) are the probabilities that X s and X t come from the source domain. z = 1 indicates that the input is a source domain image sample, and z = 0 indicates that the input is a target domain image sample.

[0117] Exemplarily, for a source domain image sample and a target domain image sample, through object detection, the source domain head of the pancreas, the source domain body of the pancreas, the target domain body of the pancreas, and the target domain tail of the pancreas are obtained respectively. Only the source domain body of the pancreas and the target domain body of the pancreas are input into the decoding discriminator for probability discrimination. If the anatomical structure is from the source domain, it is calculated using log(Dis en (X s )); if the anatomical structure is from the target domain, it is calculated using 1 - log(Dis en (X t ). Finally, the losses of all anatomical structures are added up to obtain the decoding discriminator loss.

[0118] Step S232 may include: inputting source domain image samples into the trained source domain detection network to obtain source domain detection results, inputting target domain image samples into the target domain detection network to obtain target domain detection results; inputting the source domain detection results and the target domain detection results into the decoding discriminator to obtain corresponding source domain discrimination results and target domain discrimination results; based on the target domain loss function, calculating the target domain loss according to the source domain discrimination results and the target domain discrimination results, and updating the parameters of the target domain detection network according to the target domain loss.

[0119] For the training of the target domain detection network, it is the same as the processing process of the above decoding discriminator, except that the loss calculation is different. The target domain loss can be directly calculated using the results detected by the target domain inspection network. The target domain loss function is:

[0120]

[0121] Wherein, is the target domain loss, X s , X t are the anatomical structures detected by the source domain detection network and the target domain detection network respectively. When the same category of X s and X t are detected from the source domain and the target domain at the same time, is 1, otherwise is 0, Dis de (X t ) is the probability that X t comes from the source domain.

[0122] Similar to the above decoding discriminator loss function, only the key structures detected by the target domain detection network and the source domain detection network at the same time are input into the decoding discriminator for probability discrimination, and then the target domain loss is calculated.

[0123] Exemplarily, in a possible implementation manner, referring to Figure 6 , the target domain detection network and the source domain detection network are of the Faster-RCNN structure, including a backbone network for feature extraction, an RPN network for generating candidate boxes, and an RCNN for classification and regression. That is, the backbone network is the encoder, and RPN+RCNN is the decoder. The following gives an exemplary description of one round of training.

[0124] First, input the source domain image samples into the source domain detection network for supervised training. Then, share the RPN+RCNN parameters of the source domain detection network with the RPN+RCNN of the target domain detection network, and transfer the backbone network of the source domain detection network to the backbone network of the target domain detection network through selective guidance. Next, perform adversarial training on the target domain detection network and the decoding discriminator. After the source domain image samples and the target domain image samples are respectively input into the source domain detection network and the target domain detection network, multiple categories of anatomical structures are obtained. Input the anatomical structures simultaneously detected by the source domain detection network and the target domain detection network into the decoding discriminator for source discrimination. Finally, calculate the loss according to the discrimination result, and perform backpropagation update on the target domain detection network and the decoding discriminator.

[0125] Furthermore, the source domain detection network and the target domain detection network trained by the above target detection network adversarial training method can be used to perform target detection on the images of their own domains, and thus target detection results can be obtained.

[0126] In summary, for a target detection network adversarial training method and an electronic device provided by an embodiment of the present invention, after the source domain detection network is supervised and trained, the target domain detection network and the decoding discriminator are then adversarially trained. As a result, only the source domain image samples need to be labeled, and a target domain detection network with high target detection accuracy can be trained. The source domain network teaches the target domain network how to perform target detection of key pancreatic structures through the method of parameter selective guidance, and the source domain encoder transfers parameters to the target domain encoder to control the target domain encoder to update parameters in the correct direction.

[0127] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0128] In addition, each functional module in various embodiments of the present invention may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part.

[0129] If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0130] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0131] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above-described exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to embrace all changes within the meaning and scope of the equivalent elements of the claims in the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights.

Claims

1. A method for adversarial training of a target detection network, characterized in that, The method includes: Inputting source domain image samples into a source domain detection network for supervised training to obtain a trained source domain detection network; the source domain image samples include labeled tags; Transferring the parameter matrix of the trained source domain detection network to a target domain detection network; the target domain detection network has the same structure as the source domain detection network; Using the source domain image samples and target domain image samples to conduct adversarial training on the target domain detection network and a decoding discriminator to obtain a trained target domain detection network; the decoding discriminator is used to discriminate the probability that its input comes from the source domain.

2. The method according to claim 1, wherein The transferring the parameter matrix of the trained source domain detection network to the target domain detection network includes: Transferring the parameter matrix of the decoder of the trained source domain detection network to the decoder of the target domain detection network; Assigning the source domain parameter matrix of the source domain encoder of the trained source domain detection network to a transition encoder; the transition encoder has the same structure as the source domain encoder; Using the source domain image samples and the target domain image samples to conduct adversarial training on the transition encoder and an encoding discriminator, and recording the cumulative gradient of the parameter matrix of the transition encoder in each iteration batch; the encoding discriminator is used to discriminate the probability that its input comes from the source domain; According to the cumulative gradient, selecting parameters that meet the preset conditions from the source domain parameter matrix and transferring them to the target domain encoder of the target domain detection network.

3. The method according to claim 2, wherein The according to the cumulative gradient, selecting parameters that meet the preset conditions from the source domain parameter matrix and transferring them to the target domain encoder of the target domain detection network includes: Performing a dot product of the cumulative gradient and the source domain parameter matrix to obtain a dot product matrix; Setting elements in the dot product matrix that are greater than a preset threshold to 1, and setting elements in the dot product matrix that are less than or equal to the preset threshold to 0 to obtain a selection matrix; Performing a dot product of the selection matrix and the source domain parameter matrix to obtain a transfer parameter matrix; Transferring the transfer parameter matrix to the target domain encoder of the target domain detection network according to a preset rule.

4. The method according to claim 3, wherein The preset rule is: Among them, are respectively the parameter matrix of the source domain encoder at the k-th iteration, the parameter matrix of the target domain encoder before adversarial training, is the parameter matrix of the target domain encoder after the (k - 1)-th round of adversarial training, α is the learning rate, γ is the parameter guidance smoothing coefficient, is the gradient of the parameter matrix before training at the (k - 1)-th iteration.

5. The method according to claim 1, wherein The using the source domain image samples and target domain image samples to conduct adversarial training on the target domain detection network and a decoding discriminator to obtain a trained target domain detection network includes: Fixing the parameter matrix of the target domain detection network, and inputting the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the decoding discriminator; Fixing the parameter matrix of the decoding discriminator, and inputting the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the target domain detection network; Repeatedly and alternately inputting the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network to train the decoding discriminator and the target domain detection network until a preset end condition is reached.

6. The method according to claim 5, characterized in that, Inputting the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the decoding discriminator includes: Inputting the source domain image samples into the trained source domain detection network to obtain source domain detection results, and inputting the target domain image samples into the target domain detection network to obtain target domain detection results; Inputting the source domain detection results and the target domain detection results into the decoding discriminator to obtain corresponding source domain discrimination results and target domain discrimination results; Based on the decoding discriminator loss function, calculating the decoding discriminator loss according to the source domain discrimination results and the target domain discrimination results, and updating the parameters of the decoding discriminator according to the decoding discriminator loss.

7. The method according to claim 6, wherein Both the source domain detection results and the target domain detection results include anatomical structures of multiple categories; the decoding discriminator loss function is: Among them, is the decoding discriminator loss, and X s , X t are the anatomical structures detected by the source domain detection network and the target domain detection network respectively. When X s and X t of the same category are detected from both the source domain and the target domain at the same time, is 1, otherwise is 0. Dis de (X s ), Dis de (X t ) are the probabilities that X s and X t come from the source domain. z being 1 indicates that the input is the source domain image sample, and z being 0 indicates that the input is the target domain image sample.

8. The method according to claim 5, characterized in that, Inputting the source domain image samples and the target domain image samples into the source domain detection network and the target domain detection network respectively to train the target domain detection network includes: Inputting the source domain image samples into the trained source domain detection network to obtain source domain detection results, and inputting the target domain image samples into the target domain detection network to obtain target domain detection results; Inputting the source domain detection results and the target domain detection results into the decoding discriminator to obtain corresponding source domain discrimination results and target domain discrimination results; Based on the target domain loss function, calculating the target domain loss according to the source domain discrimination results and the target domain discrimination results, and updating the parameters of the target domain detection network according to the target domain loss.

9. The method according to claim 8, wherein Both the source domain detection results and the target domain detection results include detection results of multiple categories; the target domain loss function is: Among them, is the target domain loss, X s , X t are the anatomical structures detected by the source domain detection network and the target domain detection network respectively. When X s and X t are detected from the source domain and the target domain at the same time, is 1, otherwise is 0, Dis de (X t ) is the probability that X t comes from the source domain.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 9.

Images