Model training method and device

Abstract

The present invention provides a model training method and device, which includes: obtaining multiple first images; extracting a first object from the first image; and obtaining a second object instance annotated from the first image as a second object instance label; obtaining a contour of the first object; obtaining a contour of the second object based on the contour of the first object and the second object instance as the second object contour label; inputting the first object into a model to be trained and obtaining an output of the model to be trained; the model to be trained includes a contour detection branch and an instance detection branch, and the output includes a second object contour output and a second object instance output; calculating a first loss function value and a second loss function value; calculating a third loss function value based on the first loss function value and the second loss function value; and adjusting the parameters of the model to be trained based on the third loss function value. The model trained using the model training method provided by the present invention has high detection efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent detection, and in particular to a model training method and device. Background Art

[0002] At present, most target object detection methods based on deep learning are limited to directly using complete image blocks as input to the detection model through preliminary preprocessing. The detection accuracy is low. Moreover, due to the limited detection area, the time for a single detection is long, that is, the detection efficiency is low. Summary of the Invention

[0003] Therefore, the present invention aims to solve the technical problem in the prior art of low detection accuracy and low detection efficiency when using existing detection models to detect target objects, and thus provides a model training method and device.

[0004] According to a first aspect, an embodiment of the present invention provides a model training method, comprising the following steps:

[0005] acquiring a plurality of first images;

[0006] For each of the first images, extract a first object in the first image; and obtain a second object instance annotated in the first image as a second object instance label;

[0007] obtaining a contour of the first object;

[0008] According to the outline of the first object and the second object instance, obtaining the outline of the second object as a second object outline label;

[0009] Inputting the first object into a to-be-trained model and obtaining an output of the to-be-trained model; the to-be-trained model includes a contour detection branch and an instance detection branch, and the output includes a second object contour output and a second object instance output; the instance detection branch detects a second object contour based on the first object and contour features extracted by the contour detection branch;

[0010] Calculating a first loss function value based on the second object instance label and the second object instance output, and calculating a second loss function value based on the second object contour label and the second object contour output;

[0011] Calculate a third loss function value based on the first loss function value and the second loss function value;

[0012] Based on the third loss function value, adjust the parameters of the model to be trained.

[0013] Optionally, inputting the first object into the model to be trained includes:

[0014] dividing the second object obtained by labeling the first object according to the position of the second object in the first object;

[0015] Counting the number of the second objects at each position;

[0016] Based on the number, performing position-balanced amplification on the first object by using at least one of flipping along a cross section, adding discrete Gaussian noise, and performing histogram equalization;

[0017] The first object and the expanded first object are input into the model to be trained.

[0018] Optionally, the model to be trained includes an encoding block, a feature extraction block and a decoding block, and the contour detection branch and the instance detection branch both include the feature extraction block and the decoding block;

[0019] The encoding block includes M groups of downsampling structures connected in sequence, and the M groups of downsampling structures are used to obtain downsampling results of different scales respectively. The decoding block includes M groups of upsampling structures corresponding to the downsampling structures one by one. The sampling results of each group of the downsampling structures are spliced with the features output by the previous level structure of the corresponding upsampling structure to serve as the input features of the upsampling structure;

[0020] The feature extraction block in the contour detection branch is used to extract deep features based on the output of the encoding block, and the feature extraction block in the instance detection branch is used to extract deep features based on the output of the encoding block and the upsampling result of the intermediate layer upsampling structure of the decoding block in the contour detection branch. The decoding block also includes a classification layer, which is used to perform classification detection based on the output of M groups of the upsampling structures.

[0021] Optionally, each group of the downsampling structure in the encoding block includes a convolution block and a BiA module connected in sequence, the convolution block is used for downsampling, and the BiA module includes two residual branches in parallel, and the two residual branches are used to decouple the features output by the convolution block to obtain two feature maps, and the two feature maps are respectively input into the decoding block in the contour detection branch and the decoding block in the instance detection branch.

[0022] Optionally, each residual branch of the BiA module includes two residual sub-modules connected in sequence; the BiA module also includes a spatial attention mechanism block, which includes a maximum pooling layer and an average pooling layer connected in sequence, and the input of the spatial attention mechanism block is the output of the convolution block of the same group of downsampling structures. The output of the spatial attention mechanism block obtains a weight map through a Sigmoid function, and is respectively combined with the output of the residual sub-module after the two residual branches.

[0023] Optionally, the feature extraction block includes multiple downsampling layers and multiple upsampling layers connected in sequence; each downsampling layer and upsampling layer is also preceded by a Swin-Transformer layer, and the last downsampling layer and the first upsampling layer are spliced using a convolutional layer; the output of the last downsampling layer and the output of the previous downsampling layer are spliced by short-circuiting.

[0024] Optionally, the upsampling result of the intermediate layer upsampling structure of the decoding block in the contour detection branch is downsampled to the output scale of the encoding block, and a weight map is obtained by a Sigmoid function, which is added to the output of the encoding block and then multiplied as the input of the feature extraction block of the instance detection branch.

[0025] Optionally, the first upsampling results of the decoding block at different scales are adjusted to have a number of channels through convolution, and then added to the upsampling result elements at a larger scale as deep supervision of the model to be trained.

[0026] Optionally, the first loss function value is calculated using the following formula:

[0027]

[0028] Where a is the ath connected domain, K is the number of connected domains, p(x b ) is the true value of the input contour detection branch, b is the category of the foreground 1 and background 0 of the true value, q(x ab ) is the predicted value of the contour detection branch, L CE is the cross entropy loss function, L CDE is the loss value of the contour detection branch.

[0029] Optionally, the second loss function value is calculated using the following formula:

[0030]

[0031] Where X is the matrix of the prediction results in the instance detection branch, Y is the true value input into the instance detection branch, and L DiceThe loss value of the detection branch for the instance.

[0032] According to a second aspect, an embodiment of the present invention provides a model training device, comprising: a first acquisition module, configured to acquire a plurality of first images;

[0033] a processing module, configured to extract, for each of the first images, a first object in the first image; and obtain a second object instance annotated in the first image as a second object instance label;

[0034] A second acquisition module, configured to acquire the outline of the first object;

[0035] a labeling module, configured to obtain, according to the outline of the first object and the second object instance, the outline of the second object as a second object outline label;

[0036] a detection module, configured to input the first object into a to-be-trained model and obtain an output of the to-be-trained model; the to-be-trained model includes a contour detection branch and an instance detection branch, the output including a second object contour output and a second object instance output; the instance detection branch detects a second object contour based on the first object and contour features extracted by the contour detection branch;

[0037] a first calculation module, configured to calculate a first loss function value based on the second object instance label and the second object instance output, and to calculate a second loss function value based on the second object contour label and the second object contour output;

[0038] A second calculation module is used to calculate a third loss function value based on the first loss function value and the second loss function value;

[0039] An adjustment module is used to adjust the parameters of the model to be trained based on the third loss function value.

[0040] According to the third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the above-mentioned model training method by executing the computer instructions.

[0041] According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the above-mentioned model training method.

[0042] The technical solution of the present invention has the following advantages:

[0043] The dual-supervised model trained using the model training method provided in the embodiments of the present invention takes into account that the target object (i.e., the second object) in the image to be detected is closely related to the first object. For example, an aneurysm is an abnormal bulge in the arterial wall. Therefore, before using the dual-supervised model to detect the second object in the image, the first object must be extracted from the image to be detected. The first object is then input into the dual-supervised model to output the detection result. Preprocessing the first object eliminates the need for time-consuming operations such as N4 correction, resulting in higher detection efficiency. Furthermore, during detection, the first image does not need to be segmented into small blocks, resulting in higher sensitivity and avoiding operations such as image reconstruction. Furthermore, the dual-supervised model detects the first object, which is closely related to the second object, rather than the entire image content. This improves both efficiency and accuracy. The two branches of the dual-supervised model respectively extract the contour features and overall features of the second object, detect possible second objects based on the contour features, and then further determine whether the possible second object is indeed the second object based on the overall features, thereby reducing the false detection rate and improving detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 This is a flowchart of a specific example of a model training method in Example 1 of the present application;

[0046] Figure 2 This is a principle block diagram of a specific example of preprocessing in Example 1 of the present application;

[0047] Figure 3 This is a principle block diagram of a specific example of the model to be trained in Example 1 of the present application;

[0048] Figure 4 This is a principle block diagram of a specific example of a model training device in Example 2 of the present application;

[0049] Figure 5 This is a structural diagram of a specific example of a computer device in Example 3 of the present application. DETAILED DESCRIPTION

[0050] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0052] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; internal connections between two components; wireless connections or wired connections. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0053] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0054] Example 1

[0055] This embodiment provides a model training method, such as Figure 1 As shown, the following steps are included:

[0056] Step S101, obtaining multiple first images. The first image may be an image obtained by time-of-flight angiography (TOF-MRA for short). TOF-MRA is mainly used for imaging of blood vessels in the head. Of course, the first image in this embodiment may also be a CT image, a DSA image, etc., and the first object may be a tissue structure included in the image. In this embodiment, TOF-MRA images are mainly used as an example. In this embodiment, the multiple first images include TOF-MRA images of aneurysms. Among them, the first image should avoid obtaining images with poor imaging quality and severe artifacts.

[0057] Step S102 : for each first image, extract the first object in the first image; and obtain the second object instance annotated in the first image as the second object instance label.

[0058] Before extracting the first object from the first image, the first image needs to be preprocessed, and the preprocessing includes data normalization. The voxel intensities of the multiple acquired first images are normalized to the range of 0-1024, and the origins and voxel spatial distances (i.e., voxel spacing) of all first images are unified, which can be set to (0,0,0) and (1,1,1), respectively. Here, regarding the second object instance, it can be annotated based on the first image before preprocessing. At this time, it is necessary to refer to the normalization of the first image and normalize the origin and voxel spacing of the annotated second object instance. For the first image and the second object instance obtained based on the annotation of the first image, the origin and voxel spacing are normalized to standardize the differences in voxel ranges in different images caused by different acquisition parameters.

[0059] When the first image is a TOF-MRA image, correspondingly, the first object may be a blood vessel structure, and the second object may be a blood vessel aneurysm, such as an aneurysm.

[0060] Furthermore, TOF-MRA images can be segmented by setting a threshold for voxel intensity, extracting the complete vascular structure from the TOF-MRA image and using the vascular structure as the first object. The vascular structure can then be expanded based on the aneurysm size or diameter. For smaller aneurysms, a larger expansion radius can be used, while for larger aneurysms, a smaller expansion radius can be used, ensuring that the extraction result includes the vascular structure and some surrounding tissue.

[0061] Before obtaining the second object instance, the second object in the first image must be labeled. In this embodiment, taking an aneurysm as the second object, the aneurysm in the first image is labeled to obtain the second object instance. In this embodiment, the obtained second object instance is used as the second instance label.

[0062] Step S103, obtaining the outline of the first object. As mentioned above, the first object may be a blood vessel structure. Further, the first image is preprocessed, such as Figure 2 The extracted blood vessel structure is segmented using the Canny operator to obtain the blood vessel contour. In this embodiment, the contour of the first object may be the contour of the blood vessel.

[0063] Step S104 : Acquire the contour of the second object as a second object contour label according to the contour of the first object and the second object instance.

[0064] According to the size of the aneurysm, the second object instance marked is expanded. The smaller the aneurysm, the larger the expansion radius. The larger the aneurysm, the smaller the expansion radius. The expanded second object instance is multiplied with the contour of the first object to obtain the contour of the second object, such as Figure 2 shown.

[0065] As described above, the first object's outline can be a blood vessel outline, and the second object instance can be an aneurysm annotation in the first image. In this embodiment, the second object's outline is obtained based on the first object's outline and the second object instance. Specifically, the aneurysm's outline can be obtained based on the blood vessel outline and the aneurysm annotation. In this embodiment, the second object is located on the first object. Furthermore, the second object's outline is used as the second image's outline label.

[0066] Step S105: input the first object into the model to be trained to obtain the output of the model to be trained; the model to be trained includes a contour detection branch and an instance detection branch, and the output includes a second object contour output and a second object instance output; the instance detection branch detects the second object contour based on the first object and the contour features extracted by the contour detection branch.

[0067] In this embodiment, the model to be trained may adopt a deep convolutional encoder-decoder network structure (CEDNet). The CEDNet structure is used to capture the contour and texture feature information of the second object and perform the detection task of the second object.

[0068] In this embodiment, the model to be trained includes two supervision branches, namely a contour detection branch and an instance detection branch. The second image contour label is used as a label of the contour detection branch, and the second object instance label is used as a label of the instance detection branch.

[0069] In addition to the first object, normal vascular structures obtained based on other images may also be input as training samples to the model to be trained, so as to avoid the problem of high false detection rate after the model is trained.

[0070] The contour detection branch extracts the contour features of the second object (i.e., the contour features of the blood vessels in the aneurysm region in this embodiment) to output a second object contour output. The instance detection branch further determines the extracted contour features of the second object to output a second object instance output. The second object instance output of the instance detection branch is based on the first object and the contour features of the second object extracted by the contour detection branch, thereby further detecting the second object and outputting the detection result as a second object instance.

[0071] Step S106 : Calculate a first loss function value based on the second object instance label and the second object instance output, and calculate a second loss function value based on the second object contour label and the second object contour output.

[0072] In this embodiment, the first loss function L CDE It is an improvement based on the cross entropy loss function, which is used to perform the cross entropy loss function at the connected domain level of the prediction results of the model to be trained.

[0073] When the first loss function is used, the first step is to set the voxels with probabilities higher than the threshold in the probability map predicted by the training model to be trained to 1 to obtain the predicted result; the second step is to calculate the number of connected domains in the predicted result of the first step. When the number of connected domains is 0 or 1, the first loss function is directly equal to the cross-loss entropy function. When the number of connected domains is greater than 1, the predicted result and the true result are element-wise summed; the third step is to take out each connected domain in the element-wise sum result and perform cross entropy with the true value; finally, the average of the cross entropy results of the series of connected domains obtained in the third step is taken to obtain the value of the first loss function.

[0074] Step S107: Calculate a third loss function value based on the first loss function value and the second loss function value.

[0075] In this embodiment, the third loss function can be the first loss function L CDE With the second loss function L Dice The weight sum is: Where α is the first loss function L CDE The calculation weight of β is the second loss function L Dice The calculation weight of , L is the third loss function.

[0076] Step S108: Adjust the parameters of the model to be trained based on the third loss function value.

[0077] If the third loss function value meets the preset threshold requirement, the training is terminated; otherwise, the training is continued in step S107 after adjusting the parameters.

[0078] In this embodiment, the first object is extracted from the acquired multiple first images, and the second object instance marked in the first image is obtained; the contour of the second object is obtained based on the first object contour and the second object instance. The contour of the second object is used as the second object contour label, and the second object instance is used as the second object instance label. Furthermore, the first object is input into the model to be trained, and the contour detection branch and the instance detection branch included in the model to be trained are used to output the second object contour output and the second object instance output. A first loss function value is calculated based on the second object instance label and the second object instance output, and a second loss function value is calculated based on the second object contour label and the second object contour output; and a third loss function value is calculated based on the first loss function value and the second loss function value; thereby, the parameters of the model to be trained are adjusted based on the third loss function value.

[0079] The dual-supervised model trained using the model training method provided in this embodiment takes into account that the target object (i.e., the second object) in the image to be detected is closely related to the first object. For example, an aneurysm is an abnormal bulge in the arterial wall. Therefore, before using the dual-supervised model to detect the second object in the image, the first object must be extracted from the image to be detected. The first object is then input into the dual-supervised model to output the detection result. Preprocessing the first object eliminates the need for time-consuming operations such as N4 correction, resulting in higher detection efficiency. Furthermore, during detection, the first image does not need to be segmented into small blocks, resulting in higher sensitivity and avoiding operations such as image reconstruction. Furthermore, the dual-supervised model detects the first object, which is closely related to the second object, rather than the entire image content. This improves both efficiency and accuracy. The two branches of the dual-supervised model extract the contour features and overall features of the second object, respectively. Based on the contour features, the model detects the possible second object. The overall features are then used to further determine whether the possible second object is indeed the second object, thereby reducing the false detection rate and improving detection accuracy.

[0080] As an optional implementation manner, in an embodiment of the present invention, inputting the first object into the model to be trained includes:

[0081] dividing the second object obtained by labeling the first object according to the position of the second object in the first object;

[0082] Counting the number of the second objects at each position;

[0083] Based on the number, performing position-balanced amplification on the first object by using at least one of flipping along a cross section, adding discrete Gaussian noise, and performing histogram equalization;

[0084] The first object and the expanded first object are input into the model to be trained.

[0085] In this embodiment, the first image is still taken as a TOF-MRA image as an example. The first object can be a vascular structure, and the position of the aneurysm in the vascular structure is marked to obtain the second object. In this embodiment, the second object is the aneurysm. According to the position of the aneurysm in each vascular structure, the location of the aneurysm is divided into regions, and the number of second objects in each region is counted. According to the number of each region, the first object is amplified to balance the number of first objects in each region. Finally, the first object and the first object obtained after amplification are input into the area to be detected together. 80% of the input image can be selected as the training set of the model to be trained, and the remaining 20% can be selected as the verification set of the model to be trained.

[0086] As an optional implementation manner, in an embodiment of the present invention, the model to be trained includes an encoding block (EncodeBlock), a feature extraction block (SC Block) and a decoding block (Decode Block), and the contour detection branch and the instance detection branch both include the feature extraction block and the decoding block;

[0087] The encoding block includes M groups of downsampling structures connected in sequence, and the M groups of downsampling structures are used to obtain downsampling results of different scales respectively. The decoding block includes M groups of upsampling structures corresponding one-to-one to the downsampling structures. The sampling results of each group of the downsampling structures are spliced with the features output by the previous level structure of the corresponding upsampling structure as the input features of the upsampling structure.

[0088] like Figure 3 As shown, in this embodiment, Figure 3 For example, the encoding block includes three groups of downsampling structures of different scales connected in sequence, each group of downsampling structures includes two convolution blocks, and the decoding block includes three groups of upsampling structures corresponding to the downsampling structures, and each group of upsampling structures includes two convolution blocks. The decoding block has the same structure as the convolution block in the encoding block, but the decoding block uses a deconvolution structure instead of the convolution structure in the encoding block. Each group of downsampling structures in the encoding block also includes a BiA module, and each group of downsampling structures outputs two feature maps through the BiA module. One of the feature maps is used as the splicing data in the contour detection branch, and the other feature map is used as the splicing data for the instance detection branch.

[0089] Both the contour detection branch and the instance detection branch include feature extraction blocks and decoding blocks. The feature extraction blocks in each branch are connected to the decoding blocks in sequence. The feature extraction blocks serve as the previous level of the decoding blocks, and the output results of the feature extraction blocks will serve as the input data of the decoding blocks.

[0090] The sampling results of each group of downsampling structures are concatenated with the features output by the previous level structure of the corresponding upsampling structure; wherein, the previous level of the upsampling structure corresponding to the first downsampling structure is the feature extraction block.

[0091] by Figure 3 As an example, the decoding block in the contour detection branch, the sampling result of the first group of downsampling structures is d 11 with d 21 , d 11 It is spliced with the feature extraction block in the contour detection branch as the input feature of the first group of upsampling structures; the sampling result of the second group of downsampling structures is d 12 with d 22 , d 12 The feature concatenation of the output of the previous level structure of the upsampling structure corresponding to the second group of downsampling structures is also the feature concatenation of the output of the first group of upsampling structures, and so on.

[0092] by Figure 3 For example, Figure 3 d in 1j d 2j The j in d can be 1, 2, or 3. i1 d i2 d i3 The i in can be 1 or 2.

[0093] The feature extraction block in the contour detection branch is used to extract deep features based on the output of the encoding block. The feature extraction block in the instance detection branch is used to extract deep features based on the output of the encoding block and the upsampling result of the intermediate layer upsampling structure of the decoding block in the contour detection branch. The decoding block also includes a classification layer, which is used to perform classification detection based on the output of the M groups of upsampling structures. Specifically, the classification layer performs classification detection based on the final output of the M groups of sequentially connected upsampling structures.

[0094] In this embodiment, both the feature extraction block in the contour detection branch and the feature extraction block in the instance detection branch play the following roles: strengthening the feature extraction capability of the deep network, extracting the abstract features of the second object of the deep network and the global dependency of the image.

[0095] like Figure 3 As shown in the decoding block in the contour detection branch, the decoding block in the intermediate layer upsampling result is further downsampled, and the downsampling result together with the output of the encoding block is further used as the input of the feature extraction block in the instance detection branch, so that the instance detection branch can further detect the second object in combination with the contour features detected by the contour detection branch.

[0096] The decoding blocks in the contour detection branch and the instance detection branch also include a classification layer, which uses SoftMax to classify and detect the output of the upsampling structure in the encoding block.

[0097] As an optional implementation, in an embodiment of the present invention, each group of the downsampling structure in the encoding block includes a convolution block and a BiA module connected in sequence, the convolution block is used for downsampling, and the BiA module includes two residual branches in parallel, and the two residual branches are used to decouple the features output by the convolution block to obtain two feature maps, and the two feature maps are respectively input into the decoding block in the contour detection branch.

[0098] The convolution block consists of a 3D convolution layer, a batch normalization layer, and a ReLU activation layer, referred to as the Conv+BN+relu fusion layer. Figure 3 As shown in the figure, each downsampling structure includes two convolution blocks. The output result of the first convolution block is used as the input of the next convolution block and is concatenated with the output result of the next convolution block. The output result of the last convolution block is used as the input of the BiA module.

[0099] The BiA module consists of two parallel residual branches, which are used to decouple the features of the two dual-supervised branches: the contour detection branch and the instance detection branch.

[0100] The two residual branches of the BiA module in parallel output two feature maps, one of which is input into the decoding block in the contour detection branch and concatenated with the features output by the previous level structure of the upsampling structure in the decoding block in the contour detection branch; the other feature map is input into the decoding block in the instance detection branch and concatenated with the features output by the previous level structure of the upsampling structure in the decoding block in the instance detection branch.

[0101] As an optional implementation manner, in an embodiment of the present invention, each residual branch of the BiA module includes two residual sub-modules connected in sequence;

[0102] The BiA module also includes a spatial attention mechanism block, which includes a maximum pooling layer MaxPool and an average pooling layer AveSPool connected in sequence. The input of the spatial attention mechanism block is the output of the convolution block of the same group of downsampling structures. The output of the spatial attention mechanism block is used to obtain a weight map through the Sigmoid function, and is multiplied with the output of the last residual submodule of the two residual branches and then added to obtain the two feature maps. The Sigmoid function is a type of SoftMax, and other classification functions can also be used for calculation.

[0103] In this embodiment, the residual submodule adopts ResNet Block, and two residual submodules are connected in sequence. Figure 3 As shown, each residual branch includes two residual submodules. The output of the first residual submodule serves as the input of the next residual submodule. The output of the last residual submodule is multiplied by the weight map of the output of the spatial attention mechanism block and then added to obtain a feature map. Here, the output of the spatial attention mechanism block refers to the output of the spatial attention mechanism block after passing through the maximum pooling layer MaxPool, the average pooling layer AveSPool, and the Sigmoid function. In this embodiment, the use of the spatial attention mechanism block can enhance the ability to learn more discriminative features.

[0104] As an optional implementation, in an embodiment of the present invention, the feature extraction block includes a plurality of downsampling layers and a plurality of upsampling layers connected in sequence;

[0105] Each of the downsampling layers and the upsampling layer also includes a Swin-Transformer layer before it, and the last downsampling layer and the first upsampling layer are spliced using a convolutional layer; the output of the last downsampling layer and the output of the previous downsampling layer are spliced by short-circuiting.

[0106] like Figure 3 As shown in the feature extraction block SC Block, in this embodiment, both the downsampling and upsampling layers use a Conv+BN+ReLU fusion layer. Each downsampling layer and upsampling layer is preceded by a Swin-Transformer layer. The output of the last downsampling layer is fully connected with the output of the previous downsampling layer. The last downsampling layer and the first upsampling layer are concatenated using a 1x1 convolutional layer.

[0107] As an optional implementation, in an embodiment of the present invention, the upsampling result of the intermediate layer upsampling structure of the decoding block in the contour detection branch is downsampled to the output scale of the encoding block, and a weight map is obtained by a Sigmoid function, which is added to the output of the encoding block and then multiplied as the input of the feature extraction block of the instance detection branch.

[0108] As described above, the decoding block in the contour detection branch, the intermediate layer upsampling result, is further downsampled, and the downsampled result and the output of the encoding block are further used as the input of the feature extraction block in the instance detection branch to extract the deep features in the instance detection branch. Specifically, after the intermediate layer result of the decoding block in the contour detection branch is downsampled to the output scale of the encoding block, a weight map is obtained through the Sigmoid function, which is added to the output of the encoding block and then multiplied, and the calculated result is used as the input of the feature extraction block of the instance detection branch. The feature extraction block in the instance detection branch extracts deep features based on this input. The extracted contour feature map is used as a weight, which in turn affects the extraction of instance features.

[0109] As an optional implementation, in an embodiment of the present invention, the first upsampling results of the decoding block at different scales are adjusted for the number of channels through convolution, and then added to the upsampling result elements of a larger scale as deep supervision of the model.

[0110] In this embodiment, Figure 3 As shown in the Decode Block in the figure, the first upsampling result in each upsampling structure in the decoding block adjusts the number of channels through 1x1 convolution and is added to the larger scale upsampling result element by element to achieve deep supervision of the model to be trained.

[0111] As an optional implementation manner, in an embodiment of the present invention, the first loss function value is calculated using the following formula:

[0112]

[0113] Among them, a is the ath connected domain, K is the number of connected domains, p(x b ) is the true value of the input contour detection branch, b is the category of the true value of foreground (or positive) 1 and background (or negative) 0, in other words, b is equal to 1, indicating that it is the second object; b is equal to 0, indicating that it is not the second object, q(x ab ) is the predicted value of the contour detection branch, L CE is the cross entropy loss function, L CDE is the loss value of the contour detection branch.

[0114] Specifically:

[0115]

[0116] Wherein, X is the prediction result matrix of the contour detection branch, Y is the true value, g(X+Y) is the result of adding the corresponding elements of X and Y, Th is the preset threshold of the prediction result of the contour detection branch, and δ is the minimum value.

[0117] When the number of connected domains is greater than or equal to 1, and the sum of the corresponding elements of X and Y is greater than the threshold of the prediction result, q(x ab ) takes the value of the ath connected domain position in the prediction result matrix of the contour detection branch; when the number of connected domains is greater than or equal to 1, and the result of adding the corresponding elements of X and Y is less than the threshold of the prediction result, q(x ab ) takes the minimum value δ in the prediction result matrix of the contour detection branch.

[0118] Regarding the first loss function, when the calculated number of connected domains is 0 or 1, the first loss function is directly calculated using the cross-loss entropy function; when the number of connected domains is greater than 1, the predicted result and the true result are element-wise summed; each connected domain in the element-wise sum result is taken out and cross-entropy is performed with the true value; finally, the obtained series of connected domain cross-entropy results are averaged to obtain the first loss function value.

[0119] Specifically, L CDE It is an improvement based on the cross entropy loss function. The cross entropy loss function L CE for:

[0120]

[0121] Among them, q(x b ) is the predicted value of the contour detection branch.

[0122] The connected domain is the positive area predicted by the contour detection branch. For the data input to the contour detection branch, the number of positive predictions is the same as the number of connected domains. When the prediction is positive, b is 1, and when the prediction is negative, b is 0, where 1 is foreground and 0 is background. CE Directly calculate the cross entropy between the true value and the predicted value.

[0123] As an optional implementation manner, in an embodiment of the present invention, the second loss function value is calculated using the following formula:

[0124]

[0125] Where X is the matrix of the prediction results in the instance detection branch, Y is the true value input into the instance detection branch, and L Dice The loss value of the detection branch for the instance.

[0126] Use the first loss function L of the connected domain cross entropy CDE and the second loss function L DiceThe losses for the contour detection branch and the instance detection branch are calculated separately, and their weighted summation yields the complete loss function for the model to be trained. The connected domain cross entropy loss function accounts for the cross entropy losses of all positive and annotated regions, resulting in more sensitive detection results from the model to be trained.

[0127] Example 2

[0128] This embodiment provides a model training device, which can be used to execute the model training method in the above embodiment 1. The device can be set inside a server or other device, and the modules cooperate with each other to achieve model training, such as Figure 4 As shown, the device includes:

[0129] A first acquisition module 201 is used to acquire multiple first images;

[0130] The processing module 202 is configured to extract, for each of the first images, a first object in the first image; and obtain a second object instance annotated in the first image as a second object instance label;

[0131] A second acquisition module 203 is configured to acquire the outline of the first object;

[0132] a labeling module 204, configured to obtain, according to the outline of the first object and the second object instance, the outline of the second object as a second object outline label;

[0133] A detection module 205 is configured to input the first object into a to-be-trained model and obtain an output of the to-be-trained model; the to-be-trained model includes a contour detection branch and an instance detection branch, and the output includes a second object contour output and a second object instance output; the instance detection branch detects a second object contour based on the first object and contour features extracted by the contour detection branch;

[0134] A first calculation module 206 is configured to calculate a first loss function value based on the second object instance label and the second object instance output, and calculate a second loss function value based on the second object contour label and the second object contour output;

[0135] A second calculation module 207 is configured to calculate a third loss function value based on the first loss function value and the second loss function value;

[0136] The adjustment module 208 is used to adjust the parameters of the model to be trained based on the third loss function value.

[0137] The dual-supervised model trained using the model training method provided in this embodiment takes into account that the target object (i.e., the second object) in the image to be detected is closely related to the first object. For example, an aneurysm is an abnormal bulge in the arterial wall. Therefore, before using the dual-supervised model to detect the second object in the image, the first object must be extracted from the image to be detected. The first object is then input into the dual-supervised model to output the detection result. Preprocessing the first object eliminates the need for time-consuming operations such as N4 correction, resulting in higher detection efficiency. Furthermore, during detection, the first image does not need to be segmented into small blocks, resulting in higher sensitivity and avoiding operations such as image reconstruction. Furthermore, the dual-supervised model detects the first object, which is closely related to the second object, rather than the entire image content. This improves both efficiency and accuracy. The two branches of the dual-supervised model extract the contour features and overall features of the second object, respectively. Based on the contour features, the model detects the possible second object. The overall features are then used to further determine whether the possible second object is indeed the second object, thereby reducing the false detection rate and improving detection accuracy.

[0138] For a detailed description of the above-mentioned device part, please refer to the above-mentioned method embodiment, which will not be repeated here.

[0139] Example 3

[0140] This embodiment provides a computer device, such as Figure 5 As shown, the computer device includes a processor 301 and a memory 302, wherein the processor 301 and the memory 302 can be connected via a bus or other means. Figure 5 The bus connection is taken as an example.

[0141] The processor 301 may be a central processing unit (CPU). The processor 301 may also be other general-purpose processors, digital signal processors (DSP), graphics processing units (GPU), embedded neural network processors (NPU), or other dedicated deep learning coprocessors, application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0142] Memory 302, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the model training method in the embodiments of the present invention. The corresponding program instructions / modules. Processor 301 executes the non-transitory software programs, instructions, and modules stored in memory 302 to perform various processor functions and data processing, thereby implementing the model training method in the above-mentioned method embodiments.

[0143] The memory 302 may also include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created by the processor 301, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 302 may optionally include a memory remotely located relative to the processor 301, and these remote memories may be connected to the processor 301 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The memory 302 stores one or more modules, which, when executed by the processor 301, perform the following operations: Figure 1 The model training method in the illustrated embodiment.

[0145] For details of the above computer equipment, please refer to Figure 1 The corresponding descriptions and effects in the embodiments shown can be understood and will not be repeated here.

[0146] An embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions that can execute the model training method in any of the above embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the storage medium can also include a combination of the above types of memory.

[0147] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A model training method, characterized in that: include: acquiring a plurality of first images; For each of the first images, extracting a first object in the first image; and obtaining a second object instance annotated from the first image as a second object instance label; obtaining a contour of the first object; According to the outline of the first object and the second object instance, obtaining the outline of the second object as a second object outline label; Inputting the first object into a to-be-trained model and obtaining an output of the to-be-trained model; the to-be-trained model includes a contour detection branch and an instance detection branch, and the output includes a second object contour output and a second object instance output; the instance detection branch detects a second object contour based on the first object and contour features extracted by the contour detection branch; Calculating a first loss function value based on the second object instance label and the second object instance output, and calculating a second loss function value based on the second object contour label and the second object contour output; Calculate a third loss function value based on the first loss function value and the second loss function value; Adjusting the parameters of the model to be trained based on the third loss function value; The first loss function value is calculated using the following formula: Where a is the ath connected domain, K is the number of connected domains, p(x b ) is the true value of the input contour detection branch, b is the category of the foreground 1 and background 0 of the true value, q(x ab ) is the predicted value of the contour detection branch, L CE is the cross entropy loss function, L CDE is the loss value of the contour detection branch; The second loss function value is calculated using the following formula: Where X is the matrix of the prediction results in the instance detection branch, Y is the true value input into the instance detection branch, and L Dice A loss value for the instance detection branch; The third loss function value is calculated using the following formula: Among them, α is the calculation weight of the first loss function, and β is the calculation weight of the second loss function.

2. The model training method according to claim 1, characterized in that Inputting the first object into the model to be trained includes: dividing the second object obtained by labeling the first object according to the position of the second object in the first object; Counting the number of the second objects at each position; Based on the number, performing position-balanced amplification on the first object by using at least one of flipping along a cross section, adding discrete Gaussian noise, and performing histogram equalization; The first object and the expanded first object are input into the model to be trained.

3. The model training method according to claim 1, characterized in that The model to be trained includes an encoding block, a feature extraction block and a decoding block, and the contour detection branch and the instance detection branch both include the feature extraction block and the decoding block; The encoding block includes M groups of downsampling structures connected in sequence, and the M groups of downsampling structures are used to obtain downsampling results of different scales respectively. The decoding block includes M groups of upsampling structures corresponding to the downsampling structures one by one. The sampling results of each group of the downsampling structures are spliced with the features output by the previous level structure of the corresponding upsampling structure to serve as the input features of the upsampling structure; The feature extraction block in the contour detection branch is used to extract deep features based on the output of the encoding block, and the feature extraction block in the instance detection branch is used to extract deep features based on the output of the encoding block and the upsampling result of the intermediate layer upsampling structure of the decoding block in the contour detection branch. The decoding block also includes a classification layer, which is used to perform classification detection based on the output of M groups of the upsampling structures.

4. The method according to claim 3, characterized in that Each group of the downsampling structure in the encoding block includes a convolution block and a BiA module connected in sequence, the convolution block is used for downsampling, and the BiA module includes two residual branches connected in parallel. The two residual branches are used to decouple the features output by the convolution block to obtain two feature maps, and the two feature maps are respectively input into the decoding block in the contour detection branch and the decoding block in the instance detection branch.

5. The method according to claim 4, characterized in that Each residual branch of the BiA module includes two residual submodules connected in sequence; The BiA module also includes a spatial attention mechanism block, which includes a maximum pooling layer and an average pooling layer connected in sequence. The input of the spatial attention mechanism block is the output of the convolution block of the same group of downsampling structures. The output of the spatial attention mechanism block obtains a weight map through a Sigmoid function and is respectively combined with the output of the residual sub-module after the two residual branches.

6. The method according to claim 3, characterized in that The feature extraction block includes a plurality of downsampling layers and a plurality of upsampling layers connected in sequence; Each of the downsampling layers and the upsampling layer also includes a Swin-Transformer layer before it, and the last downsampling layer and the first upsampling layer are spliced using a convolutional layer; the output of the last downsampling layer and the output of the previous downsampling layer are spliced by short-circuiting.

7. The method according to claim 3, characterized in that The upsampling result of the intermediate layer upsampling structure of the decoding block in the contour detection branch is downsampled to the output scale of the encoding block, and a weight map is obtained by a Sigmoid function, which is added to the output of the encoding block and then multiplied as the input of the feature extraction block of the instance detection branch.

8. The method according to claim 3, characterized in that The first upsampling results of the decoding block at different scales are adjusted for the number of channels through convolution, and then added to the upsampling result elements of the larger scale as deep supervision of the model to be trained.

9. A model training device, characterized in that: include: A first acquisition module, configured to acquire a plurality of first images; a processing module, configured to extract a first object in each of the first images; and obtaining a second object instance annotated from the first image as a second object instance label; A second acquisition module, configured to acquire the outline of the first object; a labeling module, configured to obtain, according to the outline of the first object and the second object instance, the outline of the second object as a second object outline label; a detection module, configured to input the first object into a to-be-trained model and obtain an output of the to-be-trained model; the to-be-trained model includes a contour detection branch and an instance detection branch, the output including a second object contour output and a second object instance output; the instance detection branch detects a second object contour based on the first object and contour features extracted by the contour detection branch; a first calculation module, configured to calculate a first loss function value based on the second object instance label and the second object instance output, and to calculate a second loss function value based on the second object contour label and the second object contour output; The first loss function value is calculated using the following formula: Where a is the ath connected domain, K is the number of connected domains, p(x b ) is the true value of the input contour detection branch, b is the category of the foreground 1 and background 0 of the true value, q(x ab ) is the predicted value of the contour detection branch, L CE is the cross entropy loss function, L CDE is the loss value of the contour detection branch; The second loss function value is calculated using the following formula: Where X is the matrix of the prediction results in the instance detection branch, Y is the true value input into the instance detection branch, and L Dice A loss value for the instance detection branch; The second calculation module is used to calculate a third loss function value based on the first loss function value and the second loss function value; the third loss function value is calculated using the following formula: L=αL CDE +βL Dice Wherein, α is the calculation weight of the first loss function, and β is the calculation weight of the second loss function; An adjustment module is used to adjust the parameters of the model to be trained based on the third loss function value.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the model training method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the model training method described in any one of claims 1-8.

Images