Object detection model training method and object detection method

By using a combination of multi-layer network feature extraction and residual network in medical imaging detection, the object detection model is trained, and the problem of low detection rate of micro lesions is solved, improving the accuracy and efficiency of detection.

CN115965829BActive Publication Date: 2024-11-15ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211482182.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-11-24
Publication Date
2024-11-15
Estimated Expiration
2042-11-24

AI Technical Summary

Technical Problem

The existing medical imaging detection algorithm has low detection rate of micro lesions, resulting in inaccurate detection results.

Method used

By inputting image training samples into feature extraction networks of multiple network layers for feature processing, and using residual networks for feature extraction and prediction, combining detection heads of different network layers for prediction, the object detection model is trained to improve the small-size target object detection capability.

Benefits of technology

It improves the detection ability of small-sized target objects, reduces false positives, and improves the accuracy and efficiency of medical imaging detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115965829B_ABST
    Figure CN115965829B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide an object detection model training method and an object detection method, which includes inputting image training samples into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training samples in the target network layer; processing the initial image features through the residual network corresponding to the target network layer to obtain the target image features of the image training samples in the target network layer; inputting the target image features into the detection head corresponding to the target network layer for prediction to obtain the prediction results of the image training samples in the target network layer; training the object detection model according to the prediction results of the image training samples in the at least two network layers and the sample labels corresponding to the image training samples. This method can be used in the fields of image and video detection to improve the detection rate of small-sized target objects and the detection effect of images by adding residual networks in different network layers for feature extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a method for training an object detection model. Background Art

[0002] As living standards improve, people pay more and more attention to their physical health. For example, people have a physical examination every year, which generates a large number of medical image reading needs (such as lung CT (Computed Tomography), chest CT, etc.). In addition, with the development of artificial intelligence technology, more and more artificial intelligence algorithms are used in intelligent reading of medical images to help doctors find potential lesions faster, such as lung nodule detection and rib fracture detection based on CT images. Algorithms assist doctors in reading images, which can reduce the burden of doctors reading images.

[0003] However, there are still some problems with this method. For example, some algorithms easily miss small lesions, resulting in a low detection rate of small lesions, and overall inaccurate detection results for CT images through the algorithm. Summary of the invention

[0004] In view of this, an embodiment of this specification provides an object detection model training method. One or more embodiments of this specification also relate to an object detection model training device, an object detection method and device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, a method for training an object detection model is provided, comprising:

[0006] Inputting the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training sample in the target network layer, wherein the target network layer is any one of the at least two network layers;

[0007] Processing the initial image features through the residual network corresponding to the target network layer to obtain the target image features of the image training sample in the target network layer, wherein the larger the number of layers of the target network layer, the smaller the number of corresponding residual networks;

[0008] Inputting the target image features into the detection head corresponding to the target network layer for prediction, and obtaining the prediction result of the image training sample in the target network layer;

[0009] The object detection model is trained according to the prediction results of the image training samples in the at least two network layers and the sample labels corresponding to the image training samples.

[0010] According to a second aspect of an embodiment of this specification, there is provided an object detection model training device, comprising:

[0011] A first feature processing module is configured to input the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing, and obtain the initial image features of the image training sample in the target network layer, wherein the target network layer is any one of the at least two network layers;

[0012] A second feature processing module is configured to perform feature processing on the initial image features through the residual network corresponding to the target network layer to obtain the target image features of the image training sample in the target network layer, wherein the larger the number of layers of the target network layer, the smaller the number of corresponding residual networks;

[0013] A prediction module is configured to input the target image features into the detection head corresponding to the target network layer for prediction, and obtain the prediction result of the image training sample at the target network layer;

[0014] The training module is configured to train the object detection model according to the prediction results of the image training samples in the at least two network layers and the sample labels corresponding to the image training samples.

[0015] According to a third aspect of the embodiments of this specification, there is provided an object detection method, including:

[0016] determining a target part image containing a target object;

[0017] Input the target part image into the object detection model to obtain the object category, position coordinates and size information of the target object in the target part image,

[0018] Wherein, the object detection model is obtained by training using the above-mentioned object detection model training method.

[0019] According to a fourth aspect of the embodiments of this specification, there is provided an object detection device, including:

[0020] An image determination module is configured to determine a target part image containing a target object;

[0021] The object detection module is configured to input the target part image into the object detection model to obtain the object category, position coordinates and size information of the target object in the target part image,

[0022] Wherein, the object detection model is obtained by training using the above-mentioned object detection model training method.

[0023] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0024] Memory and processor;

[0025] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned object detection model training method or object detection method are implemented.

[0026] According to a sixth aspect of an embodiment of the present specification, a computer-readable storage medium is provided, which stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned object detection model training method or object detection method.

[0027] According to a seventh aspect of an embodiment of the present specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned object detection model training method or object detection method.

[0028] An embodiment of the present specification implements an object detection model training method, including inputting an image training sample into a feature extraction network of at least two network layers of an object detection model for feature processing to obtain initial image features of the image training sample in a target network layer, wherein the target network layer is any one of the at least two network layers; processing the initial image features through a residual network corresponding to the target network layer to obtain target image features of the image training sample in the target network layer, wherein the larger the number of layers of the target network layer, the smaller the number of corresponding residual networks; inputting the target image features into a detection head corresponding to the target network layer for prediction to obtain prediction results of the image training sample in the target network layer; and training the object detection model according to the prediction results of the image training sample in the at least two network layers and the sample labels corresponding to the image training samples.

[0029] Specifically, the method can be used in fields such as image and video detection. Feature processing is performed on target objects of different sizes through feature extraction networks of different network layers. For example, a shallow feature extraction network (such as the first layer) is used to extract features of target objects of smaller sizes, and a deep feature extraction network is used to extract features of target objects of larger sizes. In this way, the model's detection capability for small-sized target objects is improved. At the same time, since the feature extraction network of the shallow network layer extracts fewer features, the present application further extracts features by adding different numbers of residual networks in different deep and shallow network layers to improve the detection rate of small-sized target objects and improve the overall detection effect for images. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of a specific implementation scenario of an object detection model training method provided by an embodiment of this specification;

[0031] Figure 2 is a flow chart of an object detection model training method provided by one embodiment of this specification;

[0032] Figure 3 is a flow chart of an object detection method provided by one embodiment of this specification;

[0033] Figure 4 is a schematic diagram of training samples and corresponding training labels of an image segmentation model in an object detection method provided by an embodiment of this specification;

[0034] Figure 5 is a schematic diagram of visually displaying the results on a lung CT image in an object detection method provided in an embodiment of this specification;

[0035] Figure 6 is a processing flow chart of an object detection method provided by an embodiment of this specification;

[0036] Figure 7 is a structural diagram of an object detection model training device provided by an embodiment of this specification;

[0037] Figure 8 is a schematic diagram of the structure of an object detection device provided by an embodiment of this specification;

[0038] Fig. 9 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0039] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0040] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0042] First, the terms involved in one or more embodiments of this specification are explained.

[0043] Lesion detection: Use deep learning detection algorithms to find potential lesions in medical images and provide information such as its location coordinates, lesion category, and confidence level.

[0044] In this specification, a method for training an object detection model is provided. One or more embodiments of this specification also relate to an object detection model training device, an object detection method and device, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0045] See also Figure 1 , Figure 1 A schematic diagram of a specific implementation scenario of an object detection model training method provided according to an embodiment of the present specification is shown.

[0046] Figure 1The cloud-side device 102 and the terminal-side device 104 are included, wherein the cloud-side device 102 can be understood as a cloud server. Of course, in another feasible solution, the cloud-side device 102 can also be replaced by a physical server; the terminal-side device 104 includes but is not limited to a desktop computer, a laptop computer, etc.; for ease of understanding, in the embodiments of this specification, the cloud-side device 102 is a cloud server and the terminal-side device 104 is a laptop computer for detailed introduction.

[0047] The object detection model training method provided in the embodiments of this specification is applied to the training scenario of the object detection model for predicting lung nodules in lung CT images.

[0048] During specific implementation, object detection model training is performed on the cloud-side device 102 , wherein the object detection model can be understood as a 3D (three dimensional) detection model without an anchor box.

[0049] like Figure 1 As shown, Figure 1 In the object detection model, C1-C4 represent the resnet backbone, P1-P4 represent the feature pyramid, R represents the residual network, and each detection head (HEADS) contains a classification branch and a regression branch, wherein the classification branch is used to determine whether the current position is a point within a lesion, and the regression branch is used to regress the size of the lesion.

[0050] In practical applications, the structure of the object detection model can be adjusted according to actual needs. For example, the number of layers C1-C4, P1-P4, and the number of residual networks in each network layer can be adjusted.

[0051] For ease of understanding, in the embodiments of this specification, Figure 1 Taking the object detection model in as an example, the training process of the object detection model is explained in detail.

[0052] The cloud-side device 102 determines a training sample set, wherein the training sample set includes multiple image training samples and sample labels corresponding to each image training sample, and the multiple image training samples can be determined by CT images of multiple target parts of the human body, for example, each image training sample can be understood as a CT image of a human lung. In the case where the image training sample is a CT image of a human lung, the sample label corresponding to the image training sample can be understood as the object category (ie, yes or no), coordinate position, and size information of the lung nodule in the CT image of the human lung.

[0053] Specifically, when training the object detection model, each image training sample is input into C1-C4 for downsampling, and then the downsampling result of C4 is input into P4-P1 for upsampling. At the same time, the corresponding downsampling results of C1-C4 are combined to obtain the initial image features of each image training sample at each network layer, for example Figure 1 The four initial image features of the first, second, third and fourth network layers.

[0054] Then, the initial image features of each image training sample in each network layer are convolved through the corresponding number of residual networks in its network layer to obtain the target image features of each image training sample in each network layer; for example, the initial image features of the image training sample in the first network layer are convolved through the three residual networks (i.e., R11-R13) corresponding to its network layer to obtain the target image features of the image training sample in the first network layer; the initial image features of the image training sample in the second network layer are convolved through the two residual networks (i.e., R21-R22) corresponding to its network layer to obtain the target image features of the image training sample in the second network layer; the initial image features of the image training sample in the third network layer are convolved through the one residual network (i.e., R3) corresponding to its network layer to obtain the target image features of the image training sample in the third network layer, etc. When the corresponding residual network is not set in the fourth network layer, the initial image features of the image training sample in the fourth network layer can be determined as the target image features.

[0055] After obtaining the target image features of the image training samples in each network layer, the target image features of each network layer are input into the detection head (i.e., HEADS) corresponding to each network layer to predict lung nodules to obtain prediction results, wherein the prediction results include the object category, coordinate position, and size information of the predicted lung nodules in the image training samples.

[0056] Finally, the model loss function is calculated according to the sample label of the image training sample and the prediction result, and the network parameters of the object detection model are adjusted according to the model loss function until the object detection model meets the preset model training end conditions (such as the number of iterations meets 20,000 times, or the difference between the model loss function of this round of model training and the model loss function of the previous round of model training meets the preset threshold, etc.), and the training process of the object detection model is completed.

[0057] When the terminal device 104 needs to use the object detection model, it can call the object detection model obtained after training in the cloud device 102 for functional use; in addition, if the computing resources and computing power of the terminal device 104 are sufficient, the object detection model trained in the cloud device 102 can also be deployed on the terminal device 104. The specific deployment and implementation depends on the actual application, and no limitation is made here.

[0058] The object detection model training method provided in the embodiments of this specification performs feature processing on lung nodules of different sizes through the residual network backbone and feature pyramid of different network layers. For example, the residual network backbone and feature pyramid of the shallow layer (such as the first layer) are used to extract features of smaller lung nodules, and the residual network backbone and feature pyramid of the deep layer are used to extract features of larger lung nodules. In this way, the model's detection capability for small lung nodules is improved. At the same time, different numbers of residual networks are added to different deep and shallow network layers for further feature extraction. This can avoid the problem of missing small lung nodules and poor image detection effect caused by the feature extraction network of the shallow network layer extracting fewer features.

[0059] See also Figure 2 , Figure 2 A flowchart of an object detection model training method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0060] Step 202: Input the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training sample in the target network layer.

[0061] The target network layer is any one of the at least two network layers.

[0062] Specifically, the object detection model training method provided in the embodiments of this specification can be applied to the training scenario of an object detection model for predicting lung nodules in lung CT images, can also be applied to the training scenario of an object detection model for predicting fracture sites in rib CT images, and can also be applied to other applicable scenarios.

[0063] The actual application scenarios of the object detection model training method are different, and the image training samples are also different. For example, when the object detection model training method is applied to the training scenario of the object detection model for predicting lung nodules in lung CT images, the image training samples can be understood as lung CT images; when the object detection model training method is applied to the training scenario of the object detection model for predicting fracture sites in rib CT images, the image training samples can be understood as rib CT images, etc.

[0064] For ease of understanding, in the embodiments of this specification, the object detection model training method is applied to the training scenario of the object detection model for predicting lung nodules in lung CT images, and the image training sample is understood as a lung CT image for detailed description.

[0065] The object detection model can refer to the object detection model including four network layers in the above embodiment. Of course, in specific applications, an object detection model with five or six network layers can also be used. For ease of understanding, in the embodiments of this specification, the object detection model is taken as the object detection model in the above embodiment as an example for the following introduction.

[0066] Taking the image training sample as a lung CT image as an example, the image training sample is input into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training sample in the target network layer. It can be understood that the lung CT image is input into the object detection model, and feature processing is performed through the feature extraction network of each network layer in at least two network layers of the object detection model to obtain the initial image features of the lung CT image in the target network layer, wherein the target network layer is any one of the four network layers in the object detection model.

[0067] That is, when the four network layers of the object detection model are respectively used as the target network layers, the lung CT image is input into the object detection model, and feature processing is performed through the feature extraction network of each network layer in at least two network layers of the object detection model, so as to obtain the initial image features of the lung CT image in the first, second, third, and fourth network layers respectively.

[0068] In practical applications, the feature extraction network includes the residual network backbone and feature pyramid in the above embodiment. When the image training sample is input into the object detection model, it is first downsampled through the residual network backbone, and then upsampled through the feature pyramid. Finally, according to the downsampling result and the upsampling result, the initial image features of the image training sample in each network layer are obtained. By extracting features in this upsampling and downsampling manner, each network layer can extract features for target objects of different sizes, avoiding the subsequent object detection model from missing small-sized target objects. The specific implementation method is as follows:

[0069] The step of inputting the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training sample in the target network layer includes:

[0070] Inputting the image training sample into the first feature extraction network of at least two network layers of the object detection model for feature processing, and obtaining the downsampling result of the image training sample in the first feature extraction network of the target network layer;

[0071] The down-sampling result is processed by the second feature extraction network of the at least two network layers to obtain an up-sampling result of the image training sample in the second feature extraction network of the target network layer;

[0072] According to the down-sampling result and the up-sampling result, initial image features of the image training sample at the target network layer are obtained.

[0073] Among them, the first feature extraction network of at least two network layers of the object detection model can be understood as the residual network backbone of the above embodiment, and the second feature extraction network of at least two network layers of the object detection model can be understood as the feature pyramid of the above embodiment.

[0074] Still using the above example, the lung CT image is input into the first feature extraction network corresponding to each network layer of the object detection model for downsampling, and the downsampling result of the first feature extraction network of the lung CT image in the target network layer is obtained; that is, when the four network layers of the object detection model are respectively used as the target network layers, the lung CT image is input into the first feature extraction network corresponding to each network layer of the object detection model for downsampling, and the downsampling results of the first feature extraction network of the lung CT image in the first, second, third, and fourth network layers are obtained respectively.

[0075] Then, the downsampling result of the first feature extraction network of the last network layer of the lung CT image is input into the second feature extraction network corresponding to each network layer of the object detection model for upsampling, so as to obtain the upsampling result of the second feature extraction network of the lung CT image in the target network layer; that is, when the four network layers of the object detection model are respectively used as the target network layers, the downsampling result of the first feature extraction network of the last network layer of the lung CT image is input into the second feature extraction network corresponding to each network layer of the object detection model for upsampling, so as to obtain the upsampling results of the second feature extraction networks of the first, second, third and fourth network layers of the lung CT image respectively.

[0076] Finally, according to the down-sampling results of the first feature extraction network of the lung CT image in the first, second, third and fourth network layers and the up-sampling results of the second feature extraction network of the lung CT image in the first, second, third and fourth network layers, the initial image features of the lung CT image in the first, second, third and fourth network layers are obtained.

[0077] For example, the downsampling result of the first feature extraction network of the lung CT image in the first network layer is fused with the upsampling result of the second feature extraction network of the lung CT image in the first network layer to obtain the initial image features of the lung CT image in the first network layer; similarly, the initial image features of the lung CT image in the second, third, and fourth network layers are obtained in this way.

[0078] In the object detection model training method provided in the embodiments of this specification, the image training samples are downsampled through the first feature extraction network and the second feature extraction network of multiple network layers, so that large-sized target objects in the image training samples can obtain a larger receptive field through multiple downsampling, and the large-sized target objects can be better identified subsequently; and the more times of downsampling, the more difficult it is for small-sized target objects in the image training samples to be seen on the feature map, and at this time, multiple upsampling is performed, so that small-sized target objects in the image training samples can obtain a larger receptive field through multiple upsampling, and the small-sized target objects can be better identified subsequently, avoiding missed detection of small-sized target objects in the image training samples.

[0079] Step 204: Process the initial image features through the residual network corresponding to the target network layer to obtain the target image features of the image training sample in the target network layer.

[0080] Among them, the larger the number of layers of the target network layer, the smaller the number of corresponding residual networks.

[0081] In practical applications, since the shallower (such as the upper) network layer can extract fewer features and the level of abstraction is not high, if the prediction is directly performed through the detection head of the network layer, it will affect the judgment of whether the small-sized target object is the real target object. Therefore, after the second feature extraction network of each network layer, a corresponding number of residual networks are added to further extract features, and then input to the detection head of the network layer for prediction, better prediction results can be obtained. In practical applications, the more shallow the network layer, the more residual networks are set, that is, the shallower the network layer, the fewer features are extracted. Further extracting features through multiple residual networks can increase the number of features and improve the prediction effect of the detection head of the subsequent network layer.

[0082] Specifically, when the four network layers of the object detection model are respectively used as target network layers, the initial image features are processed by the residual network corresponding to the target network layer to obtain the target image features of the image training samples in the target network layer; it can be understood that the initial image features of the first, second, third, and fourth network layers are processed by the residual networks corresponding to their network layers to obtain the target image features of the image training samples in the first, second, third, and fourth network layers respectively.

[0083] Step 206: Input the target image features into the detection head corresponding to the target network layer for prediction, and obtain the prediction result of the image training sample in the target network layer.

[0084] The detection header corresponding to the target network layer may be understood as the HEADS in the above embodiment.

[0085] Specifically, after obtaining the target image features of the image training samples at the first, second, third, and fourth network layers, each target image feature is input into the detection head corresponding to its network layer for prediction to obtain the prediction results of the image training samples at the first, second, third, and fourth network layers.

[0086] In practical applications, in order to further improve the recognition effect of small-sized target objects in image training samples, the target image features of the image training samples in the next layer of the network will be combined with the target image features of the image training samples in the current layer and input into the detection head corresponding to its network layer for prediction, so as to obtain more accurate prediction results of the image training samples in the first, second, third, and fourth network layers. The specific implementation method is as follows:

[0087] The step of inputting the target image features into the detection head corresponding to the target network layer for prediction, and obtaining the prediction result of the image training sample at the target network layer, comprises:

[0088] Determine a target image feature of the image training sample at a next network layer of the target network layer;

[0089] The target image features of the image training samples at the target network layer and the target image features of the image training samples at the next network layer of the target network layer are fused, and the results are input into the detection head corresponding to the target network layer for prediction to obtain the prediction results of the image training samples at the target network layer.

[0090] In practical applications, the process of combining the target image features of the next network layer with the target network layer is equivalent to the upsampling process, which helps predict small-sized target objects in the image training samples in the target network layer; and because the size of the target image features of the next network layer of the target network layer is only half of the target image features of the target network layer, it is necessary to first perform a transposed convolution operation on the target image features of the next network layer with a step size of 2 to obtain image features of the same size as the target image features of the target network layer, and then perform a fusion operation with the target image features of the target network layer and input them into the detection head corresponding to the target network layer for prediction.

[0091] Take the target network layer as the first network layer as an example for introduction; in specific implementation, determine the target image features of the image training sample in the next network layer (second network layer) of the first network layer; fuse the target image features of the image training sample in the first network layer with the target image features of the image training sample in the second network layer obtained after a transposed convolution operation with a step size of 2, and input them into the detection head corresponding to the image training sample in the first network layer for prediction, and obtain the prediction result of the image training sample in the first network layer. Similarly, when the target network layer is the second network layer or the third network layer, the method for obtaining the prediction result of the image training sample in the target network layer is as described above, and will not be repeated here; and when the target network layer is the fourth network layer, since there is no next network layer for the fourth network layer, there is no need to perform feature fusion in the above manner, that is, the initial image features of the fourth network layer can be directly input as the target image features into the detection head corresponding to the target network layer for prediction, and obtain the prediction result of the image training sample in the target network layer.

[0092] Step 208: training the object detection model according to the prediction results of the image training samples in the at least two network layers and the sample labels corresponding to the image training samples.

[0093] The image training samples are image training samples of target parts, such as image training samples of lungs, i.e., lung CT images, and the object detection model is a neural network model, such as the anchor-free 3D detection model in the above embodiment.

[0094] Specifically, after obtaining the prediction results of each network layer in the at least two network layers of the image training sample, the object detection model is trained in combination with the sample label corresponding to the image training sample. The specific implementation method is as follows:

[0095] The prediction result includes the predicted object category, predicted coordinate position and predicted size information of the target object in the image training sample in the target network layer, and the sample label corresponding to the image training sample is the target object category, target coordinate position and target size information of the target object in the image training sample;

[0096] Accordingly, training the object detection model according to the prediction results of the image training samples at the at least two network layers and the sample labels corresponding to the image training samples includes:

[0097] Determine the object category loss function, coordinate position loss function and size information loss function of the target network layer according to the predicted object category, predicted coordinate position and predicted size information of the target object in the image training sample at the target network layer, as well as the target object category, the target coordinate position and the target size information;

[0098] The size information loss function is weighted according to the inverse of the predicted size information as a weight to obtain a weighted size information loss function;

[0099] Determining a layer loss function of the target network layer according to the weighted size information loss function, the coordinate position loss function and the object category loss function;

[0100] A model loss function is determined according to the layer loss functions of the at least two network layers, and the object detection model is trained according to the model loss function.

[0101] In practical applications, considering that the small-sized target objects in the image training samples are more sensitive to the offset of the regression box (the position coordinates, size information and object category determined by the target object in the image training samples) than the large-sized target objects in the image training samples, in the calculation of the model loss function, the size information loss function of the regression box can be weighted according to the inverse of the size of the target object as the weight, so that during model training, the deviation penalty for small-sized target objects is greater, thereby ensuring the subsequent prediction accuracy of small-sized target objects through the model.

[0102] The target network layer is still taken as the first network layer as an example for introduction.

[0103] Specifically, according to the predicted object category, predicted coordinate position, predicted size information of the target object in the first network layer in the image training sample, as well as the target object category, target coordinate position, and target size information, the object category loss function, coordinate position loss function, and size information loss function of the first network layer are determined; then the size information loss function is weighted according to the inverse of the predicted size information as a weight to obtain the weighted size information loss function, and then the layer loss function of the first network layer is determined according to the weighted size information loss function, coordinate position loss function, and object category loss function; similarly, through the above method, when the target network layer is the second, third, or fourth network layer, the layer loss functions of the second, third, or fourth network layers are determined.

[0104] Finally, according to the layer loss functions of the first, second, third and fourth network layers, the model loss function of the entire object detection model is determined, and the object detection model is trained according to the model loss function.

[0105] according to Figure 1 It can be seen that from top to bottom in the network layer are feature maps with higher resolution. The feature maps of each network layer will be connected to the detection head of the prediction result. Each detection head includes three heads, namely the classification branch, the center branch (centerness) and the regression branch, which are used to predict whether each pixel in the feature map belongs to a target object, the centrality of the point relative to the center point of the target object, and the position coordinates and size information of the actual center point of the target object. The results predicted by these three heads will be calculated with the corresponding values ​​in the sample label. The loss (loss function) is calculated by adding the three losses with a certain weighted sum to obtain the loss of each network layer (i.e., the layer loss function), and the loss of each network layer is weighted and summed to obtain the final loss (i.e., the model loss function). The object detection model is trained by back propagation of the final loss.

[0106] The object detection model training method provided in the embodiments of this specification uses the inverse of the size of the target object in the image training sample as a weight in the calculation of the model loss function, so that more emphasis is placed on the position regression accuracy of small-sized target objects during model training, and the position regression is more accurate, thereby improving the ability to detect small-sized target objects.

[0107] In addition, in order to ensure that the detection head of each network layer predicts fewer false positives during model training, the weight of each network layer in the next round of model training will be adjusted according to the prediction results of the detection head of each network layer in the previous round of model training to effectively suppress the generation of false positives. The specific implementation method is as follows:

[0108] The step of training the object detection model according to the model loss function comprises:

[0109] When it is determined that the difference between the model loss function and the model loss function corresponding to the previous round of model training is greater than or equal to a preset threshold, adjusting the network parameters of the object detection model according to the model loss function to obtain an updated object detection model;

[0110] Inputting the image verification sample into the updated object detection model to obtain a verification result;

[0111] According to the verification result and the sample label corresponding to the image verification sample, adjust the weights of the layer loss functions of the at least two network layers in the next round of model training in the model loss function, and perform the next round of model training;

[0112] Until the preset model training end conditions are met, the trained object detection model is obtained.

[0113] Specifically, when the difference between the model loss function and the model loss function corresponding to the previous round of model training is greater than or equal to a preset threshold, it can be determined that the loss functions of the two iterations are quite different and the model needs to continue training; at this time, the network parameters of the object detection model are adjusted according to the model loss function to obtain an updated object detection model; then the image verification sample is input into the updated object detection model to obtain a verification result; then, based on the verification result and the sample label corresponding to the image verification sample, the weights of the layer loss functions of at least two network layers in the next round of model training in the model loss function are adjusted, and the next round of model training is performed until the preset model training end conditions are met to obtain a trained object detection model.

[0114] Among them, for a detailed introduction to image verification samples and sample labels corresponding to image verification samples, please refer to the detailed explanation of image training samples and sample labels corresponding to image training samples in the above embodiments. The only difference is that image training samples and sample labels corresponding to image training samples are for model training, while image verification samples and sample labels corresponding to image verification samples are for model verification.

[0115] The preset model training end condition can be understood as that the difference between the model loss function and the model loss function corresponding to the previous round of model training is less than the preset threshold. That is, when the difference between the model loss function and the model loss function corresponding to the previous round of model training is less than the preset threshold, it can be determined that the model loss function tends to be stable. At this time, the model training iteration ends.

[0116] In specific implementation, the specific implementation method of adjusting the weights of the layer loss functions of the at least two network layers in the next round of model training in the model loss function is as follows:

[0117] The verification result includes verification size information of the target object in the image verification sample in the at least two network layers, and the sample label corresponding to the image verification sample is the target size information of the target object in the image verification sample;

[0118] Accordingly, adjusting the weights of the layer loss functions of the at least two network layers in the next round of model training in the model loss function according to the verification result and the sample label corresponding to the image verification sample includes:

[0119] Determine a layer verification result corresponding to each network layer according to verification size information of the target object in the image verification sample at the at least two network layers;

[0120] Determining a layer target result corresponding to each network layer in the at least two network layers according to target size information of the target object in the image verification sample;

[0121] According to the layer verification result and the layer target result corresponding to each network layer, the weight of the layer loss function of each network layer in the model loss function in the next round of model training is adjusted.

[0122] In practical applications, for an input image training sample, the above prediction is actually performed on the feature maps sampled at different network layers by the corresponding detection head. However, the model will limit the prediction of target objects of different sizes on feature maps of different resolutions (such as lesions of different sizes in lesion detection). When a round of training is completed and tested on the validation set (image validation samples and their corresponding sample labels), for a certain image validation sample, each network layer will have several prediction results, some of which may have an IOU (Intersection over Union) > 0 with the annotated gold standard box, which is considered to be a correct detection, and IOU = 0 is a false positive. The size of the false positive prediction result can be used to know which network layer the prediction comes from. After the entire validation set is run, statistics are counted to find out which network layer the predictions of all false positives come from, and the weight of the layer loss function of each of the above network layers in the final loss (i.e., the model loss function) is adjusted accordingly. The more false positive predictions, the higher the weight.

[0123] In specific implementation, each network layer predicts a target object of a different size, that is, each network layer predicts a target object of a corresponding size. Therefore, the number of target objects predicted by each network layer (that is, the layer verification result) can be determined based on the verification size information of the target objects in the image verification sample in each of at least two network layers; at the same time, the number of target objects that should actually be predicted by each network layer in at least two network layers (that is, the layer target result) can also be determined based on the target size information of the target objects in the image verification sample; then, based on the layer verification results and the layer target results, the false positive ratio in the verification results of each network layer can be calculated; finally, based on the false positive ratio in the verification results of each network layer, the weight of the layer loss function of each network layer in the model loss function in the next round of model training is adjusted.

[0124] In another possible implementation, the object detection model can directly output the layer information of the target object in the network layer in the image verification sample when outputting the verification result, such as target object 1 belongs to the verification result predicted by the first network layer, target object 2 belongs to the verification result predicted by the second network layer, etc.; and the sample label corresponding to the image verification sample can also directly carry the layer information of the target object in the network layer in the image verification sample, and the output result of which network layer the target object belongs can be determined based on the layer information. Then, when predicting the false positive ratio in the verification result of each network layer, the false positive ratio in the verification result of each network layer can be directly calculated based on the number of real target objects and predicted target objects in each network layer.

[0125] The object detection model training method provided in the embodiments of this specification verifies the model through image verification samples after each model training iteration, and determines the false positive ratio of the detection head output corresponding to each network layer based on the verification result. Then, according to the false positive ratio predicted by the detection head corresponding to each network layer, the weight of each network layer output in the model loss function is adjusted in the next round of model training, so that the model training pays more attention to the layers with more false positive outputs, and specifically suppresses the generation of false positives during model training and subsequent model applications, thereby alleviating the false positive problem.

[0126] Specifically, the object detection model training method provided in the embodiments of this specification is applied to medical imaging, and a general lesion detection method in medical imaging is proposed. Feature processing is performed on lesions of different sizes (i.e., target objects) through feature extraction networks of different network layers, such as using a shallow layer (such as the first layer) feature extraction network to extract features for smaller lesions, and using a deep layer feature extraction network to extract features for larger lesions. In this way, the model's detection ability for small lesions is improved; at the same time, different numbers of residual networks are added to different deep and shallow network layers for further feature extraction, which can avoid the problem of too many false positives when the model detects small lesions due to the fact that the feature extraction network of the shallow network layer extracts fewer features; at the same time, this method is based on an improved feature pyramid model (i.e., an object detection model), combined with an adaptive weight adjustment strategy during training, which can well reduce the number of false positives and reduce the missed detection of tiny lesions, greatly improving the recognition effect of lung CT images.

[0127] See also Figure 3 , Figure 3 A flow chart of an object detection method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0128] Step 302: Determine a target part image containing a target object.

[0129] Step 304: Input the target part image into the object detection model to obtain the object category, position coordinates and size information of the target object in the target part image.

[0130] Wherein, the object detection model is obtained by training using the above-mentioned object detection model training method.

[0131] Specifically, the actual application scenarios of the object detection method are different, and the target part image containing the target object is also different. For example, when the object detection method is applied to the lung nodule detection scenario of the lung CT image, the target part image containing the target object can be understood as the lung CT image containing the lung nodule; and when the object detection method is applied to the fracture site detection scenario of the rib CT image, the target part image containing the target object can be understood as the rib CT image containing the fracture site.

[0132] For ease of understanding, the embodiments of this specification are described in detail by taking the object detection method applied to the lung nodule detection scenario of the lung CT image, and the target part image containing the target object is understood as a lung CT image containing lung nodules as an example.

[0133] Taking the case where the target part image of the target object is a lung CT image containing lung nodules as an example, determining the target part image containing the target object can be understood as determining the target lung CT image containing the lung nodules.

[0134] In practical applications, in order to improve the processing efficiency of the subsequent object detection model for the target part image, the target part image can be preprocessed before obtaining the target part image. The specific implementation method is as follows:

[0135] The step of determining a target part image containing a target object comprises:

[0136] An initial target part image including a target object is determined, and the initial target part image is preprocessed according to a processing rule corresponding to the initial target part image to obtain a target part image including the target object.

[0137] Different initial images of target parts have different corresponding processing rules. For example, when the initial image of the target part is an initial lung CT image containing lung nodules, the processing rule may be to perform normalization processing on the window width of the lung window at 1500HU and the window level at -450HU. When the initial image of the target part is a rib CT image containing a fracture, the processing rule will be different.

[0138] Taking the case where the initial image of the target part is an initial lung CT image containing lung nodules and the processing rule may be normalization processing at a lung window with a window width of 1500HU and a window position of -450HU, the initial image of the target part containing the target object is determined, and the initial image of the target part is preprocessed according to the processing rule corresponding to the initial image of the target part to obtain the target part image containing the target object; it can be understood that the initial lung CT image containing lung nodules is determined, and according to the processing rule corresponding to the initial lung CT image containing lung nodules, the initial lung CT image containing lung nodules is normalized at a lung window with a window width of 1500HU and a window position of -450HU to obtain the target lung CT image containing lung nodules, so as to exclude the grayscale range of no interest in the initial lung CT image containing lung nodules, so that the contrast characteristics of the lesions to be detected are more obvious.

[0139] Of course, in actual applications, preprocessing of the initial image of the target part not only includes normalization, but also may include data enhancement strategies, random flipping, random scaling, grayscale random perturbation, etc., which are specifically set according to the actual application, and the embodiments of this specification do not impose any limitations on this.

[0140] In order to further improve the processing efficiency of the subsequent object detection model for the target part image, the image can also be segmented by a pre-trained image segmentation model after the image is pre-processed to obtain the region of interest of the image, so that the subsequent object detection model has higher detection accuracy and efficiency for the target object in the image. The specific implementation method is as follows:

[0141] The preprocessing of the target part initial image according to the processing rule corresponding to the target part initial image to obtain the target part image containing the target object includes:

[0142] Preprocessing the target part initial image according to a processing rule corresponding to the target part initial image;

[0143] Input the preprocessed initial image of the target part into the image segmentation model to obtain a labeled reference image;

[0144] According to the annotated reference image, the target part is cropped in the target part initial image to obtain a target part image containing the target object, wherein the image segmentation model is a neural network model.

[0145] Specifically, the image segmentation model can be understood as a pre-trained deep learning neural network model. In the scenario of lung nodule detection in lung CT images, the image segmentation model can be understood as a lung segmentation model, and the training samples and training labels can be understood as the original lung CT images and the lung segmentation mask annotated by the doctor.

[0146] See also Figure 4 , Figure 4 A schematic diagram showing training samples and corresponding training labels of an image segmentation model in an object detection method provided according to an embodiment of the present specification is shown.

[0147] Figure 4 Part a can be understood as the original lung CT image. Figure 4 Part b can be understood as the lung segmentation mask marked by the doctor.

[0148] Still using the above example, the preprocessed initial lung CT image containing lung nodules is input into the lung segmentation model to obtain a lung segmentation mask, and then according to the lung segmentation mask, the lungs are cropped on the preprocessed initial lung CT image containing lung nodules to obtain a target lung CT image containing lung nodules.

[0149] That is, lung segmentation is performed on the given input preprocessed lung CT image, and the size of the output lung segmentation mask is the same as the input preprocessed lung CT image, where label 1 in the lung segmentation mask represents the right lung, label 2 represents the left lung, and 0 represents the non-lung area. According to the lung segmentation result mask, the lung region of interest is determined on the preprocessed lung CT image and the region is cropped out. Then, lung nodule detection and lung lobe segmentation are performed only on the lung CT image in the region of interest through the object detection model.

[0150] For the target object in the target part image predicted by the object detection model, the image block corresponding to the target object can be cut out according to the position coordinates of the object category, and then the image block is input into the pre-trained object classification model to realize the classification process of the target object. The specific implementation method is as follows:

[0151] After obtaining the object category, position coordinates and size information of the target object in the target part image, the method further includes:

[0152] Determining an object image including a target object from the target part image according to the position coordinates and the size information;

[0153] The object image containing the target object is input into an object classification model to obtain a category result of the target object, wherein the object classification model is a neural network model.

[0154] In practical applications, the object classification model can be understood as a pre-trained deep learning neural network model. In the scenario of lung nodule detection in lung CT images, the object classification model can be understood as a lung nodule classification model. The training samples and training labels can be understood as the corresponding positions on the target lung CT images obtained after preprocessing or preprocessing and segmentation, the cropped image blocks corresponding to the lung nodules, and the nodule classifications annotated by the doctor for the lung nodules: solid nodules, semi-solid nodules, ground glass nodules or calcified nodules.

[0155] Take the object classification model as a lung nodule classification model as an example.

[0156] Specifically, according to the size information and position coordinates of the lung nodules, a nodule image containing the lung nodules is cropped from the target part image; then the nodule image is input into the lung nodule classification model to obtain the category result of the lung nodule, thereby realizing the classification of the lung nodule.

[0157] At the same time, when the target part image is a lung CT image and the target object is a lung nodule, in order to further ensure the accuracy of lung nodule recognition, a lung lobe segmentation model can be pre-trained to perform lung lobe segmentation based on the input lung region of interest image and output lung lobe segmentation mask. Subsequently, the false positive prediction results can be filtered based on the lung lobe segmentation mask combined with the prediction results of the object detection model, so as to further ensure the detection accuracy of lung nodules and improve the recognition effect of lung CT images. The specific implementation method is as follows:

[0158] The target part image is a lung CT image, and the target object is a lung nodule;

[0159] Accordingly, after obtaining the category result of the target object, the method further includes:

[0160] Inputting the lung CT image containing the lung nodules into the lung lobe segmentation model to obtain a lung lobe annotated reference image;

[0161] According to the lung lobe annotated reference image and the position coordinates of the lung nodules in the lung CT image, the lung nodules in the lung CT image are screened to obtain the screened lung nodules in the lung CT image, wherein the lung lobe segmentation model is a neural network model.

[0162] Specifically, the lobe segmentation model can be understood as a pre-trained deep learning neural network model. In the scenario of lung nodule detection in lung CT images, the training samples and training labels of the lobe segmentation model can be understood as the lung CT image of interest (i.e., the cropped lung CT image of interest after preprocessing and lung segmentation) and the lobe segmentation mask annotated by the doctor.

[0163] In practical applications, a lung CT image containing lung nodules is input into a lung lobe segmentation model to obtain a lung lobe annotation reference image, i.e., a lung lobe segmentation mask. The size of the lung lobe segmentation mask is the same as the size of the lung CT image containing lung nodules, and labels 1, 2, and 3 in the lung lobe segmentation mask represent the right upper lobe, right middle lobe, and right lower lobe, respectively; labels 4 and 5 represent the left upper lobe and left lower lobe, respectively; and 0 represents a non-lung area. Then, according to the lung lobe segmentation mask and the position coordinates of the lung nodules in the lung CT image, the lung nodules in the lung CT image are screened to obtain the screened lung nodules in the lung CT image.

[0164] That is, after obtaining the object category, position coordinates and size information of the lung nodules in the lung CT image output by the object detection model, the lung nodules are located using the lung lobe segmentation result, i.e., the lung lobe segmentation mask, and each lung nodule is determined one by one on which lung lobe it is located. Lung nodules that are not in any lung lobe are filtered as false positives to ensure the recognition accuracy of lung nodules in lung CT images and improve the recognition effect of lung CT images.

[0165] After obtaining the pulmonary nodules in the screened pulmonary CT images, the target recognition result of the pulmonary nodules in the screened pulmonary CT images can be determined according to the object category, position coordinates and size information of the pulmonary nodules in the pulmonary CT images output by the object detection model and the classification of the pulmonary nodules output by the object classification model. The specific implementation method is as follows:

[0166] After obtaining the pulmonary nodules in the screened pulmonary CT image, the method further includes:

[0167] The target recognition result of the lung nodules in the screened lung CT image is determined based on the object category, position coordinates and size information of the lung nodules in the lung CT image and the category result of the lung nodules.

[0168] Among them, the target recognition results of lung nodules in the screened lung CT images can be understood as the coordinates, size, category, confidence, position in the lungs and other information of each lung nodule on the lung CT image.

[0169] The above information can then be used to visualize the results on the input lung CT image.

[0170] See also Figure 5 , Figure 5 A schematic diagram showing a method for detecting an object according to an embodiment of the present specification for visually displaying the results on a lung CT image is shown.

[0171] Figure 5 What is shown in FIG. 1 is a schematic diagram for visually displaying the detection results on the input lung CT image, that is, each lung nodule is marked on the input lung CT image, as well as the position coordinates, size information, and category information of each lung nodule.

[0172] In a specific implementation, the object detection method is applied to the detection scenario of the fracture position in the rib CT image, the target part image is the rib CT image, and the target object is the fracture position. At the same time, the lung segmentation model and the lung lobe segmentation model are not used in the fracture detection scenario. That is, when the object detection method is applied to different scenarios, the models used will be added, deleted or otherwise adjusted accordingly according to the usage scenarios.

[0173] The object detection method provided in the embodiments of this specification can quickly and accurately detect the target object in the target part image through a pre-trained object detection model, thereby improving the detection capability of small-sized target objects, and specifically suppressing the generation of false positives, thereby improving the recognition effect of the target part image; at the same time, when the object detection method is applied to the lung nodule recognition scenario in the lung CT image, it is combined with the lung segmentation module, the classification module (object classification module), the lung lobe segmentation module, etc. to achieve efficient and accurate recognition of lung nodules in the lung CT image, and give accurate recognition results.

[0174] The following combination Figure 6 Taking the application of the object detection method provided in this specification to the lung nodule detection scenario of lung CT images as an example, the object detection method is further described. Figure 6 A processing flow chart of an object detection method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0175] Step 602: preprocessing the lung CT image containing the lung nodules to obtain a preprocessed target lung CT image.

[0176] Step 604: Input the target lung CT image into the lung segmentation model to obtain a lung segmentation mask, determine a lung region of interest on the target lung CT image according to the lung segmentation mask, and crop the lung region of interest.

[0177] Step 606: Input the lung region of interest into an object detection model, namely, a lung nodule detection model, to obtain object categories, position coordinates, and size information of all lung nodules in the lung region of interest.

[0178] Step 608: Based on the object categories and position coordinates of all lung nodules in the lung region of interest, an image block corresponding to each lung nodule is cropped from the lung region of interest, and each image block is input into an object classification model, i.e., a lung nodule classification model, to obtain a category corresponding to each lung nodule.

[0179] Step 610: Input the lung region of interest into the lung lobe segmentation model to obtain a lung lobe segmentation mask.

[0180] Step 612: Post-processing is performed based on the object category, position coordinates and size information of all lung nodules in the lung region of interest, the category corresponding to each lung nodule, and the lung lobe segmentation mask to obtain the target recognition result of all lung nodules in the lung region of interest.

[0181] Step 614: Visualize the target recognition results of all lung nodules in the lung region of interest.

[0182] Among them, the post-processing in the embodiments of this specification can be understood as the process of lung nodule screening and target recognition result determination in the above embodiments, which will not be repeated here.

[0183] The object detection method provided in the embodiments of this specification, when applied to the lung nodule recognition scenario in lung CT images, is combined with a lung segmentation module, a classification module (object classification module), a lung lobe segmentation module, etc. to achieve efficient and accurate recognition of lung nodules in lung CT images and provide accurate recognition results.

[0184] Corresponding to the above method embodiment, this specification also provides an object detection model training device embodiment, Figure 7 FIG. 2 shows a schematic diagram of the structure of an object detection model training device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises:

[0185] A first feature processing module 702 is configured to input the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing, and obtain the initial image features of the image training sample in the target network layer, wherein the target network layer is any one of the at least two network layers;

[0186] The second feature processing module 704 is configured to perform feature processing on the initial image features through the residual network corresponding to the target network layer to obtain the target image features of the image training sample in the target network layer, wherein the larger the number of layers of the target network layer, the smaller the number of corresponding residual networks;

[0187] The prediction module 706 is configured to input the target image features into the detection head corresponding to the target network layer for prediction, and obtain the prediction result of the image training sample at the target network layer;

[0188] The training module 708 is configured to train the object detection model according to the prediction results of the image training samples in the at least two network layers and the sample labels corresponding to the image training samples.

[0189] Optionally, the prediction module 706 is further configured to:

[0190] Determine a target image feature of the image training sample at a next network layer of the target network layer;

[0191] The target image features of the image training samples at the target network layer and the target image features of the image training samples at the next network layer of the target network layer are fused, and the results are input into the detection head corresponding to the target network layer for prediction to obtain the prediction results of the image training samples at the target network layer.

[0192] Optionally, the prediction result includes the predicted object category, predicted coordinate position and predicted size information of the target object in the image training sample at the target network layer, and the sample label corresponding to the image training sample is the target object category, target coordinate position and target size information of the target object in the image training sample;

[0193] Accordingly, the training module 708 is further configured to:

[0194] Determine the object category loss function, coordinate position loss function and size information loss function of the target network layer according to the predicted object category, predicted coordinate position and predicted size information of the target object in the image training sample at the target network layer, as well as the target object category, the target coordinate position and the target size information;

[0195] The size information loss function is weighted according to the inverse of the predicted size information as a weight to obtain a weighted size information loss function;

[0196] Determining a layer loss function of the target network layer according to the weighted size information loss function, the coordinate position loss function and the object category loss function;

[0197] A model loss function is determined according to the layer loss functions of the at least two network layers, and the object detection model is trained according to the model loss function.

[0198] Optionally, the training module 708 is further configured to:

[0199] When it is determined that the difference between the model loss function and the model loss function corresponding to the previous round of model training is greater than or equal to a preset threshold, adjusting the network parameters of the object detection model according to the model loss function to obtain an updated object detection model;

[0200] Inputting the image verification sample into the updated object detection model to obtain a verification result;

[0201] According to the verification result and the sample label corresponding to the image verification sample, adjust the weights of the layer loss functions of the at least two network layers in the next round of model training in the model loss function, and perform the next round of model training;

[0202] Until the preset model training end conditions are met, the trained object detection model is obtained.

[0203] Optionally, the verification result includes verification size information of the target object in the image verification sample at the at least two network layers, and the sample label corresponding to the image verification sample is the target size information of the target object in the image verification sample;

[0204] Accordingly, the training module 708 is further configured to:

[0205] Determine a layer verification result corresponding to each network layer according to verification size information of the target object in the image verification sample at the at least two network layers;

[0206] Determining a layer target result corresponding to each network layer in the at least two network layers according to target size information of the target object in the image verification sample;

[0207] According to the layer verification result and the layer target result corresponding to each network layer, the weight of the layer loss function of each network layer in the model loss function in the next round of model training is adjusted.

[0208] Optionally, the first feature processing module 702 is further configured to:

[0209] Inputting the image training sample into the first feature extraction network of at least two network layers of the object detection model for feature processing, and obtaining the downsampling result of the image training sample in the first feature extraction network of the target network layer;

[0210] The down-sampling result is processed by the second feature extraction network of the at least two network layers to obtain an up-sampling result of the image training sample in the second feature extraction network of the target network layer;

[0211] According to the down-sampling result and the up-sampling result, initial image features of the image training sample at the target network layer are obtained.

[0212] Optionally, the image training samples are image training samples of target parts, and the object detection model is a neural network model.

[0213] The object detection model training device provided in the embodiments of the present specification performs feature processing on target objects of different sizes through feature extraction networks of different network layers. For example, a shallow feature extraction network (such as the first layer) is used to extract features for target objects of smaller sizes, and a deep feature extraction network is used to extract features for target objects of larger sizes. In this way, the model's detection capability for target objects of small sizes is improved. At the same time, different numbers of residual networks are added to different deep and shallow network layers for further feature extraction. This can avoid the problem of too many false positives when the model detects target objects of small sizes due to the fact that the feature extraction network of the shallow network layer extracts fewer features.

[0214] The above is a schematic scheme of an object detection model training device of this embodiment. It should be noted that the technical scheme of the object detection model training device and the technical scheme of the object detection model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the object detection model training device can be found in the description of the technical scheme of the object detection model training method described above.

[0215] Corresponding to the above method embodiment, this specification also provides an object detection device embodiment, Figure 8 FIG. 2 shows a schematic diagram of the structure of an object detection device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises:

[0216] An image determination module 802 is configured to determine a target part image containing a target object;

[0217] The object detection module 804 is configured to input the target part image into an object detection model to obtain the object category, position coordinates and size information of the target object in the target part image.

[0218] Wherein, the object detection model is obtained by training using the above-mentioned object detection model training method.

[0219] Optionally, the image determination module 802 is further configured to:

[0220] An initial target part image including a target object is determined, and the initial target part image is preprocessed according to a processing rule corresponding to the initial target part image to obtain a target part image including the target object.

[0221] Optionally, the image determination module 802 is further configured to:

[0222] Preprocessing the target part initial image according to a processing rule corresponding to the target part initial image;

[0223] Input the preprocessed initial image of the target part into the image segmentation model to obtain a labeled reference image;

[0224] According to the annotated reference image, the target part is cropped in the target part initial image to obtain a target part image containing the target object, wherein the image segmentation model is a neural network model.

[0225] Optionally, the device further comprises:

[0226] The category determination module is configured to:

[0227] Determining an object image including a target object from the target part image according to the position coordinates and the size information;

[0228] The object image containing the target object is input into an object classification model to obtain a category result of the target object, wherein the object classification model is a neural network model.

[0229] Optionally, the target part image is a lung CT image, and the target object is a lung nodule;

[0230] Accordingly, the device further comprises:

[0231] The lung lobe segmentation module is configured as follows:

[0232] Inputting the lung CT image containing the lung nodules into the lung lobe segmentation model to obtain a lung lobe annotated reference image;

[0233] According to the lung lobe annotated reference image and the position coordinates of the lung nodules in the lung CT image, the lung nodules in the lung CT image are screened to obtain the screened lung nodules in the lung CT image, wherein the lung lobe segmentation model is a neural network model.

[0234] Optionally, the device further comprises:

[0235] The result recognition module is configured as follows:

[0236] The target recognition result of the lung nodules in the screened lung CT image is determined based on the object category, position coordinates and size information of the lung nodules in the lung CT image and the category result of the lung nodules.

[0237] Optionally, the target part image is a rib CT image, and the target object is a fracture location.

[0238] The object detection device provided in the embodiments of this specification can quickly and accurately detect the target object in the target part image through a pre-trained object detection model, thereby improving the detection capability of small-sized target objects, and specifically suppressing the generation of false positives, thereby improving the recognition effect of the target part image; at the same time, when the object detection method is applied to the lung nodule recognition scenario in the lung CT image, it is combined with the lung segmentation module, the classification module (object classification module), the lung lobe segmentation module, etc. to achieve efficient and accurate recognition of the lung nodules in the lung CT image, and give accurate recognition results.

[0239] The above is a schematic scheme of an object detection model training device of this embodiment. It should be noted that the technical scheme of the object detection model training device and the technical scheme of the object detection model training method described above belong to the same concept, and the details not described in detail in the technical scheme of the object detection model training device can be found in the description of the technical scheme of the object detection model training method described above.

[0240] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0241] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0242] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0243] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0244] Among them, the processor 920 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned object detection model training method or object detection method.

[0245] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the object detection model training method or the object detection method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the object detection model training method or the object detection method mentioned above.

[0246] An embodiment of the present specification also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned object detection model training method or object detection method.

[0247] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the object detection model training method or the object detection method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the object detection model training method or the object detection method mentioned above.

[0248] An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned object detection model training method or object detection method.

[0249] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the technical scheme of the computer program and the technical scheme of the object detection model training method or the object detection method described above are of the same concept, and the details not described in detail in the technical scheme of the computer program can be found in the description of the technical scheme of the object detection model training method or the object detection method described above.

[0250] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0251] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0252] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0253] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0254] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for training an object detection model, comprising: Inputting the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training sample in the target network layer, wherein the target network layer is any one of the at least two network layers; Processing the initial image features through the residual network corresponding to the target network layer to obtain target image features of the image training sample in the target network layer, wherein different network layers in the at least two network layers correspond to different numbers of residual networks, and the number of residual networks decreases as the level of the network layer goes deeper, for extracting features; Inputting the target image features into the detection head corresponding to the target network layer for prediction, and obtaining the prediction result of the image training sample in the target network layer; The object detection model is trained according to the prediction results of the image training samples in the at least two network layers and the sample labels corresponding to the image training samples.

2. According to the object detection model training method of claim 1, the step of inputting the target image features into the detection head corresponding to the target network layer for prediction to obtain the prediction result of the image training sample at the target network layer comprises: Determine a target image feature of the image training sample at a next network layer of the target network layer; The target image features of the image training samples at the target network layer and the target image features of the image training samples at the next network layer of the target network layer are fused, and the results are input into the detection head corresponding to the target network layer for prediction to obtain the prediction results of the image training samples at the target network layer.

3. According to the object detection model training method of claim 1, the prediction result includes the predicted object category, predicted coordinate position and predicted size information of the target object in the image training sample at the target network layer, and the sample label corresponding to the image training sample is the target object category, target coordinate position and target size information of the target object in the image training sample; Accordingly, training the object detection model according to the prediction results of the image training samples at the at least two network layers and the sample labels corresponding to the image training samples includes: Determine the object category loss function, coordinate position loss function and size information loss function of the target network layer according to the predicted object category, predicted coordinate position and predicted size information of the target object in the image training sample at the target network layer, as well as the target object category, the target coordinate position and the target size information; The size information loss function is weighted according to the inverse of the predicted size information as a weight to obtain a weighted size information loss function; Determining a layer loss function of the target network layer according to the weighted size information loss function, the coordinate position loss function and the object category loss function; A model loss function is determined according to the layer loss functions of the at least two network layers, and the object detection model is trained according to the model loss function.

4. The object detection model training method according to claim 3, wherein training the object detection model according to the model loss function comprises: When it is determined that the difference between the model loss function and the model loss function corresponding to the previous round of model training is greater than or equal to a preset threshold, adjusting the network parameters of the object detection model according to the model loss function to obtain an updated object detection model; Inputting the image verification sample into the updated object detection model to obtain a verification result; According to the verification result and the sample label corresponding to the image verification sample, adjust the weights of the layer loss functions of the at least two network layers in the next round of model training in the model loss function, and perform the next round of model training; Until the preset model training end conditions are met, the trained object detection model is obtained.

5. The object detection model training method according to claim 4, wherein the verification result includes verification size information of the target object in the image verification sample at the at least two network layers, and the sample label corresponding to the image verification sample is the target size information of the target object in the image verification sample; Accordingly, adjusting the weights of the layer loss functions of the at least two network layers in the next round of model training in the model loss function according to the verification result and the sample label corresponding to the image verification sample includes: Determine a layer verification result corresponding to each network layer according to verification size information of the target object in the image verification sample at the at least two network layers; Determining a layer target result corresponding to each network layer in the at least two network layers according to target size information of the target object in the image verification sample; According to the layer verification result and the layer target result corresponding to each network layer, the weight of the layer loss function of each network layer in the model loss function in the next round of model training is adjusted.

6. The object detection model training method according to claim 1, wherein the step of inputting the image training sample into the feature extraction network of at least two network layers of the object detection model for feature processing to obtain the initial image features of the image training sample in the target network layer comprises: Inputting the image training sample into the first feature extraction network of at least two network layers of the object detection model for feature processing, and obtaining the downsampling result of the image training sample in the first feature extraction network of the target network layer; The down-sampling result is processed by the second feature extraction network of the at least two network layers to obtain an up-sampling result of the image training sample in the second feature extraction network of the target network layer; According to the down-sampling result and the up-sampling result, initial image features of the image training sample at the target network layer are obtained.

7. According to the object detection model training method according to any one of claims 1 to 6, the image training samples are image training samples of the target part, and the object detection model is a neural network model.

8. An object detection method, comprising: determining a target part image containing a target object; Input the target part image into the object detection model to obtain the object category, position coordinates and size information of the target object in the target part image, Wherein, the object detection model is obtained by training through the object detection model training method described in any one of claims 1-7.

9. The object detection method according to claim 8, wherein determining the target part image containing the target object comprises: An initial target part image including a target object is determined, and the initial target part image is preprocessed according to a processing rule corresponding to the initial target part image to obtain a target part image including the target object.

10. The object detection method according to claim 9, wherein the step of preprocessing the target part initial image according to a processing rule corresponding to the target part initial image to obtain a target part image containing the target object comprises: Preprocessing the target part initial image according to a processing rule corresponding to the target part initial image; Input the preprocessed initial image of the target part into the image segmentation model to obtain a labeled reference image; According to the annotated reference image, the target part is cropped in the target part initial image to obtain a target part image containing the target object, wherein the image segmentation model is a neural network model.

11. The object detection method according to claim 10, after obtaining the object category, position coordinates and size information of the target object in the target part image, further comprising: Determining an object image including a target object from the target part image according to the position coordinates and the size information; The object image containing the target object is input into an object classification model to obtain a category result of the target object, wherein the object classification model is a neural network model.

12. The object detection method according to claim 11, wherein the target part image is a lung CT image, and the target object is a lung nodule; Accordingly, after obtaining the category result of the target object, the method further includes: Inputting the lung CT image containing the lung nodules into the lung lobe segmentation model to obtain a lung lobe annotated reference image; According to the lung lobe annotated reference image and the position coordinates of the lung nodules in the lung CT image, the lung nodules in the lung CT image are screened to obtain the screened lung nodules in the lung CT image, wherein the lung lobe segmentation model is a neural network model.

13. The object detection method according to claim 12, after obtaining the pulmonary nodules in the screened pulmonary CT image, further comprising: The target recognition result of the lung nodules in the screened lung CT image is determined based on the object category, position coordinates and size information of the lung nodules in the lung CT image and the category result of the lung nodules.

14. The object detection method according to any one of claims 8 to 13, wherein the target part image is a rib CT image, and the target object is a fracture position.