Object detection methods and related equipment based on pre-trained networks

Abstract

This application relates to the fields of artificial intelligence and digital healthcare, and provides a target detection method and related equipment based on pre-trained networks. The method includes: acquiring a trained first target detection model, which includes multiple pre-trained network layers; then quantizing each pre-trained network layer of the first target detection model to reduce the model parameters from initial accuracy to target accuracy; using the quantized first target detection model as a second target detection model; acquiring a target image; inputting the target image into the second target detection model to obtain a prediction result for the target image. The embodiments of this application can improve model inference speed and reduce hardware resource consumption while ensuring target detection accuracy.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and digital healthcare, and in particular to a target detection method and related equipment based on a pre-trained network. Background Technology

[0002] Object detection is a widely used and important branch of computer vision. It can be simply viewed as a combination of image classification and regression localization, identifying objects of interest within an input image and providing classification results for those objects.

[0003] Unlike other computer vision technologies, object detection in real-world applications is often used on low- to mid-range devices with limited hardware performance and requires high real-time performance. For example, medical image processing is an indispensable part of medical auxiliary diagnosis. Accurate, efficient, and rapid processing and analysis of medical images are crucial for disease diagnosis and treatment. To improve doctors' work efficiency, the industry has begun to try deploying pre-trained object detection models on terminal devices closer to the data source to meet the high real-time requirements of medical auxiliary diagnosis. However, this approach faces challenges such as slow inference speeds for large pre-trained models due to the limited hardware performance of terminal devices.

[0004] Therefore, for object detection methods, how to improve model inference speed and reduce hardware resource consumption while ensuring accuracy has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to propose a target detection method, apparatus, electronic device, and computer-readable storage medium based on a pre-trained network, which can improve model inference speed and reduce hardware resource consumption while ensuring accuracy.

[0006] To achieve the above objectives, a first aspect of this application proposes a target detection method based on a pre-trained network, the method comprising:

[0007] Obtain a trained first object detection model, which includes multiple pre-trained network layers;

[0008] Each pre-trained network layer of the first target detection model is quantized, so that the model parameters of each pre-trained network layer in the first target detection model are reduced from the initial accuracy to the target accuracy, and the quantized first target detection model is used as the second target detection model.

[0009] Acquire the image to be detected;

[0010] The image to be detected is input into the second target detection model to obtain a prediction result of the image to be detected through the second target detection model. The prediction result is used to indicate whether the image to be detected includes a preset detection target.

[0011] According to some embodiments of the object detection method based on a pre-trained network provided by the present invention, the quantization processing of each pre-trained network layer of the first object detection model includes:

[0012] Obtain image data samples and perform the following processing on each of the pre-trained network layers:

[0013] The image data sample is input into the first target detection model to obtain the output accuracy of the last pre-trained network layer in the first target detection model;

[0014] If the output accuracy is greater than a preset threshold, the model parameters of the current pre-trained network are reduced from the initial accuracy to the target accuracy, and the target accuracy is used as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing. This process is repeated until the output accuracy is less than the preset threshold.

[0015] According to some embodiments of the present invention, the object detection method based on a pre-trained network further includes a classifier corresponding to each of the pre-trained network layers, wherein the classifier is used to predict a probability distribution based on the feature information output by the pre-trained network layer.

[0016] The classifier is divided into a first classifier corresponding to the last pre-trained network layer and a second classifier other than the first classifier. The second classifier is obtained by distillation of the first classifier.

[0017] The quantization process for each pre-trained network layer of the first object detection model includes:

[0018] Obtain image data samples and perform the following processing on each of the pre-trained network layers:

[0019] The image data sample is input into the first target detection model to obtain the first probability distribution output by the first classifier and the second probability distribution output by the second classifier.

[0020] Determine the distribution difference between the first probability distribution and the second probability distribution;

[0021] If the distribution difference is less than a preset threshold, the model parameters of the current pre-trained network layer are reduced from the initial accuracy to the target accuracy, and the target accuracy is used as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing. This process is repeated until the distribution difference is greater than the preset threshold.

[0022] According to some embodiments of the present invention, the object detection method based on a pre-trained network includes the following training steps for the first object detection model:

[0023] Obtain the image sample training set;

[0024] With the weight parameters of the second classifier fixed, the image sample training set is input into the first target detection model to update the weight parameters of the pre-trained network layer and the first classifier.

[0025] With the pre-trained network layers and the weight parameters of the first classifier fixed, the trained first classifier is used as a teacher model to distill the second classifier to update the weight parameters of the second classifier.

[0026] According to some embodiments of the object detection method based on a pre-trained network provided by the present invention, during the process of the second object detection model predicting the image to be detected:

[0027] The information entropy corresponding to the image to be detected is obtained sequentially through the second classifier;

[0028] If the information entropy obtained by the second classifier is less than a preset threshold, then exit the current pre-trained network layer; or,

[0029] If the information entropy obtained by the second classifier is greater than a preset threshold, then feature extraction continues through the pre-trained network layer.

[0030] According to some embodiments of the target detection method based on a pre-trained network provided by the present invention, the information entropy is determined by the following formula:

[0031]

[0032] Wherein, H(x) is the information entropy corresponding to the image to be detected x, n is the number of classification categories, and P... i (x) represents the probability distribution corresponding to the image x to be detected.

[0033] According to some embodiments of the present invention, the object detection method based on a pre-trained network, after obtaining the trained first object detection model and before quantizing each of the pre-trained network layers of the first object detection model, further includes:

[0034] Obtain image sample fine-tuning set;

[0035] The image sample fine-tuning set is input into the first target detection model to obtain the corresponding training prediction result through the first target detection model;

[0036] The loss value is determined based on the preset target detection labels in the image sample fine-tuning set, the training prediction results, and the preset loss function;

[0037] Based on the loss value, the model parameters of the first target detection model are fine-tuned to obtain the fine-tuned first target detection model.

[0038] To achieve the above objectives, a second aspect of this application provides a target detection device based on a pre-trained network, the device comprising:

[0039] The first acquisition module is used to acquire a trained first object detection model, which includes multiple pre-trained network layers.

[0040] The model quantization module is used to quantize each of the pre-trained network layers of the first target detection model, so that the model parameters of each of the pre-trained network layers in the first target detection model are reduced from the initial accuracy to the target accuracy, and the quantized first target detection model is used as the second target detection model.

[0041] The second acquisition module is used to acquire the image to be detected;

[0042] The target detection module is used to input the image to be detected into the second target detection model to obtain a prediction result of the image to be detected through the second target detection model. The prediction result is used to indicate whether the image to be detected includes a preset detection target.

[0043] To achieve the above objectives, a third aspect of this application provides an electronic device, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method described in the first aspect.

[0044] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium for computer-readable storage, wherein the storage medium stores one or more computer programs that can be executed by one or more processors to implement the method described in the first aspect.

[0045] This application proposes a target detection method, apparatus, electronic device, and computer-readable storage medium based on a pre-trained network. The pre-trained network-based target detection method acquires a trained first target detection model, which includes multiple pre-trained network layers. Each pre-trained network layer of the first target detection model is then quantized, reducing the model parameters from initial precision to a target precision. The quantized first target detection model is then used as a second target detection model. After quantization, a target image is acquired and input into the second target detection model to obtain a prediction result indicating whether the target image contains a preset target. This application utilizes a target detection model built on a pre-trained network for target detection. Furthermore, by quantizing the target detection model by reducing the precision of each pre-trained network layer before inference, it can improve model inference speed and reduce hardware resource consumption while maintaining target detection accuracy.

[0046] The object detection method based on pre-trained networks proposed in this application can be applied to the field of digital healthcare. In a medical application scenario, a first pre-trained object detection model is first obtained, wherein the first object detection model includes multiple pre-trained network layers. Then, each pre-trained network layer of the first object detection model is quantized, so that the model parameters of each pre-trained network layer in the first object detection model are reduced from the initial accuracy to the target accuracy. The quantized first object detection model is then used as a second object detection model. After quantization, the medical image to be detected is acquired and input into the second target detection model. The second target detection model obtains a prediction result indicating whether the medical image to be detected contains a preset detection target. In the field of digital healthcare, this application embodiment uses a target detection model built based on a pre-trained network to perform medical-related target detection. Before model inference, the target detection model is quantized by reducing the precision representation of each pre-trained network layer in the target detection model. This allows terminal devices that are close to the medical image data source to perform target detection using the pre-trained target detection model to complete the medical target detection task with faster inference speed and lower hardware resource consumption while ensuring target detection accuracy, thus meeting the high real-time requirements of medical auxiliary diagnosis. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a target detection method based on a pre-trained network provided in an embodiment of this application;

[0048] Figure 2This is a flowchart illustrating a target detection method based on a pre-trained network, provided in another embodiment of this application.

[0049] Figure 3 This is a flowchart illustrating a target detection method based on a pre-trained network, provided in another embodiment of this application.

[0050] Figure 4 This is a flowchart illustrating a target detection method based on a pre-trained network, provided in another embodiment of this application.

[0051] Figure 5 This is a flowchart illustrating a target detection method based on a pre-trained network, provided in another embodiment of this application.

[0052] Figure 6 This is a flowchart illustrating a target detection method based on a pre-trained network, provided in another embodiment of this application.

[0053] Figure 7 This is a schematic diagram of a target detection model structure provided in an embodiment of this application;

[0054] Figure 8 This is a schematic diagram illustrating the accuracy of a pre-trained network layer provided in an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the structure of a target detection device based on a pre-trained network provided in an embodiment of this application;

[0056] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] It should be noted that, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0059] Object detection is a widely used and important branch of computer vision. It can be simply viewed as a combination of image classification and regression localization, identifying objects of interest within an input image and providing classification results for those objects.

[0060] Unlike other computer vision technologies, object detection in real-world applications is often used on low- to mid-range devices with limited hardware performance and requires high real-time performance. For example, medical image processing is an indispensable part of medical auxiliary diagnosis. Accurate, efficient, and rapid processing and analysis of medical images are crucial for disease diagnosis and treatment. To improve doctors' work efficiency, the industry has begun to try deploying pre-trained object detection models on terminal devices closer to the data source to meet the high real-time requirements of medical auxiliary diagnosis. However, this approach faces challenges such as slow inference speeds for large pre-trained models due to the limited hardware performance of terminal devices.

[0061] Therefore, for object detection methods, how to improve model inference speed and reduce hardware resource consumption while ensuring accuracy has become an urgent technical problem to be solved.

[0062] Based on this, embodiments of this application provide a target detection method, apparatus, electronic device, and computer-readable storage medium based on a pre-trained network, which can improve model inference speed and reduce hardware resource consumption while ensuring accuracy.

[0063] This application provides a target detection method, apparatus, electronic device, and computer-readable storage medium based on a pre-trained network, which are specifically described through the following embodiments. First, the target detection method based on a pre-trained network in this application is described.

[0064] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0065] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0066] The target detection method based on pre-trained networks provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the target detection method based on pre-trained networks, etc., but is not limited to the above forms.

[0067] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0068] Please see Figure 1 , Figure 1 This document illustrates a flowchart of a target detection method based on a pre-trained network, as provided in an embodiment of this application. Figure 1 As shown, the object detection method based on the pre-trained network includes, but is not limited to, steps S110 to S140:

[0069] Step S110: Obtain the trained first object detection model, which includes multiple pre-trained network layers;

[0070] Step S120: Quantize each of the pre-trained network layers of the first object detection model, so that the model parameters of each of the pre-trained network layers in the first object detection model are reduced from the initial accuracy to the target accuracy, and the quantized first object detection model is used as the second object detection model.

[0071] Step S130: Obtain the image to be detected;

[0072] Step S140: Input the image to be detected into the second target detection model to obtain the prediction result of the image to be detected through the second target detection model. The prediction result is used to indicate whether the image to be detected includes a preset detection target.

[0073] Understandably, the first object detection model is built based on a pre-trained network; for example, see [link to example]. Figure 6 , Figure 6 This illustration shows a schematic diagram of a target detection model structure provided in an embodiment of this application, such as... Figure 6 As shown, the first object detection model is built based on a pre-trained network. The backbone network in the first object detection model includes an embedding layer, a pre-trained network layer (Transformer layer) consisting of stacked Transformer blocks, and a classifier.

[0074] In some embodiments, see Figure 6 , Figure 6 This illustration shows a flowchart of a target detection method based on a pre-trained model according to another embodiment of this application. Figure 6 As shown, after obtaining the trained first object detection model, and before quantizing each of the pre-trained network layers of the first object detection model, the method further includes:

[0075] Step S610: Obtain the image sample fine-tuning set;

[0076] Step S620: Input the image samples in the image sample fine-tuning set into the first target detection model to obtain the corresponding training prediction results through the first target detection model;

[0077] Step S630: Determine the loss value based on the preset target detection labels in the image sample fine-tuning set, the training prediction results, and the preset loss function;

[0078] Step S640: Based on the loss value, fine-tune the model parameters of the first target detection model to obtain the fine-tuned first target detection model.

[0079] It should be understood that, in order to avoid the problem of model performance degradation due to the difference between the pre-training task and the target task of the first object detection model in actual application, after obtaining the trained first object detection model, the model parameters of the first object detection model are fine-tuned using the image sample fine-tuning set of the target task so that the model parameters are suitable for the target task.

[0080] It should be noted that when the image to be detected is input into the second target detection model, if it is determined that the image to be detected includes a preset target, the second target detection model outputs the target included in the image to be detected and the location information corresponding to the target.

[0081] It should be understood that the image samples in the image sample fine-tuning set are labeled samples with corresponding target detection labels. The training prediction results corresponding to each image sample in the image sample fine-tuning set are obtained through the first target detection model. Based on the preset loss function, the loss value is determined according to the target detection label and the training prediction results. Finally, the model parameters of the first target detection model are updated based on the loss value, that is, the model is fine-tuned to make the first target detection model more suitable for the current target detection task.

[0082] It should be noted that the image to be detected is input into the second object detection model to obtain the prediction result of the image to be detected. The prediction result includes the classification category and location information of the detected object in the image to be detected.

[0083] In one specific embodiment, each of the pre-trained network layers of the first object detection model is quantized, and the quantization process further includes unifying the precision representation of the model parameters of each pre-trained network layer, such that the model parameters of each pre-trained network layer use the same precision representation.

[0084] For example, the effective accuracy of the model parameters in a pre-trained network is not consistent; see [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram illustrating the accuracy of a pre-trained network layer according to an embodiment of this application, such as... Figure 8 As shown, the accuracy in the third pre-trained network of the first object detection model is represented by 3 weights and 5 activations, the accuracy in the fourth pre-trained network is represented by 6 weights and 7 activations, the accuracy in the fifth pre-trained network is represented by 4 weights and 6 activations, and the accuracy in the sixth pre-trained network is represented by 5 weights and 6 activations.

[0085] Therefore, for the quantization processing of the first object detection model, if all pre-trained network layers adopt a uniform precision, such as directly reducing from 32-bit floating-point numbers to 8-bit integers, it may lead to a significant loss of precision in the weight parameters. If mixed-precision quantization is used, meaning each pre-trained network layer uses different precision representations based on the effective precision of the weight parameters, hardware acceleration cannot be utilized due to inconsistent data types and data representation lengths, thus slowing down the model's inference speed. Therefore, by unifying the precision representation of each pre-trained network layer, the same layer uses a uniform precision representation, while different pre-trained network layers use different precision representations after quantization. This means that different layers in the pre-trained network use mixed precision, while the same layer uses a uniform precision representation. By reducing the precision representation of the pre-trained network layers while unifying the precision representation of each layer, the computational cost of model parameters can be reduced, and hardware acceleration can be utilized to further improve the model's inference speed.

[0086] It should be understood that reducing the precision of the pre-trained network layer means reducing 32-bit floating-point precision to 16-bit floating-point precision, reducing 16-bit floating-point precision to 8-bit integer precision, reducing 8-bit integer precision to 4-bit integer precision, reducing 32-bit floating-point precision to 8-bit integer precision, etc. The embodiments of this application do not impose specific limitations here.

[0087] In some embodiments, see Figure 2 , Figure 2 This illustration shows a flowchart of a target detection method based on a pre-trained network according to another embodiment of this application, as shown below. Figure 2 As shown, the quantization process for each pre-trained network layer of the first object detection model includes:

[0088] Step S210: Obtain image data samples and perform the following processing on each of the pre-trained network layers:

[0089] Step S220: Input the image data sample into the first target detection model to obtain the output accuracy of the last pre-trained network layer in the first target detection model;

[0090] Step S230: If the output accuracy is greater than a preset threshold, reduce the model parameters of the current pre-trained network from the initial accuracy to the target accuracy, and use the target accuracy as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing, and repeat until the output accuracy is less than the preset threshold.

[0091] It should be understood that, for each pre-trained network layer, by inputting image data samples into the first object detection model, the output accuracy of the last pre-trained network layer in the first object detection model is obtained. If the output accuracy is greater than a preset threshold, the model parameters of the pre-trained network are reduced from the original 32-bit floating-point number until the output accuracy is less than the preset threshold. This step is then repeated layer by layer to complete the quantization process of the entire pre-trained network layer.

[0092] The quantization process for the first object detection model is completed by reducing the precision of the pre-trained network layers and stopping when the precision of the backbone network of the first object detection model falls below a preset threshold. This quantization process is repeated layer by layer. By using the backbone network precision threshold as a constraint, the computational load of model weight parameters and memory resource consumption can be reduced while ensuring the model's inference accuracy, thereby improving the model's inference speed.

[0093] In some embodiments, the first object detection model further includes a classifier corresponding to each of the pre-trained network layers, the classifier being used to predict a probability distribution based on the feature information output by the pre-trained network layer;

[0094] The classifier is divided into a first classifier corresponding to the last pre-trained network layer and a second classifier other than the first classifier. The second classifier is obtained by distillation of the first classifier.

[0095] See Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of a target detection method based on a pre-trained model provided in this application, as shown below. Figure 3 As shown, the quantization process for each pre-trained network layer of the first object detection model includes:

[0096] Step S310: Obtain image data samples and perform the following processing on each of the pre-trained network layers:

[0097] Step S320: Input the image data sample into the first target detection model to obtain the first probability distribution output by the first classifier and the second probability distribution output by the second classifier;

[0098] Step S330: Determine the distribution difference between the first probability distribution and the second probability distribution;

[0099] Step S340: If the distribution difference is less than a preset threshold, reduce the model parameters of the current pre-trained network layer from the initial precision to the target precision, and use the target precision as the initial precision of the model parameters of the current pre-trained network in the next round of processing, and repeat until the distribution difference is greater than the preset threshold.

[0100] For example, see Figure 7, Figure 7 This is a schematic diagram of a target detection model structure provided in an embodiment of this application, as shown below. Figure 7 As shown, the backbone network in this object detection model includes an embedding layer, pre-trained network layers (Transformer layers) composed of stacked Transformer blocks, and a classifier. Each Transformer layer corresponds to a classifier, which is used to acquire the feature information extracted by the Transformer layer and predict the probability distribution based on these features, thus obtaining the prediction result of image object detection. The classifier corresponding to the last Transformer layer is denoted as the first classifier, and the classifiers corresponding to the remaining Transformer layers are denoted as the second classifier. The second classifier is obtained by knowledge distillation using the first classifier as a teacher model.

[0101] Based on this, for each pre-trained network layer in the first object detection model, the first probability distribution and the second probability distribution are obtained through the second classifier corresponding to the current pre-trained network layer and the second classifier corresponding to the last pre-trained network layer, respectively. Then, the distribution difference between the second classifier and the first classifier corresponding to the current pre-trained network layer is obtained according to the first probability distribution and the second probability distribution. If the distribution difference is within the prediction threshold, the accuracy of the model parameters of the current pre-trained network layer is reduced until the distribution difference is greater than the prediction threshold. This step is repeated layer by layer to complete the quantization process of the entire pre-trained network layer.

[0102] By reducing the precision of pre-trained network layers and stopping when the probability distribution difference between the second classifier corresponding to the current pre-trained network layer and the first classifier corresponding to the last pre-trained network layer exceeds a preset threshold, this quantization step is repeated layer by layer to complete the quantization process of the first object detection model. Using the probability distribution difference between the main branch and the sub-branch as a constraint, this approach can reduce the computational cost of model weight parameters and memory resource consumption while ensuring model inference performance, thereby improving model inference speed.

[0103] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of a target detection method based on a pre-trained model provided in this application, as shown below. Figure 4 As shown, the training steps of the first object detection model include:

[0104] Step S410: Obtain the image sample training set;

[0105] Step S420: With the weight parameters of the second classifier fixed, the image sample training set is input into the first object detection model to update the pre-trained network layer and the weight parameters of the first classifier.

[0106] Step S430: With the weight parameters of the pre-trained network layer and the first classifier fixed, the trained first classifier is used as the teacher model to distill the second classifier to update the weight parameters of the second classifier.

[0107] For example, the first object detection model is built based on a Transformer pre-trained network. Each Transformer layer corresponds to its own classifier. The Transformer layer and the second classifier are used as the backbone network, and the first classifier is used as a branch network. During the training process of the first object detection model, its backbone network is trained first, and the weight parameters of the Transformer layer and the second classifier are updated. Then, with the weight parameters of the backbone network fixed, that is, with the weight parameters of the Transformer layer and the first classifier fixed, the first classifier is used as a teacher model to distill the second classifier to update the weight parameters of the second classifier, so that the second classifier can acquire the knowledge learned by the first classifier.

[0108] By training the main branch and branches sequentially, the knowledge learned by the first classifier is distilled into the second classifier, enabling the second classifier to acquire knowledge from the image sample training set. At the same time, this reduces the training time of the entire first object detection model.

[0109] In one specific embodiment, the image sample training set uses the VOC2007 public dataset, which includes 20 preset detection targets: airplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorcycle, person, potted plant, sheep, sofa, train, and television monitor. The first target detection model is trained using this image sample training set, enabling it to accurately determine whether the image to be detected contains the aforementioned preset targets.

[0110] Understandably, depending on the actual application scenario, the first object detection model can be trained using a corresponding image sample training set, so that the first object detection model has the corresponding object detection capability.

[0111] In one possible implementation, the target detection method based on a pre-trained network provided in this application is applied to the field of digital healthcare. In medical application scenarios, this application acquires medical images, and the objects contained in the medical images belong to the type of lesions, that is, the parts of the body where diseases have occurred. Medical images refer to images of internal tissues obtained non-invasively for medical treatment or medical research, such as images of the stomach, abdomen, heart, knee, and brain, such as images generated by medical instruments such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasonic), X-ray images, electroencephalograms, and optical photography.

[0112] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating another embodiment of a target detection method based on a pre-trained model provided in this application, as shown below. Figure 5 As shown, during the process of the second object detection model predicting the image to be detected:

[0113] Step S510: The information entropy corresponding to the image to be detected is obtained sequentially through the second classifier;

[0114] Step S520: If the information entropy obtained by the current second classifier is less than a preset threshold, then exit the pre-trained network layer; or,

[0115] Step S530: If the information entropy obtained by the second classifier is greater than the preset threshold, then feature extraction continues through the pre-trained network layer.

[0116] Understandably, on the one hand, in the first object detection model, each pre-trained network layer is assigned a classifier, which allows for model quantization while ensuring model performance, using the difference in probability distribution between classifiers as a constraint. On the other hand, in the second object detection model, during the prediction of the image to be detected, each second classifier is used to make an early exit judgment. If the early exit condition is met, the subsequent processing of the pre-trained network is no longer performed, reducing the model's inference time.

[0117] Specifically, during the reasoning process of the second detection model on the image to be detected, the information entropy corresponding to the image to be detected is obtained sequentially through the second classifier, that is, the probability distribution of the image to be detected is obtained through the second classifier, and the information entropy corresponding to the detected image is calculated based on the probability distribution. When the information entropy is less than a preset threshold, the pre-trained network layer is exited; otherwise, when the information entropy is greater than the preset threshold, feature extraction continues through the pre-trained network.

[0118] In some embodiments, the information entropy is determined by the following formula:

[0119]

[0120] Where H(x) is the information entropy corresponding to the image x to be detected, n is the number of classification categories, and P... i (x) represents the probability distribution corresponding to the image x to be detected.

[0121] The object detection method based on a pre-trained network provided in this application is described below through a specific embodiment:

[0122] Obtain the pre-trained first object detection model, which is built based on a pre-trained network, such as... Figure 7 As shown, the backbone network in the first object detection model includes an embedding layer, a pre-trained network layer (Transformer layer) composed of stacked Transformer blocks, and a classifier. The classifier is used to predict the probability distribution based on the feature information output by the pre-trained network layer, and the classifier is divided into a first classifier corresponding to the last pre-trained network layer and a second classifier other than the first classifier. The second classifier is obtained by distillation of the first classifier.

[0123] The pre-trained network layers in the first object detection model are quantized, including:

[0124] For each pre-trained network layer, image data samples are input into the first object detection model to obtain the output precision of the last pre-trained network layer in the first object detection model. The model is then quantized using the output precision as a constraint. Specifically, if the output precision is greater than a preset threshold, the precision representation of that layer is reduced until the output precision is less than the preset threshold. This quantization step is repeated layer by layer to complete the quantization of the entire first object detection model, resulting in the second object detection model; or,

[0125] For each pre-trained network layer, image data samples are input into the first object detection model to obtain the first probability distribution output by the first classifier and the second probability distribution output by the second classifier corresponding to the current pre-trained network layer. The distribution difference between the first classifier and the second classifier is determined, and the model is quantized using the distribution difference as a constraint. Specifically, if the distribution difference is less than a preset threshold, the accuracy of that layer is reduced until the distribution difference is greater than the prediction threshold. This quantization step is repeated layer by layer to complete the quantization of the entire first object detection model and obtain the second object detection model.

[0126] The second object detection model is used to predict the image to be detected, and the prediction result corresponding to the image to be detected is obtained. The prediction result includes the classification category and location information of the detected object.

[0127] During the reasoning process using the second object detection model, the probability distribution of the image to be detected is obtained sequentially through the second classifier. The information entropy of the image to be detected is calculated based on the probability distribution. The information entropy is used as a judgment condition to perform an early exit operation in the model reasoning process. Specifically, when the information entropy is less than a preset threshold, the current pre-trained network layer is exited; otherwise, when the information entropy is greater than the preset threshold, the processing of subsequent pre-trained network layers continues.

[0128] The application scenarios of the embodiments of this application are described below through specific examples:

[0129] The target detection method based on pre-trained networks provided in this application is applied to an artificial intelligence-based auxiliary diagnostic system in the field of digital healthcare. The auxiliary diagnostic system includes multiple medical terminal devices, which are used to execute the target detection method based on pre-trained networks provided in this application.

[0130] The medical terminal device first acquires a pre-trained first target detection model. It then quantizes each pre-trained network layer within the first target detection model, reducing the model parameters of each pre-trained network from initial precision to target precision. This quantized first target detection model is then used as the second target detection model. After quantization, the terminal device acquires the medical image to be detected and inputs it into the second target detection model. The second target detection model then provides a prediction result indicating whether the medical image contains a preset detection target. By reducing the precision of each pre-trained network layer in the target detection model, the model is quantized. This allows terminal devices located close to the medical image data source to perform target detection using the pre-trained model, ensuring target detection accuracy while achieving fast inference speed and low hardware resource consumption, thus meeting the high real-time requirements of medical auxiliary diagnosis.

[0131] This application proposes a pre-trained network-based object detection method. This method obtains a trained first object detection model, which includes multiple pre-trained network layers. Then, each pre-trained network layer of the first object detection model is quantized to obtain a second object detection model. The quantization process includes reducing the precision representation of the pre-trained network layers. After quantization, a target image is acquired and input into the second object detection model to obtain a prediction result. This application utilizes a pre-trained network-based object detection model for object detection. Furthermore, by quantizing the object detection model by reducing the precision representation of each pre-trained network layer before inference, it can improve inference speed and reduce hardware resource consumption while maintaining object detection accuracy.

[0132] Please see Figure 9 This application also provides a target detection device 100 based on a pre-trained network, which includes:

[0133] The first acquisition module 110 is used to acquire a trained first object detection model, which includes multiple pre-trained network layers.

[0134] The model quantization module 120 is used to quantize each of the pre-trained network layers of the first object detection model, so that the model parameters of each of the pre-trained network layers in the first object detection model are reduced from the initial accuracy to the target accuracy, and the quantized first object detection model is used as the second object detection model.

[0135] The second acquisition module 130 is used to acquire the image to be detected;

[0136] The target detection module 140 is used to input the image to be detected into the second target detection model so as to obtain the prediction result of the image to be detected through the second target detection model. The prediction result is used to indicate whether the image to be detected includes a preset detection target.

[0137] In some embodiments, the model quantization module 120 further includes:

[0138] The sample acquisition sub-model is used to acquire image data samples;

[0139] The output accuracy acquisition submodule is used to input the image data sample into the first target detection model to obtain the output accuracy of the last pre-trained network layer in the first target detection model.

[0140] The judgment submodule is used to reduce the model parameters of the current pre-trained network from the initial accuracy to the target accuracy if the output accuracy is greater than a preset threshold, and use the target accuracy as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing, and repeat until the output accuracy is less than the preset threshold.

[0141] In some embodiments, the first object detection model further includes a classifier corresponding to each of the pre-trained network layers, the classifier being used to predict a probability distribution based on the feature information output by the pre-trained network layer;

[0142] The classifier is divided into a first classifier corresponding to the last pre-trained network layer and a second classifier other than the first classifier. The second classifier is obtained by distillation of the first classifier.

[0143] The model quantization module 120 also includes:

[0144] The sample acquisition sub-model is used to acquire image data samples;

[0145] The probability distribution acquisition submodule is used to input the image data sample into the first target detection model to obtain the first probability distribution output by the first classifier and the second probability distribution output by the second classifier.

[0146] The distribution difference acquisition submodule is used to determine the distribution difference between the first probability distribution and the second probability distribution.

[0147] The judgment submodule is used to reduce the model parameters of the current pre-trained network layer from the initial accuracy to the target accuracy if the distribution difference is less than a preset threshold, and use the target accuracy as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing, and repeat until the distribution difference is greater than the preset threshold.

[0148] This application proposes a target detection device based on a pre-trained network. This device acquires a trained first target detection model, which includes multiple pre-trained network layers. Each pre-trained network layer of the first target detection model is then quantized to obtain a second target detection model. The quantization process includes reducing the precision representation of the pre-trained network layers. After quantization, a target image is acquired and input into the second target detection model to obtain a prediction result. This embodiment utilizes a target detection model built on a pre-trained network for target detection. Furthermore, by quantizing the target detection model by reducing the precision representation of each pre-trained network layer before inference, it can improve model inference speed and reduce hardware resource consumption while maintaining target detection accuracy.

[0149] It should be noted that the information interaction and execution process between the modules of the above-mentioned device are based on the same concept as the method embodiment of this application. For details on their specific functions and technical effects, please refer to the method embodiment section, and they will not be repeated here.

[0150] Please see Figure 10 , Figure 10 This application illustrates the hardware structure of an electronic device according to an embodiment of the present application. The electronic device includes:

[0151] The processor 210 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant computer programs to implement the technical solutions provided in the embodiments of this application.

[0152] The memory 220 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 220 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 220 and called and executed by the processor 210 using the target detection method based on a pre-trained network according to the embodiments of this application.

[0153] Input / output interface 230 is used to implement information input and output;

[0154] The communication interface 240 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 250 is used to transmit information between each component of the device (such as processor 210, memory 220, input / output interface 230 and communication interface 240).

[0155] The processor 210, memory 220, input / output interface 230 and communication interface 240 are connected to each other within the device via bus 250.

[0156] This application also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more computer programs, which can be executed by one or more processors to implement the above-described target detection method based on a pre-trained network.

[0157] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0158] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0161] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0162] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0163] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0165] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0166] Furthermore, the functional units in each embodiment of this application can be integrated into a single processing unit, or each unit can exist independently, or two or more units can be integrated into a single unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0167] If the assembled units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of each embodiment of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A target detection method based on a pre-trained network, characterized in that, The method includes: Obtain a trained first object detection model, which includes multiple pre-trained network layers; Each pre-trained network layer of the first target detection model is quantized, so that the model parameters of each pre-trained network layer in the first target detection model are reduced from the initial accuracy to the target accuracy, and the quantized first target detection model is used as the second target detection model. Acquire the image to be detected; The image to be detected is input into the second target detection model to obtain a prediction result of the image to be detected. The prediction result is used to indicate whether the image to be detected includes a preset detection target. The first target detection model further includes a classifier corresponding to each of the pre-trained network layers, wherein the classifier is used to predict the probability distribution based on the feature information output by the pre-trained network layer; The classifier is divided into a first classifier corresponding to the last pre-trained network layer and a second classifier other than the first classifier. The second classifier is obtained by distillation of the first classifier. The quantization process performed on each of the pre-trained network layers of the first object detection model includes: Obtain image data samples and perform the following processing on each of the pre-trained network layers: The image data sample is input into the first target detection model to obtain the first probability distribution output by the first classifier and the second probability distribution output by the second classifier. Determine the distribution difference between the first probability distribution and the second probability distribution; If the distribution difference is less than a preset threshold, the model parameters of the current pre-trained network layer are reduced from the initial accuracy to the target accuracy, and the target accuracy is used as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing. This process is repeated until the distribution difference is greater than the preset threshold.

2. The method according to claim 1, characterized in that, The quantization process for each pre-trained network layer of the first object detection model includes: Obtain image data samples and perform the following processing on each of the pre-trained network layers: The image data sample is input into the first target detection model to obtain the output accuracy of the last pre-trained network layer in the first target detection model; If the output accuracy is greater than a preset threshold, the model parameters of the current pre-trained network are reduced from the initial accuracy to the target accuracy, and the target accuracy is used as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing. This process is repeated until the output accuracy is less than the preset threshold.

3. The method according to claim 1, characterized in that, The training steps of the first object detection model include: Obtain the image sample training set; With the weight parameters of the second classifier fixed, the image sample training set is input into the first target detection model to update the weight parameters of the pre-trained network layer and the first classifier. With the pre-trained network layers and the weight parameters of the first classifier fixed, the trained first classifier is used as a teacher model to distill the second classifier to update the weight parameters of the second classifier.

4. The method according to claim 1, characterized in that, During the process of the second object detection model predicting the image to be detected: The information entropy corresponding to the image to be detected is obtained sequentially through the second classifier; If the information entropy obtained by the second classifier is less than a preset threshold, then exit the current pre-trained network layer. or, If the information entropy obtained by the second classifier is greater than a preset threshold, then feature extraction continues through the pre-trained network layer.

5. The method according to claim 4, characterized in that, The information entropy is determined by the following formula: Among them, the Image to be detected The corresponding information entropy, the For the number of classification categories, the Image to be detected The corresponding probability distribution.

6. The method according to claim 1, characterized in that, After obtaining the trained first object detection model, and before quantizing each of the pre-trained network layers of the first object detection model, the method further includes: Obtain image sample fine-tuning set; The image sample fine-tuning set is input into the first target detection model to obtain the corresponding training prediction result through the first target detection model; The loss value is determined based on the preset target detection labels in the image sample fine-tuning set, the training prediction results, and the preset loss function; Based on the loss value, the model parameters of the first target detection model are fine-tuned to obtain the fine-tuned first target detection model.

7. A target detection device based on a pre-trained network, characterized in that, The device includes: The first acquisition module is used to acquire a trained first object detection model, which includes multiple pre-trained network layers. The model quantization module is used to quantize each of the pre-trained network layers of the first target detection model, so that the model parameters of each of the pre-trained network layers in the first target detection model are reduced from the initial accuracy to the target accuracy, and the quantized first target detection model is used as the second target detection model. The second acquisition module is used to acquire the image to be detected; The target detection module is used to input the image to be detected into the second target detection model so as to obtain the prediction result of the image to be detected through the second target detection model. The prediction result is used to indicate whether the image to be detected includes a preset detection target. The first target detection model further includes a classifier corresponding to each of the pre-trained network layers, wherein the classifier is used to predict the probability distribution based on the feature information output by the pre-trained network layer; The classifier is divided into a first classifier corresponding to the last pre-trained network layer and a second classifier other than the first classifier. The second classifier is obtained by distillation of the first classifier. The quantization process performed on each of the pre-trained network layers of the first object detection model includes: Obtain image data samples and perform the following processing on each of the pre-trained network layers: The image data sample is input into the first target detection model to obtain the first probability distribution output by the first classifier and the second probability distribution output by the second classifier. Determine the distribution difference between the first probability distribution and the second probability distribution; If the distribution difference is less than a preset threshold, the model parameters of the current pre-trained network layer are reduced from the initial accuracy to the target accuracy, and the target accuracy is used as the initial accuracy of the model parameters of the current pre-trained network in the next round of processing. This process is repeated until the distribution difference is greater than the preset threshold.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that is executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images