Target detection method and device based on Diffusion model, equipment and storage medium
By using the Diffusion model in object detection, combining preset neural networks and diffusion Diffusion model, the problems of low accuracy and poor adaptability of traditional methods are solved, and more efficient object detection and classification are achieved, and business processing efficiency is improved.
Patent Information
- Application Number
- CN202411071537.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-12-17
AI Technical Summary
Traditional target detection methods have low accuracy in medical and financial business scenarios, and have low processing efficiency for small data volume business needs, making it difficult to quickly adapt to new data needs.
The object detection method based on the Diffusion model is adopted to extract the feature information of the target image through the preset neural network backbone model, and the object detection box is generated using the pre-trained diffusion Diffusion model, and fine-tune it in combination with the bounding box regression function to improve the accuracy of object detection. and efficiency.
It significantly improves the accuracy and efficiency of object detection in the image, can better generalize to new data, reduces the cost of business processing, and improves business delivery efficiency.
Smart Images
Figure CN119152484A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of image processing and finance, and in particular to a target detection method, device, equipment and storage medium based on a Diffusion model. Background Art
[0002] In medical and financial business scenarios, the target detection task requires identifying different objects from the image and marking the identified objects with prediction boxes and assigning corresponding labels. For target detection tasks, in addition to having excellent recognition capabilities, the ability to quickly identify objects is also crucial, although this will come at the cost of a certain loss of accuracy. Among them, the mainstream target detection tasks are mainly divided into the following steps:
[0003] 1. Use the neural network backbone model (such as ResNet, Vision Transformer, etc.) to extract the feature information of the image. The extracted image feature information will be shared for subsequent operations;
[0004] 2. Use RPN (Region Proposal Network) to classify a series of anchors, and then fine-tune the anchors through the boundingbox regression function to generate a rough detection box where the target may exist;
[0005] 3. Combine the feature information extracted by the neural network backbone model and the detection frame information generated by RPN to process the feature information in the detection frame and send it to the subsequent network to determine the target category;
[0006] 4. Classify the target by the feature information in the detection box, and use methods such as non-maximum suppression (NMS) to filter a large number of overlapping prediction boxes so that each target is framed by a single prediction box.
[0007] However, in practical applications, the accuracy of traditional detection methods is relatively low and their ability to continuously learn is limited. At the same time, when dealing with small data volume business needs, traditional methods can only respond through a high-cost method of overall fine-tuning, which greatly limits the efficiency of business delivery. Therefore, how to improve the efficiency of target detection in images and thus improve the efficiency of business processing has become a technical problem that needs to be solved urgently. Summary of the invention
[0008] The present application provides a target detection method, apparatus, device and storage medium based on a Diffusion model, so as to improve the efficiency of business processing by improving the efficiency of target detection in an image.
[0009] In a first aspect, the present application provides a target detection method based on a Diffusion model, the method comprising:
[0010] Extract target feature information of the target image through a preset neural network backbone model;
[0011] Based on the target detection frame generated by the pre-trained diffusion model and the target feature information, the target object in the target image is determined.
[0012] In a second aspect, the present application also provides a target detection device based on a Diffusion model, the device comprising:
[0013] A target feature information extraction module is used to extract target feature information of a target image through a preset neural network backbone model;
[0014] The target detection module is used to determine the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information.
[0015] In a third aspect, the present application also provides a computer device, comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the above-mentioned Diffusion model-based target detection method when executing the computer program.
[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the target detection method based on the Diffusion model as described above.
[0017] The present application discloses a target detection method, device, equipment and storage medium based on a Diffusion model. The target detection method based on the Diffusion model includes extracting target feature information of a target image through a preset neural network backbone model; determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information. In the above manner, the present application can use the rich features that have been learned by using a pre-trained neural network backbone model to more accurately identify the target object in the image. The pre-trained models are usually trained on a variety of data sets, which enables them to better generalize to new and unseen data. By fine-tuning on the basis of the pre-trained model, target detection and classification can be performed simultaneously, and the efficiency of business processing is improved by improving the efficiency of target detection in the image. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 is a schematic flow chart of a target detection method based on a Diffusion model provided in the first embodiment of the present application;
[0020] Figure 2 is a schematic flow chart of a target detection method based on a Diffusion model provided in the second embodiment of the present application;
[0021] Figure 3 A schematic diagram of LORA lightweight fine-tuning of a target detection method based on a Diffusion model provided in the second embodiment of the present application;
[0022] Figure 4 A schematic block diagram of a target detection device based on a Diffusion model provided in an embodiment of the present application;
[0023] Figure 5 A schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0026] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0027] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0028] The embodiments of the present application provide a method, device, equipment and storage medium for target detection based on a Diffusion model. The target detection method based on the Diffusion model can be applied to a server. By using a pre-trained neural network backbone model, the rich features that have been learned can be used to more accurately identify target objects in images. Pre-trained models are usually trained on diverse data sets, which enables them to better generalize to new and unseen data. By fine-tuning based on the pre-trained model, target detection and classification can be performed simultaneously, and the efficiency of business processing is improved by improving the efficiency of target detection in images. The server can be an independent server or a server cluster.
[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0030] See also Figure 1 , Figure 1 This is a schematic flow chart of a target detection method based on a Diffusion model provided in the first embodiment of the present application. The target detection method based on the Diffusion model can be applied to a server to more accurately identify target objects in an image by using a pre-trained neural network backbone model, which can utilize the rich features that have been learned. Pre-trained models are usually trained on diverse data sets, which enables them to better generalize to new and unseen data. By fine-tuning based on the pre-trained model, target detection and classification can be performed simultaneously, thereby improving the efficiency of business processing by improving the efficiency of target detection in images.
[0031] like Figure 1 As shown, the target detection method based on the Diffusion model specifically includes steps S10 to S20.
[0032] Step S10, extracting target feature information of the target image through a preset neural network backbone model;
[0033] In one embodiment, similar to conventional target detection methods, conventional feature extraction is performed on the input image. The specific backbone network structure selected for use can be trained from scratch using the currently popular backbone network structures (such as ResNet, VisionTransformer, etc.), or the existing network model and corresponding weights can be directly called to fine-tune the current data set. The former requires a higher time cost, but the trained model parameters are more suitable for the current data set; the latter calls the existing model structure and weights, and usually these models themselves have extremely strong versatility and feature extraction capabilities, so they can also show excellent performance; but the adaptability to the current data set is relatively weak.
[0034] The inference process is also called the reverse denoising process. The traditional Diffusion model is to randomly sample in the Gaussian distribution, that is, and gradually denoise and restore according to the Step step length. The method of this embodiment is also similar to the traditional method. This embodiment also randomly samples a noise frame and gradually restores it to a frame close to the real frame according to the Step step length.
[0035] For example, taking the COCO2017val dataset as an example, under the same benchmark condition of using ResNet-50 as the backbone network, compared with the 40.2AP (Average precision) value of FasterR-CNN, the AP value of this embodiment can reach 45.8, which is much higher than the traditional detection method. And with the upgrade of the backbone network, the highest AP value of the method of this embodiment can reach a high accuracy of 53.
[0036] Step S20: determining a target object in the target image based on a target detection frame generated by a pre-trained diffusion model and the target feature information.
[0037] In one embodiment, the prediction box coordinates obtained by the Diffusion model inference and the image feature information extracted by the image encoder are combined to calculate the specific category (such as people, cars, traffic lights, etc.) to which each detection box belongs with the help of the fully connected layer and the Softmax function. At the same time, the boundingbox regression function is used again to obtain the position offset of each detection box, and the specific position of the real box is compared, so as to fine-tune the position of the prediction box again to improve the final recognition accuracy.
[0038] This embodiment discloses a target detection method, device, equipment and storage medium based on a Diffusion model. The target detection method based on the Diffusion model includes extracting target feature information of a target image through a preset neural network backbone model; determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information. In the above manner, the present application can use the rich features that have been learned by using a pre-trained neural network backbone model to more accurately identify the target object in the image. The pre-trained models are usually trained on a variety of data sets, which enables them to better generalize to new and unseen data. By fine-tuning on the basis of the pre-trained model, target detection and classification can be performed simultaneously, and the efficiency of business processing is improved by improving the efficiency of target detection in the image.
[0039] based on Figure 1 In the embodiment shown, in this embodiment, before step S10, the following steps are included:
[0040] Get the image training set;
[0041] The initial Diffusion model is trained based on the image training set to determine the pre-trained Diffusion model.
[0042] In one embodiment, the Diffusion model mainly consists of a forward process and a reverse process. The forward process is also called a diffusion process, and the reverse process is also called a denoising process, or a generating process. Compared with the traditional Diffusion model, which uses pixels as the basic unit for adding noise, this method uses the detection box as the basic unit for adding noise, that is, adding a noise box. The original real box is taken as X0, and then based on x0, any t steps x t Take samples.
[0043] Specifically, the above formula is: in and is the combination coefficient. It should be noted that in this process, the traditional method predicts that the image is not noise, while the prediction obtained in this embodiment is a detection frame, which is not a single dimension but a 4-dimensional one (i.e., the coordinates of the detection frame). The steps of training the Diffusion model are as follows:
[0044] "def train(image,bbox):
[0045] #bbox:ground truth
[0046] img_features=image_encoder(image)
[0047] #Pad gt_boxes to N
[0048] pre_box=pad_box(bbox)#padded boxes:[B,N,4]
[0049] pre_box=(pre_box*2-1)*scale
[0050] t = randint(0,T)
[0051] eps=normal(mean=0,std=1)
[0052] #Predict
[0053] new_pred_bbox=sqrt(alpha_com(t))*pred_box+sqrt(1–alpha_com(t))*eps
[0054] loss=pre_loss(new_pred_bbox,bbox)
[0055] return loss”.
[0056] This embodiment discloses a target detection method, device, equipment and storage medium based on a Diffusion model. The target detection method based on the Diffusion model includes obtaining an image training set; training an initial Diffusion model based on the image training set to determine the pre-trained Diffusion model; extracting target feature information of the target image through a preset neural network backbone model; and determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information. In the above manner, the present application can ensure that the model can perform well in different scenes and different lighting conditions by obtaining a diverse image training set, thereby improving the generalization ability of the model. Training the Diffusion model based on the image training set can enable the pre-trained model to better understand and learn the features in the image, so that it can make more accurate predictions when facing new images. By improving the efficiency of target detection in images, the efficiency of business processing is thereby improved.
[0057] In a specific embodiment, the initial Diffusion model is trained based on the image training set to determine the pre-trained Diffusion model, including:
[0058] Taking each training image in the image training set as a real training frame;
[0059] Based on the initial Diffusion model, each of the real training frames is diffused and randomly distributed to generate a noise training frame;
[0060] De-noising each of the noise training frames to generate a prediction training frame;
[0061] Prediction classification is performed on each of the prediction training frames, and the pre-trained Diffusion model is determined according to the prediction classification result.
[0062] In one embodiment, the detection box of the next stage is continuously estimated in the Step time step in the trained Diffusion model, and the Step is often set manually. A too large Step will lead to overfitting of the restored target, while a too small Step will lead to poor target quality. In the process of iterative restoration, the optimizer corresponding to Diffusion, such as DDPM and DDIM, can be used for reverse optimization. Finally, after multiple iterations and meeting the termination condition, the final predicted detection box can be output for the next stage.
[0063] This embodiment discloses a target detection method, device, equipment and storage medium based on a Diffusion model. The target detection method based on the Diffusion model includes obtaining an image training set; taking each training image in the image training set as a real training frame; performing diffusion random distribution on each real training frame based on the initial Diffusion model to generate a noise training frame; performing denoising on each noise training frame to generate a prediction training frame; performing prediction classification on each prediction training frame, and determining the pre-trained Diffusion model according to the result of the prediction classification; extracting the target feature information of the target image through a preset neural network backbone model; and determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion Diffusion model and the target feature information. In the above manner, the present application generates a noise training frame by diffusing the real training frame and performs denoising, so that the model can learn how to restore image features in the presence of noise, thereby improving the robustness to noise. The pre-trained Diffusion model is determined by prediction classification, which can ensure that the model is continuously optimized during the training process to improve the final prediction accuracy. Using the preset neural network backbone model to extract target feature information can ensure that the model focuses on the most important parts of the image, thereby improving the quality and efficiency of feature extraction. By predicting the type of target object, the model can provide more accurate information for image classification and scene understanding. By improving the efficiency of target detection in images, the efficiency of business processing is improved.
[0064] In a further embodiment, predicting and classifying each of the prediction training frames includes:
[0065] Determine the prediction frame coordinates of the prediction training frame through the initial Diffusion model, and extract the prediction training frame feature information of the prediction training frame through the preset neural network backbone model;
[0066] Based on the prediction box coordinates and the prediction training box feature information, the type of the prediction training box is predicted through a fully connected layer and a Softmax function.
[0067] The present embodiment discloses a target detection method, device, equipment and storage medium based on a Diffusion model. The target detection method based on the Diffusion model includes obtaining an image training set; taking each training image in the image training set as a real training frame; performing diffusion random distribution on each real training frame based on the initial Diffusion model to generate a noise training frame; performing denoising processing on each noise training frame to generate a predicted training frame; determining the predicted frame coordinates of the predicted training frame through the initial Diffusion model, and extracting the predicted training frame feature information of the predicted training frame through the preset neural network backbone model; predicting the type of the predicted training frame through a fully connected layer and a Softmax function based on the predicted frame coordinates and the predicted training frame feature information, and determining the pre-trained Diffusion model according to the result of the prediction classification; extracting the target feature information of the target image through the preset neural network backbone model; and determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion Diffusion model and the target feature information. In the above manner, the present application can use the rich features that have been learned by using the pre-trained neural network backbone model to more accurately identify the target object in the image. The pre-trained models are usually trained on a variety of data sets, which enables them to better generalize to new and unseen data. By fine-tuning the pre-trained model, target detection and classification can be performed simultaneously, which improves the efficiency of target detection in the image and thus improves the efficiency of business processing.
[0068] Further, the pre-trained Diffusion model is determined according to the result of the prediction classification, including:
[0069] The predicted position offset of each predicted training frame is determined by a bounding box regression function, and the predicted position offset is fine-tuned with the real training frame to determine the pre-trained Diffusion model.
[0070] In a specific embodiment, the prediction frame coordinates obtained by the Diffusion model inference and the image feature information extracted by the image encoder are combined to calculate the category (such as people, cars, traffic lights, etc.) to which each detection frame belongs with the help of the fully connected layer and the Softmax function. At the same time, the bounding box regression function is used again to obtain the position offset of each detection frame, and the specific position of the real frame is compared, so as to fine-tune the position of the prediction frame again to improve the final recognition accuracy.
[0071] The present embodiment discloses a target detection method, device, equipment and storage medium based on a Diffusion model. The target detection method based on the Diffusion model includes acquiring an image training set; taking each training image in the image training set as a real training frame; performing diffusion random distribution on each real training frame based on the initial Diffusion model to generate a noise training frame; performing denoising processing on each noise training frame to generate a predicted training frame; determining the predicted frame coordinates of the predicted training frame through the initial Diffusion model, and extracting the predicted training frame feature information of the predicted training frame through the preset neural network backbone model; predicting the type of the predicted training frame through a fully connected layer and a Softmax function based on the predicted frame coordinates and the predicted training frame feature information, determining the predicted position offset of each predicted training frame through a bounding box regression function, and fine-tuning the predicted position offset with the real training frame to determine the pre-trained Diffusion model; extracting the target feature information of the target image through the preset neural network backbone model; determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion Diffusion model and the target feature information. In the above manner, the present application can use the rich features that have been learned by using the pre-trained neural network backbone model to more accurately identify the target object in the image. The pre-trained models are usually trained on a variety of data sets, which enables them to better generalize to new and unseen data. By fine-tuning the pre-trained model, target detection and classification can be performed simultaneously, which improves the efficiency of target detection in the image and thus improves the efficiency of business processing.
[0072] See also Figure 2 , Figure 2This is a schematic flow chart of a target detection method based on a Diffusion model provided in the second embodiment of the present application. The target detection method based on the Diffusion model can be applied to a server to more accurately identify target objects in an image by using a pre-trained neural network backbone model, which can utilize the rich features that have been learned. Pre-trained models are usually trained on diverse data sets, which enables them to better generalize to new and unseen data. By fine-tuning based on the pre-trained model, target detection and classification can be performed simultaneously, thereby improving the efficiency of business processing by improving the efficiency of target detection in images.
[0073] based on Figure 1 The embodiment shown, this embodiment Figure 2 As shown, step S20 includes step S21.
[0074] Step S21: lightweight adjustment is performed on the pre-trained Diffusion model by using a preset LORA fine-tuning model.
[0075] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of LORA lightweight fine-tuning of a target detection method based on a Diffusion model provided in the second embodiment of the present application. LORA (Low-Rank Adaptation of LargeLanguage Model) is a training technology for fine-tuning models. By making slight modifications to the standard weight model, it can be 10 to 100 times smaller than a directly fine-tuned model. Specifically, LORA adds a bypass next to the original pre-trained model to perform a dimensionality reduction and then dimensionality increase operation to simulate the intrinsic rank process. Unlike conventional fine-tuning, this method fixes the parameters of the pre-trained model during the training process and only trains the dimensionality reduction matrix A and the dimensionality increase matrix B. The overall input and output dimensions of the model remain unchanged, and only the parameters of the bypass and the parameters of the pre-trained model are superimposed at the final output. Therefore, during the inference process, the number of parameters will increase compared to the original. Subsequently, the A matrix is initialized with a Gaussian distribution, and the B matrix is initialized with 0 to ensure that the bypass matrix is still a zero matrix at the beginning of training. Generally speaking, LORA is applied in the correlation-attention layer of Unet, which is the layer where images and texts interact.
[0076] In actual business scenarios, this method uses extremely low cost to test whether the target detection model based on the Diffusion model can adapt to new requirements (i.e. new data). If the LORA technology can be directly used to fine-tune the performance requirements of the business, the amount of calculation can be greatly reduced without fine-tuning the entire model and consuming a lot of computing resources.
[0077] based on Figure 2 In the embodiment shown, in this embodiment, step S21 includes:
[0078] Performing a dimensionality reduction operation on the pre-trained Diffusion model through a preset dimensionality reduction matrix to generate a dimensionality reduction Diffusion model;
[0079] Performing a dimension-upgrading operation on the dimensionality-reducing Diffusion model by using a preset dimension-upgrading matrix to generate lightweight adjustment parameters;
[0080] The pre-trained Diffusion model is lightweight adjusted based on the lightweight adjustment parameters.
[0081] Specifically, when faced with new data provided by new business needs, traditional methods can only respond through the high-cost method of overall fine-tuning. Compared with traditional methods, the method of this embodiment can use lightweight LORA technology for fast fine-tuning, and can also achieve relatively excellent performance. Taking the CrowdHuman dataset as an example, the recall rate of FasterR-CNN can reach 90.2 after fine-tuning, while the method of this embodiment can achieve a recall rate of 96.2 using only LORA technology for fast fine-tuning. This also shows that after using LORA lightweight fine-tuning, the method of this embodiment can not only complete the fine-tuning process more quickly, but also ensure a certain accuracy rate to complete rapid delivery.
[0082] This embodiment discloses a method, device, equipment and storage medium for target detection based on a Diffusion model. The method for target detection based on a Diffusion model includes obtaining an image training set; training an initial Diffusion model based on the image training set to determine the pre-trained Diffusion model; extracting target feature information of the target image through a preset neural network backbone model; determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion Diffusion model and the target feature information; performing a dimensionality reduction operation on the pre-trained Diffusion model through a preset dimensionality reduction matrix to generate a reduced dimensionality Diffusion model; performing a dimensionality increase operation on the reduced dimensionality Diffusion model through a preset dimensionality increase matrix to generate lightweight adjustment parameters; and performing lightweight adjustment on the pre-trained Diffusion model based on the lightweight adjustment parameters. In the above manner, the present application can use the rich features that have been learned by the pre-trained neural network backbone model to more accurately identify the target object in the image. The pre-trained models are usually trained on a variety of data sets, which enables them to better generalize to new and unseen data. By fine-tuning the pre-trained model, target detection and classification can be performed simultaneously, which improves the efficiency of target detection in images and thus improves the efficiency of business processing.
[0083] See also Figure 4 , Figure 4 The embodiment of the present application provides a schematic block diagram of a target detection device based on a Diffusion model, wherein the target detection device based on a Diffusion model is used to execute the aforementioned target detection method based on a Diffusion model. The target detection device based on a Diffusion model can be configured on a server.
[0084] like Figure 4 As shown, the target detection device based on the Diffusion model includes:
[0085] The target feature information extraction module 410 is used to extract the target feature information of the target image through a preset neural network backbone model;
[0086] The target detection module 420 is used to determine the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information.
[0087] Furthermore, the target detection device 400 based on the Diffusion model further includes:
[0088] An image training set acquisition module, used to acquire an image training set;
[0089] The pre-trained Diffusion model determination module is used to train the initial Diffusion model based on the image training set to determine the pre-trained Diffusion model.
[0090] Furthermore, the pre-trained Diffusion model determination module includes:
[0091] A real training frame determination submodule, used to use each training image in the image training set as a real training frame;
[0092] A noise training frame generation submodule, used for performing diffusion random distribution on each of the real training frames based on the initial Diffusion model to generate a noise training frame;
[0093] A prediction training frame generation submodule, used for performing denoising processing on each of the noise training frames to generate a prediction training frame;
[0094] The pre-trained Diffusion model determination submodule is used to perform prediction classification on each of the prediction training frames, and determine the pre-trained Diffusion model according to the prediction classification results.
[0095] Furthermore, the pre-trained Diffusion model determines a submodule, including:
[0096] A prediction training frame feature information extraction unit, used to determine the prediction frame coordinates of the prediction training frame through the initial Diffusion model, and extract the prediction training frame feature information of the prediction training frame through the preset neural network backbone model;
[0097] A predicted training frame type prediction unit is used to predict the type of the predicted training frame through a fully connected layer and a Softmax function based on the predicted frame coordinates and the predicted training frame feature information.
[0098] Furthermore, the pre-trained Diffusion model determines the submodule, further comprising:
[0099] The fine-tuning unit is used to determine the predicted position offset of each predicted training frame through a bounding box regression function, and fine-tune the predicted position offset with the real training frame to determine the pre-trained Diffusion model.
[0100] Furthermore, the target detection device 400 based on the Diffusion model further includes:
[0101] The lightweight adjustment module is used to perform lightweight adjustment on the pre-trained Diffusion model by presetting the LORA fine-tuning model.
[0102] Furthermore, the lightweight adjustment module includes:
[0103] A dimensionality reduction unit, used for performing a dimensionality reduction operation on the pre-trained Diffusion model through a preset dimensionality reduction matrix to generate a dimensionality reduction Diffusion model;
[0104] A dimension-increasing unit, used for performing a dimension-increasing operation on the dimension-reduced Diffusion model through a preset dimension-increasing matrix to generate lightweight adjustment parameters;
[0105] A lightweight adjustment unit is used to perform lightweight adjustment on the pre-trained Diffusion model based on the lightweight adjustment parameters.
[0106] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0107] The above-mentioned device can be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.
[0108] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.
[0109] See also Figure 5 The computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.
[0110] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any target detection method based on the Diffusion model.
[0111] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0112] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any target detection method based on the Diffusion model.
[0113] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0114] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0115] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0116] Extract target feature information of the target image through a preset neural network backbone model;
[0117] Based on the target detection frame generated by the pre-trained diffusion model and the target feature information, the target object in the target image is determined.
[0118] In one embodiment, before extracting target feature information of a target image by using a preset neural network backbone model, it is used to implement:
[0119] Get the image training set;
[0120] The initial Diffusion model is trained based on the image training set to determine the pre-trained Diffusion model.
[0121] In one embodiment, the initial Diffusion model is trained based on the image training set to determine the pre-trained Diffusion model for achieving:
[0122] Taking each training image in the image training set as a real training frame;
[0123] Based on the initial Diffusion model, each of the real training frames is diffused and randomly distributed to generate a noise training frame;
[0124] De-noising each of the noise training frames to generate a prediction training frame;
[0125] Prediction classification is performed on each of the prediction training frames, and the pre-trained Diffusion model is determined according to the prediction classification result.
[0126] In one embodiment, prediction classification is performed on each of the prediction training frames to achieve:
[0127] Determine the prediction frame coordinates of the prediction training frame through the initial Diffusion model, and extract the prediction training frame feature information of the prediction training frame through the preset neural network backbone model;
[0128] Based on the prediction box coordinates and the prediction training box feature information, the type of the prediction training box is predicted through a fully connected layer and a Softmax function.
[0129] In one embodiment, the pre-trained Diffusion model is determined according to the prediction classification result to achieve:
[0130] The predicted position offset of each predicted training frame is determined by a boundingbox regression function, and the predicted position offset is fine-tuned with the real training frame to determine the pre-trained Diffusion model.
[0131] In one embodiment, after determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information, it is used to implement:
[0132] The pre-trained Diffusion model is light-weighted by using the preset LORA fine-tuning model.
[0133] In one embodiment, the pre-trained Diffusion model is lightweight adjusted by presetting the LORA fine-tuning model to achieve:
[0134] Performing a dimensionality reduction operation on the pre-trained Diffusion model through a preset dimensionality reduction matrix to generate a dimensionality reduction Diffusion model;
[0135] Performing a dimension-upgrading operation on the dimensionality-reducing Diffusion model by using a preset dimension-upgrading matrix to generate lightweight adjustment parameters;
[0136] The pre-trained Diffusion model is lightweight adjusted based on the lightweight adjustment parameters.
[0137] A computer-readable storage medium is also provided in an embodiment of the present application, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and the processor executes the program instructions to implement any one of the target detection methods based on the Diffusion model provided in the embodiments of the present application.
[0138] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A target detection method based on a Diffusion model, characterized in that: include: Extract target feature information of the target image through a preset neural network backbone model; Based on the target detection frame generated by the pre-trained diffusion model and the target feature information, the target object in the target image is determined.
2. The target detection method based on the Diffusion model according to claim 1, characterized in that: Before extracting target feature information of the target image by using the preset neural network backbone model, the method includes: Get the image training set; The initial Diffusion model is trained based on the image training set to determine the pre-trained Diffusion model.
3. The target detection method based on the Diffusion model according to claim 2, characterized in that: The step of training the initial Diffusion model based on the image training set to determine the pre-trained Diffusion model includes: Taking each training image in the image training set as a real training frame; Based on the initial Diffusion model, each of the real training frames is diffused and randomly distributed to generate a noise training frame; Performing denoising on each of the noise training frames to generate a prediction training frame; Prediction classification is performed on each of the prediction training frames, and the pre-trained Diffusion model is determined according to the prediction classification result.
4. The target detection method based on the Diffusion model according to claim 3 is characterized in that: The predicting and classifying each of the prediction training frames includes: Determine the prediction frame coordinates of the prediction training frame through the initial Diffusion model, and extract the prediction training frame feature information of the prediction training frame through the preset neural network backbone model; Based on the prediction box coordinates and the prediction training box feature information, the type of the prediction training box is predicted through a fully connected layer and a Softmax function.
5. The target detection method based on the Diffusion model according to claim 4, characterized in that: Determining the pre-trained Diffusion model according to the prediction classification result includes: The predicted position offset of each predicted training frame is determined by a bounding box regression function, and the predicted position offset is fine-tuned with the real training frame to determine the pre-trained Diffusion model.
6. The target detection method based on the Diffusion model according to claim 1, characterized in that: After determining the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information, the method includes: The pre-trained Diffusion model is light-weighted by using the preset LORA fine-tuning model.
7. The target detection method based on the Diffusion model according to claim 6, characterized in that: The lightweight adjustment of the pre-trained Diffusion model by presetting the LORA fine-tuning model includes: Performing a dimensionality reduction operation on the pre-trained Diffusion model through a preset dimensionality reduction matrix to generate a dimensionality reduction Diffusion model; Performing a dimension-upgrading operation on the dimensionality-reducing Diffusion model by using a preset dimension-upgrading matrix to generate lightweight adjustment parameters; The pre-trained Diffusion model is lightweight adjusted based on the lightweight adjustment parameters.
8. A target detection device based on a Diffusion model, characterized in that: include: A target feature information extraction module is used to extract target feature information of a target image through a preset neural network backbone model; The target detection module is used to determine the target object in the target image based on the target detection frame generated by the pre-trained diffusion model and the target feature information.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the target detection method based on the Diffusion model as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the target detection method based on the Diffusion model as described in any one of claims 1 to 7.