Target detection model fine tuning method and device, equipment and medium
Through sparse convolutional units and mixed prompt learning, the problem of insufficient extraction ability of the object detection model for image spatial features and semantic information is solved, which reduces training costs and storage requirements, and improves the generalization and robustness of the model.
Patent Information
- Application Number
- CN202411130892.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-06
AI Technical Summary
The existing object detection model lacks the ability to extract image spatial features and semantic information, resulting in the need for large-scale data set training, which increases training cost and storage requirements, and affects the generalization ability and robustness of the model.
Sparse convolutional units are used to build a general network for object detection. Through self-supervised training and mixed prompt learning, multi-scale feature information is obtained, which makes up for the shortcomings of implicit prompts, reduces training costs and storage parameters, and improves the generalization ability and application scenarios of the model. Extensibility.
It realizes effective representation of data, reduces the cost of fine-tuning training and storage parameters, improves the generalization and scalability of the model in different scenarios, and enhances the understanding of spatial and semantic information of image data.
Smart Images

Figure CN119089971A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of model training technology, and more specifically, to a method, device, equipment and medium for fine-tuning a target detection model. Background Art
[0002] Existing target detection models have poor ability to extract image spatial features and semantic information. Therefore, large-scale datasets are required for training. However, the data diversity of large-scale datasets will greatly increase the training cost of target detection models and will also affect the detection performance of target detection models. To solve this problem, various transfer learning methods and model fine-tuning techniques have emerged. By pre-training the model on large-scale datasets and fine-tuning it on the target task, the model can be better adapted to specific scenarios and conditions.
[0003] However, after using large-scale data sets to train pre-trained weights, the existing technology generally adopts end-to-end fine-tuning or a fine-tuning method that freezes some parameters and trains other parameters. The above fine-tuning method requires the storage and deployment of a separate backbone parameter copy for each task. For fine-tuning of models based on a large number of parameters, this requires huge training costs and the storage of significantly more parameters.
[0004] At the same time, the above fine-tuning methods will affect the network's overall understanding of the input data, and the training cost and GPU memory occupancy rate are relatively high; and in the pre-training stage, the original CNN structure is used, and the network model does not have a clear representation of the spatial knowledge and semantic information of the image; at the same time, the existing implicit prompts cannot fully capture the original information of the data source, which is not enough to input representative information of the image into the network, so important information will be lost, resulting in a weakened generalization ability of the model.
[0005] During the training process, the sliding window operation and regular convolution kernel shape of the basic convolution unit of the original CNN will cause the target detection model to be seriously affected by the masked part, and the calculated statistical histogram will seriously deviate from the original data distribution, thus affecting the reconstruction of missing pixels. Summary of the invention
[0006] The purpose of the embodiments of the present application is to provide a method, device, equipment and medium for fine-tuning a target detection model to solve the above-mentioned problems existing in the prior art, and to achieve effective characterization of the data by learning the sparse representation and reconstruction features of the data to be detected; to reduce the fine-tuning training cost and the amount of storage parameters in specific scenarios, and to improve the generalization and scalability in different scenarios.
[0007] In a first aspect, a method for fine-tuning a target detection model is provided, and the method may include:
[0008] Acquire data to be detected including target objects to be detected and target detection general data including multiple types of objects to be detected;
[0009] Using the target detection general data, self-supervised training is performed on a pre-built target detection general network to obtain a trained target detection general network and corresponding network parameters;
[0010] Based on the network parameters, adjusting the model parameters of the pre-built target detection model to obtain an initially adjusted target detection model;
[0011] The initially adjusted target detection model is trained using the data to be detected to obtain a fine-tuned target detection model.
[0012] In an optional implementation, the target detection general data is used to perform self-supervisory training on a pre-built target detection general network to obtain a trained target detection general network and corresponding network parameters, including:
[0013] Dividing the target detection general data into a plurality of pixel blocks;
[0014] Randomly selecting at least one pixel block from the multiple pixel blocks to obtain a target pixel block;
[0015] Performing mask processing on the target pixel block to obtain masked target detection general data;
[0016] The obtained masked universal target detection data are input into the universal target detection network, and the universal target detection network is pre-trained to obtain a pre-trained universal target detection network and corresponding network parameters.
[0017] In an optional implementation, the target detection general network includes a first backbone network, a first neck network and a decoder; wherein the convolution units of the first backbone network and the first neck network are both sparse convolution units;
[0018] The first backbone network is used to extract image features from input masked general object detection data;
[0019] The first neck network is used to integrate the extracted image features to obtain multi-scale feature information;
[0020] A decoder is used to reconstruct an image based on the multi-scale feature information.
[0021] In an optional implementation, the target detection model includes a second backbone network, a second neck network, an explicit cue learning network, an implicit cue learning network and a detection head; wherein the convolution units of the second backbone network and the second neck network are both ordinary convolution units;
[0022] The second backbone network is used to extract image features from the input data to be detected;
[0023] The explicit prompt learning network is used to perform multiple explicit prompt learning on the input data to be detected to obtain multiple explicit prompt vectors of the data to be detected;
[0024] The implicit prompt learning network is used to perform multiple implicit prompt learning on the input data to be detected to obtain multiple implicit prompt vectors of the data to be detected;
[0025] The second neck network is used to integrate the image features extracted by the second backbone network and the added explicit prompt vector and implicit prompt vector to obtain multi-scale feature information;
[0026] The detection head is used to output a target detection result based on the multi-scale feature information and the added explicit prompt vector and implicit prompt vector.
[0027] In an optional implementation, based on the network parameters, adjusting the model parameters of the pre-built target detection model to obtain the initially adjusted target detection model includes:
[0028] Performing convolution parameter conversion on the network parameters to obtain converted network parameters;
[0029] The converted network parameters are used as parameters of the second backbone network and the second neck network to obtain an initially adjusted target detection model.
[0030] In an optional implementation, the initially tuned target detection model is trained using the data to be detected to obtain a fine-tuned target detection model, including:
[0031] Freezing parameters of the second backbone network and the second neck network of the initially adjusted target detection model to obtain a frozen target detection model;
[0032] Determining explicit learning parameters of an explicit cue learning network and implicit learning parameters of an implicit cue learning network based on the number of output channels and spatial feature parameters of the second backbone network of the frozen target detection model;
[0033] Based on the explicit learning parameters, performing explicit prompt learning on the data to be detected to obtain explicit prompt vectors corresponding to different output channels of the second backbone network;
[0034] Based on the implicit learning parameters, implicit prompt learning is performed on the data to be detected to obtain implicit prompt vectors corresponding to different output channels of the second backbone network;
[0035] Inputting the to-be-detected data into the second backbone network of the frozen target detection model to obtain image features output by different output channels of the second backbone network;
[0036] For any output channel, the explicit prompt vector and the implicit prompt vector corresponding to the output channel are added together to obtain a mixed prompt vector;
[0037] Inputting the mixed prompt vector of the output channel and the image features output by the output channel into the second neck network of the frozen object detection model to obtain multi-scale feature information;
[0038] Inputting the multi-scale feature information and the mixed prompt vector into the detection head of the frozen target detection model to obtain the target detection result of the data to be detected, and completing the training of the frozen target detection model by the data to be detected;
[0039] The frozen target detection model trained with the data to be detected is used as the fine-tuned target detection model.
[0040] In an optional implementation, the explicit learning parameters include a plurality of first learning parameters; the implicit learning parameters include a plurality of second learning parameters; wherein each first learning parameter and each second learning parameter respectively corresponds to an output channel of the second backbone network;
[0041] Based on the explicit learning parameters, explicit prompt learning is performed on the data to be detected to obtain explicit prompt vectors corresponding to different output channels of the second backbone network, including:
[0042] Performing grayscale processing on the data to be detected to obtain a grayscale image to be detected;
[0043] Using the Canny edge detection operator to process the grayscale image to be detected, to obtain explicit prompt information for explicit prompt learning;
[0044] For any first learning parameter, based on the first learning parameter, compressing the spatial information of the explicit prompt information to obtain compressed explicit prompt information;
[0045] Performing nonlinear operation on the compressed explicit prompt information to obtain the calculated explicit prompt information;
[0046] Based on the first learning parameter, upsampling and convolution processing are performed on the calculated explicit prompt information to obtain an explicit prompt vector of an output channel corresponding to the first learning parameter;
[0047] Based on the implicit learning parameters, implicit prompt learning is performed on the grayscale image to be detected to obtain implicit prompt vectors corresponding to different output channels of the second backbone network, including:
[0048] For any second learning parameter, initializing the implicit prompt learning network based on the second learning parameter to obtain an initialized implicit prompt learning network;
[0049] Using the initialized linear layer of the implicit cue learning network, autonomous learning of implicit cue learning is performed on the grayscale image to be detected to obtain an implicit cue vector of the output channel corresponding to the second learning parameter.
[0050] In a second aspect, a fine-tuning device for a target detection model is provided, and the device may include:
[0051] An acquisition unit, used for acquiring a data set to be detected containing a target object to be detected and a general data set for target detection containing multiple types of objects to be detected;
[0052] A pre-training unit, used to perform self-supervisory training on a pre-built general target detection network using the general target detection data, to obtain a trained general target detection network and corresponding network parameters;
[0053] An adjustment unit, configured to adjust model parameters of a pre-built target detection model based on the network parameters to obtain a target detection model after initial adjustment;
[0054] The fine-tuning unit is used to train the initially adjusted target detection model using the data to be detected to obtain a fine-tuned target detection model.
[0055] In a third aspect, an electronic device is provided, the electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;
[0056] Memory, used to store computer programs;
[0057] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.
[0058] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.
[0059] This application makes up for the deficiency of implicit information prompts by adding explicit prompts of high-frequency auxiliary information of edge detection operators, which is called hybrid prompt learning. In the explicit prompt part, in order to fuse global information, the downsampling (Down) convolution and then the upsampling (Up) convolution are aligned to realize global information representation. The second backbone network and the second neck network transition part and the output part of the second neck network are fine-tuned with extremely small parameters, which greatly reduces the training cost and the storage cost of the trainable parameters.
[0060] This application constructs a CNN network model architecture with sparse convolution as the basic unit, embeds the backbone network and neck network of the target detection model, and obtains a general network for target detection; uses the general network for target detection to randomly mask the input image blocks, obtains multi-scale feature information at the output end of the general network for target detection, reconstructs the feature information of multi-scale missing pixels, enhances the network subject's understanding of the spatial information and semantic information of the image data, and then trains the parameters of the entire target detection model.
[0061] This application combines two technologies, explicit cue learning and implicit cue learning. Explicit cue learning inputs explicit cue information after edge detection of the data to be detected into the target detection model, while implicit cue learning fine-tunes the target detection model through implicit cue information inside the target detection model. It is an intermediate representation automatically generated by the model in the current task. By considering both explicit and implicit cue information at the same time, a very small number of parameters is used to help the target detection model better understand and solve specific downstream tasks, thereby improving the generalization ability and performance of the target detection model.
[0062] The universal target detection network of the present application adopts sparse convolution as the basic unit. The first neck network of the universal target detection network outputs multi-scale feature reconstruction information. Through multi-scale image reconstruction, the network parameters of the universal target detection network obtained through training can initialize the second backbone network of the target detection model and all basic parameters of the second neck network.
[0063] During the fine-tuning process of the target detection network after the initial adjustment, this application adds explicit prompts of the edge detection operator, thereby making up for the shortcomings of implicit prompts, so that downstream tasks can be well aligned to the pre-training tasks, and improving the detection and application capabilities of the model under specific tasks.
[0064] The target detection general data of the present application is large-scale image data containing multiple categories of objects to be detected in different scenarios. The data contains more samples and richer data distribution, which can help the target detection model to better learn the characteristics and structure of the data, thereby improving the generalization ability and robustness of the target detection model. At the same time, it expands the application scenarios of the target detection model fine-tuning method of the present application and has strong scalability.
[0065] The fine-tuning method of the present application is highly flexible and widely applicable. It can be applied to various character scenes in the field of visual applications. Different downstream tasks can be achieved by replacing the detection head part of the fine-tuning model. That is, the fine-tuning method of the present application can not only fine-tune the target detection model, but also fine-tune the image classification model, semantic segmentation model and image generation model, etc.
[0066] The fine-tuning method of the present application has low training and storage costs. For the data to be tested, only a very small amount of parameters needs to be trained (2% of the original model parameters) and the weight parameters of the trainable part need to be stored, which greatly improves the efficiency of model training and the convenience of deployment. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0068] Figure 1 A flow chart of a method for fine-tuning a target detection model provided in an embodiment of the present application;
[0069] Figure 2 A schematic diagram of the structure of a general target detection network provided in an embodiment of the present application;
[0070] Figure 3 A schematic diagram of the structure of a target detection model provided in an embodiment of the present application;
[0071] Figure 4 A schematic diagram of the structure of a fine-tuning device for a target detection model provided in an embodiment of the present application;
[0072] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0074] Prompt learning: Prompt learning originated from natural language processing tasks. It is a technology used to fine-tune large language models for specific tasks or fields. This involves training the language model by providing specific prompts or examples to it and optimizing its parameters to generate relevant outputs based on these prompts. The model learns to understand the context provided in the prompt and generates responses accordingly. When transitioning to visual models, prompt learning is often used as a fine-tuning technique for large models. Corresponding to tasks in different scenarios, prompt learning freezes the original backbone network and trains with a very small amount of trainable parameters to reduce training costs and fine-tune on specific tasks.
[0075] Self-supervision: Self-supervised learning is a learning method that uses the data itself to generate labels or supervisory signals to train the model without manually annotated labels. It is a special form of supervised learning in which the model learns from the input data without the help of external labels or human annotations. The core idea of self-supervised learning is to use the intrinsic structure or information of the data itself to guide the learning process. In visual tasks, self-supervised learning can be achieved by designing some tasks or goals and letting the model try to solve these tasks without the need for external labels. The design of these tasks requires the use of some kind of structure or contextual information in the image data so that the model can learn useful representations or features from it, thereby improving the understanding and processing capabilities of the image content.
[0076] CNN object detection: CNN object detection refers to the use of convolutional neural networks (CNN) to detect target objects. This method is often used in applications such as autonomous driving systems or traffic monitoring systems to identify and locate objects such as vehicles, pedestrians, and signs on the road.
[0077] The fine-tuning method of the target detection model provided in the embodiment of the present application can be applied in a server or in a terminal with strong computing power. The server can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminal can be a user equipment (User Equipment, UE) such as a mobile phone, a smart phone, a laptop, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing equipment connected to a wireless modem, a mobile station (Mobile Station, MS), a mobile terminal (Mobile Terminal), etc. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application is not limited here.
[0078] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.
[0079] Figure 1 The following is a flow chart of a method for fine-tuning a target detection model provided in an embodiment of the present application. Figure 1 As shown, the method may include:
[0080] Step S110, obtaining data to be detected including target objects to be detected and target detection general data including multiple types of objects to be detected; using the target detection general data, performing self-supervised training on a pre-built target detection general network to obtain a trained target detection general network and corresponding network parameters.
[0081] In the embodiment of the present application, the data to be detected is composed of images of the target object to be detected of the target detection model; the target detection general data includes not only the target object to be detected of the target detection model, but also multiple types of objects to be detected in the application scenario of the target detection model. For example, if the target object to be detected of the target detection model is a license plate, that is, the target detection model is a license plate detection model, then the data to be detected is the license plate image data; and the target detection general data is the image data of the license plate (target detection object), pedestrians (object to be detected but not the target detection object) and traffic signs (object to be detected but not the target detection object) in the traffic scenario where the license plate detection model is applied.
[0082] In an embodiment of the present application, a method for acquiring general target detection data includes:
[0083] Collect original image data containing multiple types of objects to be detected in different environments; for example, if the application scenario of the target detection model is a traffic scenario, and the target detection model is a detection model for license plates or pedestrians, then obtain original image data containing multiple types of detection objects under different weather conditions (sunny, cloudy, rainy, etc.), different lighting conditions (day, night, etc.), different road types (highways, city streets, rural roads, etc.), and different seasons (spring, summer, autumn, and winter);
[0084] The collected original image data is labeled to mark the position and category information of various objects to be detected in the image; the category information is the category of the object to be detected; for example, if the application scenario of the target detection model is a traffic scene, if the application scenario of the target detection model is a traffic scene, and the target detection model is a detection model for license plates or pedestrians, then the position and category information of vehicles, pedestrians, traffic signs, etc. in the original image are labeled.
[0085] In an embodiment of the present application, after obtaining the data to be detected and the general data for target detection, it is necessary to perform data enhancement on the data to be detected and the general data for target detection respectively, so as to improve the generalization and robustness of the detection performance of the target detection model.
[0086] In an embodiment of the present application, a method for data enhancement includes: obtaining data to be enhanced (data to be detected or general data for target detection), multiple enhancement methods, and probability factors and expansion coefficients of different enhancement methods; using one or more enhancement methods, based on the probability factors and expansion coefficients corresponding to different enhancement methods, the data to be enhanced is enhanced to obtain enhanced data. The present application limits image enhancement to an ultra-range range by presetting image enhancement hyperparameters (i.e., probability factors and expansion sparsity), so that when the enhanced data is used to train a target detection model or a general network for target detection, the distribution of image data loaded in each round of training is different from that in the previous round, thereby effectively improving the diversity of the data.
[0087] In one embodiment of the present application, the values of probability factors and expansion coefficients of different enhancement modes may be as shown in Table 1:
[0088] Table 1 Probability factors and expansion coefficients of different enhancement methods
[0089]
[0090]
[0091] In the embodiments of the present application, Figure 2 As shown, the general network for target detection is a CNN neural network, including a first backbone network, a first neck network and a decoder; wherein the convolution units of the first backbone network and the first neck network are both sparse convolution units;
[0092] The first backbone network is used to extract image features from the input masked object detection general data;
[0093] The first neck network is used to integrate the extracted image features to obtain multi-scale feature information;
[0094] Decoder, used to reconstruct images based on multi-scale feature information.
[0095] In the embodiment of the present application, the convolution units of the first backbone network and the first neck network are both sparse convolution units; sparse convolution does not calculate masked pixels, and thus can improve the effect of image reconstruction.
[0096] In the embodiment of the present application, the pre-built general target detection network is self-supervised trained using general target detection data to obtain a trained general target detection network and corresponding network parameters, including:
[0097] The target detection general data is divided into a plurality of pixel blocks; at least one pixel block is randomly selected from the plurality of pixel blocks to obtain a target pixel block; the target pixel block is masked to obtain masked target detection general data; each masked target detection general data is input into a target detection general network, and the target detection general network is pre-trained to obtain a pre-trained target detection general network and corresponding network parameters.
[0098] Step S120: Based on the network parameters, the model parameters of the pre-built target detection model are adjusted to obtain the initially adjusted target detection model; the initially adjusted target detection model is trained using the data to be detected to obtain the fine-tuned target detection model.
[0099] like Figure 3 As shown, in the embodiment of the present application, the target detection model is a CNN neural network, including a second backbone network, a second neck network, an explicit prompt learning network, an implicit prompt learning network and a detection head; wherein the convolution units of the second backbone network and the second neck network are both ordinary convolution units;
[0100] The second backbone network is used to extract image features from the input data to be detected;
[0101] An explicit prompt learning network is used to perform multiple explicit prompt learning on the input data to be detected to obtain multiple explicit prompt vectors of the data to be detected;
[0102] An implicit prompt learning network is used to perform multiple implicit prompt learning on the input data to be detected to obtain multiple implicit prompt vectors of the data to be detected;
[0103] The second neck network is used to integrate the image features extracted by the second backbone network and the added explicit prompt vector and implicit prompt vector to obtain multi-scale feature information;
[0104] The detection head is used to output the target detection result based on the multi-scale feature information and the added explicit hint vector and implicit hint vector.
[0105] In an embodiment of the present application, the target detection model includes a second backbone network, a second neck network, an explicit prompt learning network, an implicit prompt learning network and a detection head; the parameters of the second backbone network and the second neck network in the target detection model after initial adjustment have been frozen (i.e. fixed and no longer involved in training and updating), so the data to be detected only needs to train the parameters of the explicit prompt learning network, the implicit prompt learning network and the detection head, thereby realizing fine-tuning of the target detection model after initial adjustment using the data set to be detected (i.e., the target object to be detected containing the target detection model).
[0106] In the embodiment of the present application, based on the network parameters, the model parameters of the pre-built target detection model are adjusted to obtain the initially adjusted target detection model, including:
[0107] The network parameters are subjected to convolution parameter conversion to obtain converted network parameters; the converted network parameters are used as parameters of the second backbone network and the second neck network to obtain the initially adjusted target detection model.
[0108] In an embodiment of the present application, since the first backbone network and the first neck network of the target detection general network use sparse convolution as basic units, and the second backbone network and the second neck network of the target detection model use ordinary convolution as basic units, the network parameters of the target detection general network cannot be directly applied to the target detection model. Therefore, it is necessary to perform convolution parameter conversion on the obtained network parameters, and then input the converted network parameters into the second backbone network and the second neck network, and freeze all parameters of the second backbone network and the second neck network. After freezing, all parameters of the second backbone network and the second neck network are non-trainable parameters.
[0109] In the embodiment of the present application, the parameters of the network are frozen, that is, the parameters of the corresponding network are not updated.
[0110] In the embodiment of the present application, the initially adjusted target detection model is trained using the data to be detected to obtain a fine-tuned target detection model, including:
[0111] Freeze the parameters of the second backbone network and the second neck network of the initially adjusted target detection model to obtain the frozen target detection model; determine the explicit learning parameters of the explicit cue learning network and the implicit learning parameters of the implicit cue learning network based on the number of output channels and spatial feature parameters of the second backbone network of the frozen target detection model; perform explicit cue learning on the data to be detected based on the explicit learning parameters to obtain explicit cue vectors corresponding to different output channels of the second backbone network;
[0112] Based on the implicit learning parameters, implicit prompt learning is performed on the data to be detected to obtain implicit prompt vectors corresponding to different output channels of the second backbone network; the data to be detected is input into the second backbone network of the frozen target detection model to obtain image features output by different output channels of the second backbone network; for any output channel, the explicit prompt vector and the implicit prompt vector corresponding to the output channel are added to obtain a mixed prompt vector; the mixed prompt vector of the output channel and the image features output by the output channel are input into the second backbone network of the frozen target detection model to obtain multi-scale feature information;
[0113] The multi-scale feature information and the mixed prompt vector are input into the detection head of the frozen target detection model to obtain the target detection result of the data to be detected, and the training of the frozen target detection model on the data to be detected is completed; the frozen target detection model that has completed the training of the data to be detected is used as the fine-tuned target detection model.
[0114] In an embodiment of the present application, the explicit learning parameters include multiple first learning parameters; the implicit learning parameters include multiple second learning parameters; wherein each first learning parameter and each second learning parameter respectively corresponds to an output channel of the second trunk network and the second neck network.
[0115] In the embodiment of the present application, since the second backbone network and the second neck network both have multiple output channels, and each output channel of the second backbone network and each output channel of the second neck network needs to add explicit prompt vectors and implicit prompt vectors, that is, it is necessary to align the explicit prompt vectors and implicit prompt vectors with the output channels of the second backbone network and the second neck network in spatial dimensions; therefore, the embodiment of the present application determines the first learning parameters and the second learning parameters corresponding to each output channel according to the spatial characteristics of each output channel of the second backbone network and the second neck network.
[0116] In the embodiment of the present application, based on the explicit learning parameters, explicit prompt learning is performed on the data to be detected to obtain explicit prompt vectors corresponding to different output channels of the second backbone network, including:
[0117] The data to be detected is gray-scaled to obtain a gray-scale image to be detected; the gray-scale image to be detected is processed by using a Canny edge detection operator to obtain explicit prompt information of explicit prompt learning; for any first learning parameter, based on the first learning parameter, the explicit prompt information is spatially compressed to obtain compressed explicit prompt information; nonlinear operation is performed on the compressed explicit prompt information to obtain the explicit prompt information after operation; based on the first learning parameter, upsampling and convolution processing are performed on the explicit prompt information after operation to obtain the explicit prompt vector of the output channel corresponding to the first learning parameter.
[0118] In the embodiment of the present application, the grayscale image to be detected is processed using the Canny edge detection operator to obtain explicit prompt information for explicit prompt learning, including:
[0119] Perform edge detection on the grayscale image to be detected using the Canny edge detection operator to obtain an edge detection image (including the grayscale value of each edge point) corresponding to the grayscale image to be detected; obtain a preset judgment threshold and judgment rule; based on the judgment threshold, the judgment rule and the edge detection image, obtain explicit prompt information for explicit prompt learning; the explicit prompt information may include the thickness of the edge line and / or the depth of the edge line and the shape of the edge line.
[0120] In the embodiment of the present application, based on the implicit learning parameters, implicit prompt learning is performed on the grayscale image to be detected to obtain implicit prompt vectors corresponding to different output channels of the second backbone network, including:
[0121] For any second learning parameter, based on the second learning parameter, the implicit prompt learning network is initialized to obtain the initialized implicit prompt learning network; the linear layer of the initialized implicit prompt learning network is used to perform autonomous learning of implicit prompt learning on the grayscale image to be detected to obtain the implicit prompt vector of the output channel corresponding to the second learning parameter.
[0122] In the embodiment of the present application, the method for determining the explicit hint vector may adopt the following steps:
[0123] The data to be detected is grayed and converted into a single-channel image. The Canny edge detection operator is then used to process the grayscale image to obtain prompt information for display prompt learning. The prompt information is subjected to a down-sampling convolution operation with a step size of 16 to compress spatial information (corresponding to different output channels, the parameters of spatial information compression are different, that is, the first learning parameters are different), the number of output channels is aligned with the outputs at different positions of the backbone network, and then a nonlinear activation function is applied to perform nonlinear operations. Finally, up-sampling Up operations and convolution operations of different degrees (corresponding to different output channels) are performed to align the spatial dimensions of the outputs at different positions of the backbone network to obtain the explicit prompt vector for explicit prompt learning.
[0124] In one embodiment of the present application, the second backbone network and the second neck network include three output channels, so the method for determining the explicit cue vector needs to be repeated three times to obtain explicit cue vectors corresponding to the number of channels and spatial scales at different positions of the backbone network.
[0125] In the embodiment of the present application, the method for determining the implicit hint vector may adopt the following steps:
[0126] Initialize trainable parameters of different shapes (i.e., parameters of explicit cue learning network, implicit cue learning network, and detection head), corresponding to the feature information dimensions at different positions of the backbone network, and perform autonomous learning of implicit cue learning with a linear layer to obtain the cue vector for implicit cue learning. This operation is also performed three times, corresponding to the number of channels and spatial scales at different positions of the backbone network.
[0127] In an embodiment of the present application, since the output channels of the second backbone network correspond one-to-one to the output channels of the second neck network, their spatial feature parameters are also the same; therefore, after obtaining the explicit prompt vector or implicit prompt vector corresponding to each output channel of the second backbone network, it can be directly matched one-to-one with each output channel of the second neck network and added to the multi-scale features output by each output channel of the corresponding second neck network.
[0128] In an embodiment of the present application, after obtaining the mixed prompt vectors corresponding to each output channel of the second backbone network and the second neck network, the main elements of the mixed prompt vectors are added to the preset positions; thereby only training the parameters of the explicit prompt learning network, the implicit prompt learning network and the detection head is achieved, thereby achieving fine-tuning training of the target detection model after initial adjustment on the data to be detected.
[0129] In the embodiment of the present application, the preset position is a transition portion between the second trunk network and the second neck network and an output portion of the second neck network.
[0130] In the embodiment of the present application, for the hardware equipment and training strategy, the Ubuntu 18.04LTS system is used, the graphics card is GeForce RTX 4090Ti, the network development framework is pytorch2.0.1, and the integrated development environment is Pycharm; the training rounds are uniformly set to 300, the batch size is 4, the input image size is 1280×1280, the AdamW optimizer is used, the initial learning rates of the encoder and decoder are set to 0.002 and 0.01, and the linear decay learning rate adjustment strategy is used, and the weight decay is set to 0.05.
[0131] In an embodiment of the present application, after obtaining the fine-tuned target detection model, the target detection model can be used to perform target detection on the target to be detected, and the method may include:
[0132] Acquire an image to be detected; resize the image to be detected based on a preset size to obtain a resized image to be detected; input the resized image to be detected into a fine-tuned target detection model, and the detection head of the fine-tuned target detection model obtains a corresponding detection result through an appropriate or preset prediction frame decoding method.
[0133] Corresponding to the above method, the embodiment of the present application also provides a fine-tuning device for a target detection model, such as Figure 4 As shown, the fine-tuning device of the target detection model includes:
[0134] An acquisition unit 410 is used to acquire a data set to be detected containing a target object to be detected and a general data set for target detection containing multiple types of objects to be detected;
[0135] A pre-training unit 420 is used to perform self-supervisory training on a pre-built general target detection network using general target detection data to obtain a trained general target detection network and corresponding network parameters;
[0136] An adjustment unit 430 is used to adjust the model parameters of the pre-built target detection model based on the network parameters to obtain a target detection model after initial adjustment;
[0137] The fine-tuning unit 440 is used to train the initially tuned target detection model using the data to be detected to obtain a fine-tuned target detection model.
[0138] The functions of each functional unit of the fine-tuning device of the target detection model provided in the above-mentioned embodiments of the present application can be realized through the above-mentioned method steps. Therefore, the specific working process and beneficial effects of each unit in the fine-tuning device of the target detection model provided in the embodiments of the present application will not be repeated here.
[0139] The present application also provides an electronic device, such as Figure 5 As shown, it includes a processor 510 , a communication interface 520 , a memory 530 and a communication bus 540 , wherein the processor 510 , the communication interface 520 , and the memory 530 communicate with each other via the communication bus 540 .
[0140] Memory 530, for storing computer programs;
[0141] The processor 510 is used to execute the program stored in the memory 530 to implement the following steps:
[0142] Obtain a data set to be detected containing a target object to be detected and a general target detection data set containing multiple types of objects to be detected;
[0143] Using the universal target detection data, the pre-built universal target detection network is trained through self-supervision to obtain the trained universal target detection network and the corresponding network parameters.
[0144] Based on the network parameters, the model parameters of the pre-built target detection model are adjusted to obtain the initially adjusted target detection model;
[0145] The initially tuned target detection model is trained using the data to be detected to obtain a fine-tuned target detection model.
[0146] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0147] The communication interface is used for communication between the above electronic device and other devices.
[0148] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0149] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0150] The implementation methods and beneficial effects of the components of the electronic device in the above embodiments to solve the problems can be seen in Figure 1 The various steps in the illustrated embodiment are implemented, therefore, the specific working process and beneficial effects of the electronic device provided by the embodiment of the present application are not repeated here.
[0151] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the method for fine-tuning the target detection model in any one of the above embodiments.
[0152] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the target detection model fine-tuning methods in the above embodiments.
[0153] Those skilled in the art will appreciate that the embodiments in the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt a complete hardware embodiment, a complete software embodiment, or a form of an embodiment combining software and hardware. Moreover, the present application may adopt a form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0154] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0157] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.
[0158] Obviously, those skilled in the art can make various changes and modifications to the embodiments in the present application without departing from the spirit and scope of the embodiments in the present application. Thus, if these modifications and variations of the embodiments in the present application fall within the scope of the claims and their equivalents in the embodiments of the present application, the embodiments of the present application are also intended to include these modifications and variations.
Claims
1. A fine-tuning method for a target detection model, characterized in that: The method comprises: Acquire data to be detected including target objects to be detected and target detection general data including multiple types of objects to be detected; Using the target detection general data, self-supervised training is performed on a pre-built target detection general network to obtain a trained target detection general network and corresponding network parameters; Based on the network parameters, adjusting the model parameters of the pre-built target detection model to obtain an initially adjusted target detection model; The initially adjusted target detection model is trained using the data to be detected to obtain a fine-tuned target detection model.
2. The method according to claim 1, characterized in that Using the target detection general data, self-supervised training is performed on the pre-built target detection general network to obtain a trained target detection general network and corresponding network parameters, including: Dividing the target detection general data into a plurality of pixel blocks; Randomly selecting at least one pixel block from the multiple pixel blocks to obtain a target pixel block; Performing mask processing on the target pixel block to obtain masked target detection general data; The obtained masked universal target detection data are input into the universal target detection network, and the universal target detection network is pre-trained to obtain a pre-trained universal target detection network and corresponding network parameters.
3. The method according to claim 2, characterized in that The target detection general network comprises a first backbone network, a first neck network and a decoder; wherein the convolution units of the first backbone network and the first neck network are both sparse convolution units; The first backbone network is used to extract image features from input masked general object detection data; The first neck network is used to integrate the extracted image features to obtain multi-scale feature information; A decoder is used to reconstruct an image based on the multi-scale feature information.
4. The method according to claim 1, characterized in that The target detection model comprises a second backbone network, a second neck network, an explicit cue learning network, an implicit cue learning network and a detection head; wherein the convolution units of the second backbone network and the second neck network are both common convolution units; The second backbone network is used to extract image features from the input data to be detected; The explicit prompt learning network is used to perform multiple explicit prompt learning on the input data to be detected to obtain multiple explicit prompt vectors of the data to be detected; The implicit prompt learning network is used to perform multiple implicit prompt learning on the input data to be detected to obtain multiple implicit prompt vectors of the data to be detected; The second neck network is used to integrate the image features extracted by the second backbone network and the added explicit prompt vector and implicit prompt vector to obtain multi-scale feature information; The detection head is used to output a target detection result based on the multi-scale feature information and the added explicit hint vector and implicit hint vector.
5. The method according to claim 4, characterized in that Based on the network parameters, the model parameters of the pre-built target detection model are adjusted to obtain the initially adjusted target detection model, including: Performing convolution parameter conversion on the network parameters to obtain converted network parameters; The converted network parameters are used as parameters of the second backbone network and the second neck network to obtain an initially adjusted target detection model.
6. The method according to claim 4, characterized in that Using the data to be detected, training the initially adjusted target detection model to obtain a fine-tuned target detection model includes: Freezing parameters of the second backbone network and the second neck network of the initially adjusted target detection model to obtain a frozen target detection model; Determining explicit learning parameters of an explicit cue learning network and implicit learning parameters of an implicit cue learning network based on the number of output channels and spatial feature parameters of the second backbone network of the frozen target detection model; Based on the explicit learning parameters, performing explicit prompt learning on the data to be detected to obtain explicit prompt vectors corresponding to different output channels of the second backbone network; Based on the implicit learning parameters, implicit prompt learning is performed on the data to be detected to obtain implicit prompt vectors corresponding to different output channels of the second backbone network; Inputting the to-be-detected data into the second backbone network of the frozen target detection model to obtain image features output by different output channels of the second backbone network; For any output channel, the explicit prompt vector and the implicit prompt vector corresponding to the output channel are added together to obtain a mixed prompt vector; Inputting the mixed prompt vector of the output channel and the image features output by the output channel into the second neck network of the frozen object detection model to obtain multi-scale feature information; Inputting the multi-scale feature information and the mixed prompt vector into the detection head of the frozen target detection model to obtain the target detection result of the data to be detected, and completing the training of the frozen target detection model by the data to be detected; The frozen target detection model trained with the data to be detected is used as the fine-tuned target detection model.
7. The method according to claim 6, characterized in that The explicit learning parameters include a plurality of first learning parameters; the implicit learning parameters include a plurality of second learning parameters; wherein each first learning parameter and each second learning parameter respectively corresponds to an output channel of the second backbone network; Based on the explicit learning parameters, explicit prompt learning is performed on the data to be detected to obtain explicit prompt vectors corresponding to different output channels of the second backbone network, including: Performing grayscale processing on the data to be detected to obtain a grayscale image to be detected; Using the Canny edge detection operator to process the grayscale image to be detected, to obtain explicit prompt information for explicit prompt learning; For any first learning parameter, based on the first learning parameter, compressing the spatial information of the explicit prompt information to obtain compressed explicit prompt information; Performing nonlinear operation on the compressed explicit prompt information to obtain the operated explicit prompt information; Based on the first learning parameter, upsampling and convolution processing are performed on the calculated explicit prompt information to obtain an explicit prompt vector of an output channel corresponding to the first learning parameter; Based on the implicit learning parameters, implicit prompt learning is performed on the grayscale image to be detected to obtain implicit prompt vectors corresponding to different output channels of the second backbone network, including: For any second learning parameter, initializing the implicit prompt learning network based on the second learning parameter to obtain an initialized implicit prompt learning network; Using the initialized linear layer of the implicit cue learning network, autonomous learning of implicit cue learning is performed on the grayscale image to be detected to obtain an implicit cue vector of the output channel corresponding to the second learning parameter.
8. A fine-tuning device for a target detection model, characterized in that: The device comprises: An acquisition unit, used for acquiring a data set to be detected containing a target object to be detected and a general data set for target detection containing multiple types of objects to be detected; A pre-training unit, used to perform self-supervisory training on a pre-built general target detection network using the general target detection data, to obtain a trained general target detection network and corresponding network parameters; An adjustment unit, configured to adjust model parameters of a pre-built target detection model based on the network parameters to obtain a target detection model after initial adjustment; The fine-tuning unit is used to train the initially adjusted target detection model using the data to be detected to obtain a fine-tuned target detection model.
9. An electronic device, characterized in that: The electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-7 when executing a program stored in a memory.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
