Fine-tuning-based cross-scenario object detection method and system, device, and medium

Abstract

The present application relates to a fine-tuning-based cross-scenario object detection method and system, a device, and a medium. The method comprises: improving a backbone network of a YOLOv8 algorithm by means of Transformer modules, and inputting an image in a first scenario into an optimized YOLOv8 model for training to obtain a first object detection model; embedding a LORA model into each Transformer module in the first object detection model, training the model on the basis of an image in a second scenario, fixing parameters of the first object detection model, and training model parameters of each LORA model; and on the basis of the trained model parameters of each LORA model, fine-tuning a weight parameter generated by each Transformer module in the first object detection model, to obtain a second object detection model applicable to the second scenario. The present application enables object detection models to adapt to different actual application scenarios within a short period of time, thereby improving the accuracy and efficiency of cross-scenario object detection and recognition.

Description

A cross-scene target detection method, system, device, and medium based on fine-tuning Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a cross-scene target detection method, system, device and medium based on fine-tuning. Background Technology

[0002] In surveillance scenarios, the application scenarios of target detection algorithms often change continuously. Therefore, it is of great significance to quickly improve the performance of the model in new scenarios.

[0003] Existing methods have the following problems: when dealing with cross-scene situations, a lot of time is often required to train the object detection model, and the model cannot be quickly fine-tuned and updated in new scenes; the accuracy of lightweight object detection algorithms cannot be guaranteed; the model fine-tuning part only trains the classifier and regressor, without adjusting the backbone network, thus limiting the model's capabilities. Summary of the Invention

[0004] Therefore, it is necessary to provide a cross-scene target detection method, system, device, and medium based on fine-tuning to address the aforementioned technical problems.

[0005] In a first aspect, embodiments of this application provide a cross-scene target detection method based on fine-tuning, the method comprising:

[0006] The first scene image is input into the YOLOv8 optimization model for training to obtain the first object detection model. The YOLOv8 optimization model includes a connected backbone network and a head network. The backbone network includes multiple Transformer modules, each of which performs hierarchical feature extraction based on the first scene image to obtain a first feature map and generates weight parameters. The head network is used to perform feature fusion on the first feature map and then perform object detection to obtain the first object detection result.

[0007] A LoRA model is embedded in each of the Transformer modules in the first object detection model. The second scene image in the second scene is input into the first object detection model embedded with the LoRA model. The parameters of the first object detection model are fixed, and the model parameters of each LoRA model are trained. Based on the trained model parameters of each LoRA model, the weight parameters generated by each Transformer module in the first object detection model are fine-tuned to obtain the second object detection model.

[0008] The target image to be detected in the second scene is input into the second target detection model to obtain the second target detection result.

[0009] In one embodiment, the backbone network further includes multiple Patch Merging layers, each Patch Merging layer being connected to the output of each Transformer module, wherein:

[0010] The Patch Merging layer is used to downsample the first feature map output by the connected Transformer module to obtain a second feature map, and then input the second feature map into the next Transformer module for processing.

[0011] The Transformer module is used to sequentially perform standard convolution, point convolution, unfold, encode, fold, point convolution, concatenation, and standard convolution operations on the input second feature map.

[0012] In one embodiment, the step of fixing the parameters of the first target detection model and training the model parameters of each of the LORA models includes:

[0013] Based on the weight parameters generated by each Transformer module, initialize the model parameters of each corresponding LORA model;

[0014] During the iterative training process of inputting the second scene image in the second scene into the first object detection model embedded in the LoRA model, the parameters of the first object detection model are fixed, and the model parameters of each LoRA model are trained.

[0015] In one embodiment, the model parameters of the LORA model include a dimensionality reduction matrix and an dimensionality increase matrix; the fine-tuning of the weight parameters generated by each Transformer module in the first object detection model based on the model parameters of each trained LORA model includes:

[0016] The weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the following formula: W = W0 + BA

[0017] Wherein, W0 is the weight parameter generated by each Transformer module, A is the dimensionality reduction matrix, B is the dimensionality increase matrix, and W is the weight parameter generated by each Transformer module after fine-tuning.

[0018] In one embodiment, the weight parameters generated by each of the Transformer modules are calculated using the following formula:

[0019] Where Q, K, and V are the weight parameters generated by the Transformer module, and d is the dimension of the second feature map input to each Transformer module.

[0020] In one embodiment, the method further includes:

[0021] During each iteration of training, the weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the updated model parameters of each LORA model.

[0022] Based on the first target detection model with the fine-tuned weight parameters of each Transformer module, target detection is performed on the second scene image in the second scene to obtain the evaluation index.

[0023] When the evaluation index meets the preset model accuracy, the second target detection model is obtained.

[0024] In one embodiment, the backbone network further includes a convolutional layer connected to the input of the backbone network, the convolutional layer being used for:

[0025] Perform standard convolution processing on the first scene image under the first scene input.

[0026] Secondly, embodiments of this application also provide a cross-scene target detection system based on fine-tuning, the system comprising:

[0027] The first scene model training module is used to input the first scene image under the first scene into the YOLOv8 optimization model for training to obtain the first object detection model; wherein, the YOLOv8 optimization model includes a connected backbone network and a head network, the backbone network includes multiple Transformer modules, each Transformer module performs hierarchical feature extraction based on the first scene image to obtain a first feature map and generates weight parameters; the head network is used to perform feature fusion on the first feature map and then perform object detection to obtain the first object detection result;

[0028] The second scene model training module is used to embed LoRA models into each of the Transformer modules in the first object detection model, input the second scene image in the second scene into the first object detection model embedded with the LoRA model, fix the parameters of the first object detection model, train the model parameters of each of the LoRA models, and fine-tune the weight parameters generated by each of the Transformer modules in the first object detection model based on the trained model parameters of each of the LoRA models to obtain the second object detection model.

[0029] The detection module is used to input the target image to be detected in the second scene into the second target detection model to obtain the second target detection result.

[0030] Thirdly, embodiments of this application also provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method described in the first aspect above.

[0031] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the method described in the first aspect above.

[0032] Compared with existing technologies, this application improves the backbone network of the YOLOv8 algorithm based on the Transformer module, thereby improving the accuracy of the object detection model and facilitating further fine-tuning. For the improved YOLOv8 object detection algorithm, a LoRA-based fine-tuning model is constructed to achieve efficient and rapid fine-tuning training of the object detection model. This allows the model to adapt to different real-world application scenarios in a short time, improving the efficiency of object detection and quickly enhancing the model's performance in new scenarios.

[0033] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0035] Figure 1 is a hardware structure block diagram of a terminal device based on a fine-tuning cross-scene target detection method in one embodiment;

[0036] Figure 2 is a flowchart illustrating a cross-scene target detection method based on fine-tuning in one embodiment;

[0037] Figure 3 is a schematic diagram of the network structure of the YOLOv8 optimization model in one embodiment;

[0038] Figure 4 is a schematic diagram of the processing flow of the Transformer module in one embodiment;

[0039] Figure 5 is a flowchart illustrating the process of fine-tuning the weight parameter Q based on the LoRA model in one embodiment;

[0040] Figure 6 is a flowchart illustrating the fine-tuning process of a target detection model based on the LoRA model across different scenarios in one embodiment.

[0041] Figure 7 is a structural block diagram of a cross-scene target detection system based on fine-tuning in one embodiment;

[0042] Figure 8 is a schematic diagram of the computer device structure in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0044] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0045] The method embodiments provided in this example can be executed in a terminal, computer, or similar computing device. For example, when running on a terminal, Figure 1 is a hardware structure block diagram of the terminal of the fine-tuning cross-scene target detection method of this embodiment. As shown in Figure 1, the terminal may include one or more processors 102 (only one is shown in Figure 1) and a memory 104 for storing data, wherein the processor 102 may be, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. It will be understood by those skilled in the art that the structure shown in Figure 1 is only illustrative and does not limit the structure of the terminal. For example, the terminal may also include more or fewer components than shown in Figure 1, or have a different configuration than that shown in Figure 1.

[0046] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the fine-tuning cross-scene target detection method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module for wireless communication with the Internet.

[0048] The following explains the terminology used in this application:

[0049] Object detection: The goal of object detection is to identify the specific category and location of objects in a given image. Since the application scenarios and training scenarios of object detection algorithms often differ, rapidly improving the model's performance in new scenarios is of great importance.

[0050] Fine-tuning: Algorithm fine-tuning aims to further train a base model that has already been trained on the original dataset on a new task, thereby adapting the model to the requirements of the new task.

[0051] This application provides a cross-scene target detection method based on fine-tuning. Taking the application of this method to the terminal in Figure 1 as an example, as shown in Figure 2, the method includes the following steps:

[0052] Step 201: Input the first scene image under the first scene into the YOLOv8 optimization model for training to obtain the first object detection model; wherein, the YOLOv8 optimization model includes a connected backbone network and a head network, the backbone network includes multiple Transformer modules, each Transformer module performs hierarchical feature extraction based on the first scene image to obtain a first feature map and generates weight parameters; the head network is used to perform feature fusion on the first feature map and then perform object detection to obtain the first object detection result.

[0053] In this application, the backbone network of the existing YOLOv8 algorithm is improved based on the Transformer structure, which can improve the accuracy of the target detection model and facilitate the fine-tuning of the target detection model in the future.

[0054] Step 202: Embed LoRA models in each Transformer module of the first object detection model, input the second scene image in the second scene into the first object detection model embedded with the LoRA model, fix the parameters of the first object detection model, and train the model parameters of each LoRA model; fine-tune the weight parameters generated by each Transformer module in the first object detection model based on the trained model parameters of each LoRA model to obtain the second object detection model.

[0055] Specifically, embedding the LoRA model in each Transformer module of the first object detection model is equivalent to adding a LoRA model plugin next to each Transformer module structure in the first object detection model.

[0056] Step 203: Input the target image to be detected in the second scene into the second target detection model to obtain the second target detection result.

[0057] The first scenario and the second scenario are different scenarios. For example, the first scenario may be daytime and the second scenario may be nighttime; or the first scenario may be outdoors and the second scenario may be indoors. The targets detected by the first target detection model and the second target detection model are the same, for example, both are vehicles or pedestrians in the detection scenario.

[0058] In the aforementioned cross-scene object detection method based on fine-tuning, the backbone network of the YOLOv8 algorithm is improved by using the Transformer module, which improves the accuracy of the object detection model and facilitates further fine-tuning. For the improved YOLOv8 object detection algorithm, a LoRA-based fine-tuning model is constructed to achieve efficient and rapid fine-tuning training of the object detection model. This allows the model to adapt to different real-world application scenarios in a short time, improving the efficiency of object detection and quickly enhancing the model's performance in new scenarios.

[0059] In one embodiment, as shown in Figure 3, the backbone network of the YOLOv8 optimization model further includes multiple Patch Merging layers, each of which is connected to the output of each Transformer module, wherein:

[0060] The Patch Merging layer is used to downsample the first feature map output by the connected Transformer module to obtain a second feature map, and then input the second feature map into the next Transformer module for processing.

[0061] The Transformer module is used to sequentially perform standard convolution, point convolution, unfold, encode, fold, point convolution, concatenation, and standard convolution operations on the input second feature map.

[0062] Furthermore, as shown in Figure 3, the backbone network also includes a convolutional layer connected to the input end of the backbone network. The convolutional layer is used to perform standard convolution processing on the input first scene image under the first scene.

[0063] The following section, using the YOLOv8 optimized model structure shown in Figure 3, will provide a detailed explanation of how to train the first object detection model within the YOLOv8 optimized model.

[0064] The first scene image in the first scene is input into the backbone network of the YOLOv8 optimization model. The input image first undergoes a standard convolution with a stride of 2 and a kernel of 3, which reduces the length H and width W of the image by a factor of 2. Then it is input into the Transformer module for processing.

[0065] The processing flow of the Transformer module is shown in Figure 4:

[0066] For a feature map with input size H×W×C, an N×N standard convolution (typically 3×3) is first applied, followed by a 1×1 dotted convolution to increase the channel dimension of the feature map to d dimensions. The unfold operation expands the feature map to P×N×d, where P = wh, N = HW / P is the number of patches, and h and w are the height and width of the patch, respectively. After unfolding, the patched feature map is flattened, i.e., the P×N×d feature map is flattened to N×(Pd), reducing the dimension to N vectors of size P×d. The flattened sequence is then input into a standard Transformer encoder, and the output feature map after encoding is of size N×(Pd). Finally, it is restored to P×N×d. The fold operation is the opposite of the unfold operation, restoring the feature map to H×W×d. Then, a dotted convolution projects the feature map to a lower-dimensional space (C dimensions), and a concat operation combines it with the input feature map. Finally, another n×n convolutional layer is used to fuse local and global features.

[0067] The weight parameters generated by each Transformer module are calculated using the following formula:

[0068] Where Q, K, and V are the weight parameters generated by the Transformer module, and d is the dimension of the second feature map input to each Transformer module.

[0069] The purpose of the Patch Merging layer is to reduce the resolution and adjust the number of output channels, thereby saving subsequent computation. Each operation selects elements at intervals of 2 in both the row and column directions; these elements are concatenated together as a single tensor, and then expanded. At this point, the channel dimension becomes four times the original, and H and W are each reduced by a factor of 2; finally, a fully connected layer is used to adjust the channel dimension back to twice its original size.

[0070] The C2f module in the head network structure is designed to obtain richer gradient flow information while ensuring the model is lightweight. The detection module is based on the anchor-free method and transforms the original coupled head into a decoupled head. The target detection structure is finally obtained through the detection module.

[0071] In one embodiment, the step of fixing the parameters of the first target detection model and training the model parameters of each of the LORA models includes the following:

[0072] Step 301: Initialize the model parameters of each corresponding LORA model based on the weight parameters generated by each Transformer module.

[0073] Step 302: During the iterative training process of inputting the second scene image in the second scene into the first object detection model embedded in the LoRA model, the parameters of the first object detection model are fixed, and the model parameters of each LoRA model are trained.

[0074] The model parameters of the LORA model include a dimensionality reduction matrix A and an increased dimensionality matrix B; the fine-tuning of the weight parameters generated by each Transformer module in the first object detection model based on the model parameters of each trained LORA model includes:

[0075] The weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the following formula: W = W0 + BA

[0076] W0 represents the weight parameters generated by each Transformer module, A is the dimensionality reduction matrix, B is the dimensionality increase matrix, and W is the fine-tuned weight parameters generated by each Transformer module.

[0077] The weight parameters W0 generated by the Transformer module are Q, K, and V. Here, we take the weight parameter Q as an example and, in conjunction with the LoRA model fine-tuning flowchart shown in Figure 5, explain the LoRA fine-tuning of the weight parameter Q:

[0078] Taking the LoRA model with input X as d-dimensional and output Y as k-dimensional as an example, the input of the LoRA model is consistent with the input of the Transformer module in the first object detection model; where the weight parameter Q0 is the weight parameter Q in the corresponding Transformer module of the LoRA model. The fine-tuning steps of LoRA are as follows:

[0079] Step 401: The dimension of the weight parameter Q0 is d×k. The r×k dimension reduction matrix A is randomly initialized with a Gaussian distribution, and the d×r dimension increase matrix B is initialized with a 0 matrix, where r is much smaller than the input dimension and the output dimension.

[0080] Step 402: During training, the parameters of the first object detection model are fixed. For the LoRA model, the weight parameter Q0 is fixed, and only the dimension reduction matrix A and the dimension increase matrix B are trained. After training, BA and Q0 are added to obtain the updated weight parameter Q, thereby ensuring that the input and output dimensions of the trained model are consistent with those of the first object detection model.

[0081] The formula for updating the weight parameter Q is Q = Q0 + BA;

[0082] The output formula of the LoRA model is Y = Q0X + BAX;

[0083] The update process for weight parameters K and V is the same as that for weight parameter Q.

[0084] In this embodiment, to enable the object detection algorithm trained in one scene to be quickly transferred to another, a LoRA fine-tuning method is used. This method updates only the Q, K, and V weights in the Transformer structure, thereby accelerating the iteration speed of the algorithm across different scenes. This allows the algorithm to be quickly applied to different scenarios while maintaining its accuracy. The LoRA model can be flexibly merged with the original model, making the algorithm applicable to new scenes. Furthermore, the LoRA model can be separated from the merged model to restore the capabilities of the original model.

[0085] In one embodiment, the step of inputting the second scene image in the second scene into a first object detection model embedded in the LoRA model for training includes the following:

[0086] During each iteration of training, the weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the updated model parameters of each LORA model; based on the fine-tuned weight parameters of each Transformer module, the first object detection model performs object detection on the second scene image in the second scene to obtain an evaluation index; when the evaluation index meets the preset model accuracy, the second object detection model is obtained.

[0087] In one example embodiment, as shown in Figure 6, the fine-tuning steps of the target detection model across scenes include:

[0088] Step 1: In scenario A, train the YOLOv8 optimized model based on the labeled image dataset of scenario A to obtain model A suitable for scenario A. Model A is used to perform vehicle detection and recognition on images collected in scenario A.

[0089] Step 2: In scenario B, after embedding the LoRA model into each Transformer module of the first object detection model, the model is trained based on the labeled image dataset of scenario B. During the training process, the parameters of model A are fixed, and the LoRA model is trained for the weight parameters Q, K, and V in the Transformer structure to obtain the trained LoRA model parameters.

[0090] Step 3: Merge the trained LoRA model parameters with the corresponding original weight parameters Q, K, and V in Model A to obtain Model B, which is suitable for scenario B. Model B is used to perform vehicle detection and recognition on images acquired in scenario B.

[0091] Step 4: Based on the labeled image datasets collected from different scenarios, repeat the above operations to fine-tune the object detection model, and you can obtain the LoRA fine-tuned model and the merged object detection model applicable to multiple scenarios.

[0092] In this embodiment, image data is acquired through a camera to obtain image datasets of multiple real-world scenes. The image datasets are then labeled. First, the targets to be identified are pre-labeled using an open-set object detector (GroundingDINO), and then the labeled data is manually cleaned. During model training, the DeepSpeed ​​library is used to train the object detection model on multiple GPUs. The algorithm is deployed more quickly using either the OpenVINO framework or the TensorRT framework. The video stream from the camera is decoded using a multimedia processing toolkit (FFmpeg) and provided to the deployed object detection model for detection and recognition, outputting vehicle detection results.

[0093] This application also provides a cross-scene target detection system based on fine-tuning, as shown in Figure 7. The system includes:

[0094] The first scene model training module 10 is used to input the first scene image under the first scene into the YOLOv8 optimization model for training to obtain a first object detection model; wherein, the YOLOv8 optimization model includes a connected backbone network and a head network, the backbone network includes multiple Transformer modules, each Transformer module performs hierarchical feature extraction based on the first scene image to obtain a first feature map and generates weight parameters; the head network is used to perform feature fusion on the first feature map and then perform object detection to obtain a first object detection result;

[0095] The second scene model training module 20 is used to embed LoRA models into each of the Transformer modules in the first object detection model, input the second scene image in the second scene into the first object detection model embedded with the LoRA model, fix the parameters of the first object detection model, train the model parameters of each of the LoRA models, and fine-tune the weight parameters generated by each of the Transformer modules in the first object detection model based on the trained model parameters of each of the LoRA models to obtain the second object detection model.

[0096] The detection module 30 is used to input the image of the target to be detected in the second scene into the second target detection model to obtain the second target detection result.

[0097] In one embodiment, the backbone network further includes multiple Patch Merging layers, each Patch Merging layer being connected to the output of each Transformer module, wherein:

[0098] The Patch Merging layer is used to downsample the first feature map output by the connected Transformer module to obtain a second feature map, and then input the second feature map into the next Transformer module for processing.

[0099] The Transformer module is used to sequentially perform standard convolution, point convolution, unfold, encode, fold, point convolution, concatenation, and standard convolution operations on the input second feature map.

[0100] In one embodiment, the second scene model training module 20 is further configured to:

[0101] Based on the weight parameters generated by each Transformer module, initialize the model parameters of each corresponding LORA model;

[0102] During the iterative training process of inputting the second scene image in the second scene into the first object detection model embedded in the LoRA model, the parameters of the first object detection model are fixed, and the model parameters of each LoRA model are trained.

[0103] In one embodiment, the second scene model training module 20 is further configured to:

[0104] The weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the following formula: W =W0+BA

[0105] Among them, w o The weight parameters generated by each Transformer module are: A is the dimensionality reduction matrix, B is the dimensionality increase matrix, and w is the fine-tuned weight parameters generated by each Transformer module.

[0106] In one embodiment, the weight parameters generated by each of the Transformer modules are calculated using the following formula:

[0107] Where Q, K, and V are the weight parameters generated by the Transformer module, and d is the dimension of the second feature map input to each Transformer module.

[0108] In one embodiment, the second scene model training module 20 is further configured to:

[0109] During each iteration of training, the weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the updated model parameters of each LORA model.

[0110] Based on the first target detection model with the fine-tuned weight parameters of each Transformer module, target detection is performed on the second scene image in the second scene to obtain the evaluation index.

[0111] When the evaluation index meets the preset model accuracy, the second target detection model is obtained.

[0112] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram is shown in Figure 8. The computer device includes a processor, memory, communication interface, display screen, and input device connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a fine-tuning-based cross-scene target detection method.

[0113] Those skilled in the art will understand that the structure shown in Figure 8 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0114] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above embodiments of the fine-tuning-based cross-scene target detection method.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0117] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A cross-scene target detection method based on fine-tuning, characterized in that, The method includes: The first scene image is input into the YOLOv8 optimization model for training to obtain the first object detection model. The YOLOv8 optimization model includes a connected backbone network and a head network. The backbone network includes multiple Transformer modules, each of which performs hierarchical feature extraction based on the first scene image to obtain a first feature map and generates weight parameters. The head network is used to perform feature fusion on the first feature map and then perform object detection to obtain the first object detection result. A LoRA model is embedded in each of the Transformer modules in the first object detection model. The second scene image in the second scene is input into the first object detection model embedded with the LoRA model. The parameters of the first object detection model are fixed, and the model parameters of each LoRA model are trained. Based on the trained model parameters of each LoRA model, the weight parameters generated by each Transformer module in the first object detection model are fine-tuned to obtain the second object detection model. The target image to be detected in the second scene is input into the second target detection model to obtain the second target detection result.

2. The method according to claim 1, characterized in that, The backbone network also includes multiple Patch Merging layers, each of which is connected to the output of each Transformer module, wherein: The Patch Merging layer is used to downsample the first feature map output by the connected Transformer module to obtain a second feature map, and then input the second feature map into the next Transformer module for processing. The Transformer module is used to sequentially perform standard convolution, point convolution, unfold, encode, fold, point convolution, concatenation, and standard convolution operations on the input second feature map.

3. The method according to claim 1, characterized in that, The step of fixing the parameters of the first target detection model and training the model parameters of each of the LORA models includes: Based on the weight parameters generated by each Transformer module, initialize the model parameters of each corresponding LORA model; During the iterative training process of inputting the second scene image in the second scene into the first object detection model embedded in the LoRA model, the parameters of the first object detection model are fixed, and the model parameters of each LoRA model are trained.

4. The method according to claim 1, characterized in that, The model parameters of the LORA model include a dimensionality reduction matrix and an dimensionality increase matrix; the fine-tuning of the weight parameters generated by each Transformer module in the first object detection model based on the model parameters of each trained LORA model includes: The weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the following formula: W = W0 + BA Wherein, W0 is the weight parameter generated by each Transformer module, A is the dimensionality reduction matrix, B is the dimensionality increase matrix, and W is the weight parameter generated by each Transformer module after fine-tuning.

5. The method according to claim 1, characterized in that, The weight parameters generated by each Transformer module are calculated using the following formula: Where Q, K, and V are the weight parameters generated by the Transformer module, and d is the dimension of the second feature map input to each Transformer module.

6. The method according to claim 1, characterized in that, The method further includes: During each iteration of training, the weight parameters generated by each Transformer module in the first object detection model are fine-tuned using the updated model parameters of each LORA model. Based on the first target detection model with the fine-tuned weight parameters of each Transformer module, target detection is performed on the second scene image in the second scene to obtain the evaluation index. When the evaluation index meets the preset model accuracy, the second target detection model is obtained.

7. The method according to claim 1, characterized in that, The backbone network further includes convolutional layers connected to the input of the backbone network, and the convolutional layers are used for: Perform standard convolution processing on the first scene image under the first scene input.

8. A cross-scene target detection system based on fine-tuning, characterized in that, The system includes: The first scene model training module is used to input the first scene image under the first scene into the YOLOv8 optimization model for training to obtain the first object detection model; wherein, the YOLOv8 optimization model includes a connected backbone network and a head network, the backbone network includes multiple Transformer modules, each Transformer module performs hierarchical feature extraction based on the first scene image to obtain a first feature map and generates weight parameters; the head network is used to perform feature fusion on the first feature map and then perform object detection to obtain the first object detection result; The second scene model training module is used to embed LoRA models into each of the Transformer modules in the first object detection model, input the second scene image in the second scene into the first object detection model embedded with the LoRA model, fix the parameters of the first object detection model, train the model parameters of each of the LoRA models, and fine-tune the weight parameters generated by each of the Transformer modules in the first object detection model based on the trained model parameters of each of the LoRA models to obtain the second object detection model. The detection module is used to input the target image to be detected in the second scene into the second target detection model to obtain the second target detection result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images