Visible light target detection training enhancement method and system and storage medium
By combining infrared and visible images in the object detection model, the problem of poor object detection effect in complex background, dark and strong light environments is solved, and higher detection accuracy and robustness are achieved.
Patent Information
- Application Number
- CN202411474279.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-22
- Publication Date
- 2025-02-14
AI Technical Summary
Existing visible light target detection technology shows poor detection effect in complex background, dark and strong light environments, especially because the model is difficult to accurately identify and locate the target, and the details and edge information of the target are easily distorted or lost in these environments. .
By inputting infrared images and visible light images into the backbone network of the object detection model for inference, and distilling the feature maps obtained by visible light image inference by using shallow and middle layer feature maps obtained by infrared image inference, back propagating and retaining the gradient, thereby Update the model weights and improve the detection effect.
This method effectively improves the detection accuracy and robustness of the visible light target detection model in complex background, dark and strong light environments, reduces false detection and missed detection, and improves the accurate capture ability of target shape and structure.
Smart Images
Figure CN119445213A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning, and in particular to a visible light target detection training enhancement method, system and storage medium. Background Art
[0002] The importance of visible light target detection technology in the field of computer vision is self-evident. However, its limitations and shortcomings in the face of challenges such as complex backgrounds, dark light, and strong light are still one of the focuses of current research. In complex backgrounds, targets are often mixed with the surrounding environment, making the boundaries of the targets unclear and difficult to accurately identify and locate. Dark light and strong light environments will cause the contrast of the image to be reduced or too high, which will affect the visibility and feature expression of the target, making it difficult for the detection algorithm to effectively distinguish the target from the background. These challenges have brought a series of problems to visible light target detection: First, it is difficult for the model to fully express the feature information of the target, resulting in the detection accuracy and stability being affected, and it is easy to have false detection and missed detection; second, in dark light and strong light environments, the details and edge information of the target are often distorted or lost, making it difficult to accurately capture the shape and structure of the target, which in turn affects the accuracy and robustness of the detection results. In order to meet these challenges, researchers are exploring various innovative methods and technologies. For example, by combining deep learning with traditional image processing technology, images are preprocessed using traditional methods, such as changing the contrast and sharpness of the image, and then sent to the target detection model for inference. However, the effect is not good and can only achieve a certain optimization effect for similar effects in the same scene. Summary of the invention
[0003] Therefore, in order to solve the above-mentioned shortcomings, the present invention provides a visible light target detection training enhancement method, system and storage medium. The present invention can solve the problem of poor target detection effect under complex background, dark light and strong light conditions in the prior art.
[0004] In a first aspect, the present invention provides a visible light target detection training enhancement method, comprising: The infrared processed image is passed through the backbone network of the first target detection model for inference; Inferring the visible light image through a backbone network of a first target detection model or a second target detection model, wherein the backbone network of the first target detection model and the backbone network of the second target detection model share weights; The shallow and middle feature maps obtained by inferring the infrared image using the first target detection model are distilled respectively on the shallow and middle feature maps obtained by inferring the visible light using the first target detection model or the second target detection model, and back-propagated and their gradients are retained.
[0005] Optionally, after the first target detection model or the second target detection model completes a round of distillation, the loss obtained by the training set label, the gradient obtained by back propagation, and the gradients obtained by the shallow and middle layers of the distillation backbone network are added according to the weight ratio to update the weight of the first target detection model or the second target detection model.
[0006] Optionally, after the first target detection model or the second target detection model has the extraction capability, the weights of its backbone network are frozen, and then infrared processing image phase reasoning is used to perform model self-distillation on visible light reasoning.
[0007] Optionally, a method for determining whether the first target detection model or the second target detection model after self-distillation has extraction capability is: Using feature map visualization, images with visible light and dark light are used for inference using the first target detection model or the second target detection model after self-distillation. The feature map calculated by the backbone network is intercepted, and the feature extraction situation in the feature map is used to determine whether the model has the extraction capability.
[0008] Optionally, a method for determining whether the first target detection model or the second target detection model after self-distillation has extraction capability is: The backbone network of the first target detection model or the second target detection model is used to infer the feature map of the infrared image, and the peak signal-to-noise ratio or structural consistency is calculated to determine whether the model has extraction capability.
[0009] Optionally, the infrared processed image refers to converting the infrared image from a single-channel image to a three-channel image.
[0010] Optionally, the distillation method includes but is not limited to MGD, CWD or AMD.
[0011] Optionally, the gradient weight ratio of distillation is adjusted according to the actual training effect.
[0012] The principle of the above visible light target detection training enhancement method is: Infrared images can better obtain the shape information of target objects in complex backgrounds, dark light, and strong light. They can assist the visible light target detection model in having better feature extraction capabilities during training. The target detection model is generally divided into three parts: the backbone network, the neck network, and the detection head. The backbone network is used to extract features. The shallow and middle layers extract the edge texture and shape information of the target. During the training process, the weight-sharing backbone network is used to simultaneously infer the visible light and infrared images of the same field of view, and the shallow and middle layer feature maps obtained by inferring infrared are used to distill the feature maps of inferred visible light, thereby helping to extract features from the visible light perspective.
[0013] In a second aspect, the present invention provides a visible light target detection training enhancement system, comprising: An infrared processing image module, used for inferring the infrared processed image through the backbone network of the first target detection model; A visible light image module, used to infer the visible light image through the backbone network of the first target detection model or the second target detection model, wherein the backbone network of the first target detection model and the backbone network of the second target detection model share weights; The model enhancement module is used to use the shallow and middle feature maps obtained by inferring the infrared image using the first target detection model, and to distill the shallow and middle feature maps obtained by inferring the visible light using the first target detection model or the second target detection model, and to back-propagate and retain their gradients.
[0014] In a third aspect, the present invention provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the visible light target detection training enhancement method.
[0015] The present invention has the following advantages: The present invention solves the problem of poor target detection effect under complex background, dark light and strong light conditions. The present invention uses the shallow and middle feature maps obtained by inferring infrared images with a detection model, distills the shallow and middle feature maps obtained by inferring visible light with the detection model, back-propagates and retains their gradients, thereby improving the detection effect of the visible light target detection model under complex background, dark light and strong light. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a flow chart of the visible light target detection training enhancement method of the present invention; Figure 2 It is a systematic block diagram of the visible light target detection training enhancement system of the present invention; In the figure: 100, infrared reasoning module; 200, visible light reasoning module; 300, distillation module. DETAILED DESCRIPTION
[0017] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0018] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises", or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or apparatus.
[0019] As mentioned in the background technology, the importance of visible light target detection technology in the field of computer vision is self-evident. However, its limitations and shortcomings in the face of challenges such as complex backgrounds, dark light, and strong light are still one of the focuses of current research. In complex backgrounds, targets are often mixed with the surrounding environment, making the boundaries of the targets unclear and difficult to accurately identify and locate. Dark light and strong light environments will cause the contrast of the image to be reduced or too high, which will affect the visibility and feature expression of the target, making it difficult for the detection algorithm to effectively distinguish the target from the background. These challenges have brought a series of problems to visible light target detection: First, it is difficult for the model to fully express the feature information of the target, resulting in the detection accuracy and stability being affected, and it is easy to have false detection and missed detection; second, in dark light and strong light environments, the details and edge information of the target are often distorted or lost, making it difficult to accurately capture the shape and structure of the target, which in turn affects the accuracy and robustness of the detection results. In order to meet these challenges, researchers are exploring various innovative methods and technologies. For example, by combining deep learning with traditional image processing technology, images are preprocessed using traditional methods, such as changing the contrast and sharpness of the image, and then sent to the target detection model for inference. However, the effect is not good and can only achieve a certain optimization effect for similar effects in the same scene.
[0020] In view of the above shortcomings, Figure 1 As shown, this embodiment provides a visible light target detection training enhancement method, the method comprising the following steps: Step S100, inferring the infrared processed image through the backbone network of the first target detection model, wherein the infrared processed image refers to converting the infrared image from a single channel to a three-channel image; Step S300, inferring the visible light image through the backbone network of the first target detection model or the second target detection model, wherein the backbone network of the first target detection model and the backbone network of the second target detection model share weights; Step S300: Using the shallow and middle feature maps obtained by inferring the infrared image using the first target detection model, distill the shallow and middle feature maps obtained by inferring the visible light using the first target detection model or the second target detection model, back-propagate and retain their gradients; optionally, the distillation method includes but is not limited to the MGD, CWD or AMD method. Optionally, the gradient weight ratio of the distillation is adjusted according to the actual training effect.
[0021] The above-mentioned technical features use the shallow and middle feature maps obtained by inferring infrared images through the detection model, distill the shallow and middle feature maps obtained by inferring visible light through the detection model, back-propagate and retain their gradients, thereby improving the detection effect of the visible light target detection model under complex backgrounds, dark light and strong light. In the implementation process, it can be implemented only by the first template detection model, or by the first target detection model and the second target detection model. When it is implemented only by the first target detection model, the backbone network of the first target detection model is inferred by the visible light image and the infrared image respectively, and the shallow and middle feature maps obtained by inferring the visible light image are distilled by the result of the infrared image inference, back-propagated and retained their gradients. When the first target detection model and the second target detection model are used to cooperate to implement, the shallow and middle feature maps obtained by inferring the infrared image through the first target detection model are distilled, back-propagated and retained their gradients.
[0022] The technical principle of the above technical features is: infrared images can better obtain the shape information of target objects in complex backgrounds, dark light and strong light, and can assist the visible light target detection model to have better feature extraction capabilities during training. The target detection model is generally divided into three parts: backbone network, neck network and detection head. The backbone network is used to extract features, and the edge texture and shape information of the target extracted in the shallow and middle layers. During the training process, the backbone network with shared weights is used to simultaneously infer the visible light and infrared images of the same field of view, and the shallow and middle feature maps obtained by inferring infrared are used to distill the feature maps of inferred visible light, thereby helping to extract features from the visible light perspective.
[0023] In one embodiment, after the first target detection model or the second target detection model completes a round of distillation, the loss obtained by the training set labels, the gradient obtained by back propagation, and the gradients obtained by the shallow and middle layers of the distilled backbone network are added according to the weight ratio to update the weight of the first target detection model or the second target detection model. In this technical feature, the training set labels obtain the gradient of the overall weight of the model, and the distillation only obtains the gradient of the shallow and middle weights of the backbone network.
[0024] In one embodiment, after the first target detection model or the second target detection model has the extraction capability that meets the requirements in complex background, dark light, and strong light, the weight of its backbone network is frozen, and then infrared processing image phase reasoning is used to perform model self-distillation for visible light reasoning. In this technical feature, since the network model is divided into a backbone network, a neck network, and a head network, and the backbone network is feature extraction, this step is intended to freeze the backbone network to update the weights of the neck network and the detection head after it has reached a certain feature extraction capability.
[0025] In one embodiment, a method for determining whether the first object detection model or the second object detection model after self-distillation has extraction capability is: Using feature map visualization, we use visible light and dark light images to perform inference using the first target detection model or the second target detection model after self-distillation, and intercept the feature map calculated by the backbone network. Based on the feature extraction in the feature map, we determine whether the model has the extraction capability.
[0026] In one embodiment, a method for determining whether the first object detection model or the second object detection model after self-distillation has extraction capability is: The backbone network of the first target detection model or the second target detection model is used to infer the feature map of the infrared image, and PSNR (peak signal-to-noise ratio) or SSIM (structural consistency) is calculated to determine whether the model has extraction capabilities.
[0027] In another embodiment, if Figure 2 As shown, a visible light target detection training enhancement system is also provided, the system comprising: The infrared reasoning module 100 is used to perform reasoning on the infrared processed image through the backbone network of the first target detection model; A visible light reasoning module 200, configured to reason the visible light image through a backbone network of a first target detection model or a second target detection model, wherein the backbone network of the first target detection model and the backbone network of the second target detection model share weights; and The distillation module 300 is used for the third target detection model to use the shallow and middle feature maps obtained by infrared processing map reasoning to distill the shallow and middle feature maps obtained by visible light reasoning, back-propagate and retain the gradient of the third target detection model.
[0028] The visible light target detection training enhancement system relies on the visible light target detection training enhancement method, which can solve the problem of poor target detection effect under complex background, dark light and strong light conditions. The shallow and middle layer feature maps obtained by inferring infrared images using the detection model are distilled, and the shallow and middle layer feature maps obtained by inferring visible light by the detection model are back-propagated and their gradients are retained, thereby improving the detection effect of the visible light target detection model under complex background, dark light and strong light.
[0029] It should be noted that, in the above embodiments, the description of each embodiment has its own focus, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0030] It should be understood by those skilled in the art that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-readable program codes.
[0031] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0032] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0033] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0034] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the above-mentioned visible light target detection training enhancement method.
[0035] If the module / unit integrated in the visible light target detection training enhancement system / terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented.
[0036] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.
[0037] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0038] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0039] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0040] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present application. The above embodiments are only used to illustrate the technical solution of the present application, not to limit it; although the present application is described in detail with reference to the above embodiments, ordinary technicians in this field should understand that: it is still possible to modify the technical solutions recorded in the above embodiments, or to replace some of the technical features therein by equivalent; and these modifications or replacements do not make the essence of the corresponding technical solution deviate from the spirit and scope of the technical solutions of each embodiment of the present application.
[0041] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A visible light target detection training enhancement method, characterized in that: include The infrared processed image is passed through the backbone network of the first target detection model for inference; Reasoning the visible light image through a backbone network of a first target detection model or a second target detection model, wherein the backbone network of the first target detection model and the backbone network of the second target detection model share weights; The shallow and middle feature maps obtained by inferring the infrared image using the first target detection model are distilled respectively on the shallow and middle feature maps obtained by inferring the visible light using the first target detection model or the second target detection model, and back-propagated and their gradients are retained.
2. According to claim 1, a visible light target detection training enhancement method is characterized in that: After the first target detection model or the second target detection model completes a round of distillation, the loss obtained by the training set label, the gradient obtained by back propagation, and the gradients obtained by the shallow and middle layers of the distillation backbone network are added according to the weight ratio to update the weight of the first target detection model or the second target detection model.
3. According to claim 1, a visible light target detection training enhancement method is characterized in that: After the first target detection model or the second target detection model has the extraction capability, the weight of its backbone network is frozen, and then infrared processing image phase reasoning is used to perform model self-distillation on visible light reasoning.
4. According to claim 3, a visible light target detection training enhancement method is characterized in that: The method for judging whether the first target detection model or the second target detection model after self-distillation has extraction capability is: Using feature map visualization, images with visible light and dark light are inferred using the first target detection model or the second target detection model after self-distillation, and the feature map calculated by the backbone network is intercepted. The feature extraction in the feature map is used to determine whether the model has the extraction capability.
5. According to claim 3, a visible light target detection training enhancement method is characterized in that: The method for judging whether the first target detection model or the second target detection model after self-distillation has extraction capability is: The backbone network of the first target detection model or the second target detection model is used to infer the feature map of the infrared image, and the peak signal-to-noise ratio or structural consistency is calculated to determine whether the model has extraction capability.
6. According to claim 1, a visible light target detection training enhancement method is characterized in that: The infrared processing image refers to converting the infrared image from a single channel to a three-channel image.
7. According to claim 1, a visible light target detection training enhancement method is characterized in that: The distillation method includes, but is not limited to, MGD, CWD or AMD.
8. The visible light target detection training enhancement method according to claim 1, characterized in that: The gradient weight ratio of distillation is adjusted according to the actual training effect.
9. A visible light target detection training enhancement system, characterized in that: include: An infrared processing image module, used for inferring the infrared processed image through the backbone network of the first target detection model; A visible light image module, used to infer the visible light image through the backbone network of the first target detection model or the second target detection model, wherein the backbone network of the first target detection model and the backbone network of the second target detection model share weights; The model enhancement module is used to use the shallow and middle feature maps obtained by inferring the infrared image using the first target detection model, and to distill the shallow and middle feature maps obtained by inferring the visible light by the first target detection model or the second target detection model, and to back-propagate and retain their gradients.
10. A storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the visible light target detection training enhancement method as described in any one of claims 1 to 7.