Target detection method and system based on illumination invariant features, device, medium

Abstract

The application relates to the technical field of computer graphics and computer vision, and discloses a target detection method and system based on illumination-invariant features, which comprises the following steps: constructing a cross-color feature processing module; constructing an illumination-invariant feature module based on the cross-color feature module, and constructing a network model based on the illumination-invariant feature module; collecting low-illumination pictures as a low-illumination dataset, and inputting the low-illumination dataset into the network model; extracting illumination-invariant features by using illumination-invariant constraints, and guiding network model module learning by using loss constraints; training the network model, obtaining a trained network model, and inputting low-illumination pictures in real time to perform target detection. The system comprises a model construction unit, a data collection unit, a model learning unit and a model training unit. The application further discloses an electronic device and a computer readable storage medium. In a low-illumination object detection task, the application can significantly improve performance.

Description

Technical Field

[0001] This invention relates to the fields of computational graphics and computer vision, and in particular to a target detection method, system, device, and medium based on illumination-invariant features. Specifically, it relates to using deep neural networks to extract illumination-invariant features for target detection in low-light scenes. Background Technology

[0002] Object detection aims to identify and locate the position of specific objects in images or videos. It utilizes machine learning and deep learning techniques to analyze and process input images. Object detection algorithms can detect and label the bounding boxes of multiple objects in an image, providing information on object type and location. This technology is widely used in fields such as autonomous driving, object tracking, and face recognition. However, in practical applications, adverse imaging conditions (such as nighttime driving or low-light environments) can lead to false positives and false negatives in object detection, thus affecting the accuracy of subsequent algorithms.

[0003] Previous research has primarily focused on image enhancement, transforming low-light images into normal-light images before downstream detection tasks. However, this approach often neglects the detection task itself, potentially leading to poor detection results. Meanwhile, some researchers have attempted to fine-tune detectors on low-light datasets to adapt them to images with that lighting distribution. However, this technique often relies on synthetic datasets, resulting in poor performance in real-world scenes. Recent research employs the Laplacian pyramid to extract multi-scale and edge information to improve object detection. However, due to varying lighting conditions, these task-specific methods typically require handling a larger solution space.

[0004] Therefore, in order to address the above-mentioned problems, this invention provides a target detection method, system, device, and medium based on illumination-invariant features, which utilizes deep neural networks to extract illumination-independent features to perform target detection tasks in low-light scenarios. Summary of the Invention

[0005] This invention provides a target detection method, system, device, and medium based on illumination-invariant features, which utilizes deep neural networks to extract illumination-independent features for target detection in low-light scenarios.

[0006] This invention is achieved through the following technical solution: a target detection method based on illumination-invariant features, comprising the following steps:

[0007] Step S1: Construct a cross-color feature processing module in a two-dimensional image in the RGB domain;

[0008] Step S2: Construct an illumination-invariant feature module based on the cross-color feature module, and build a network model based on the illumination-invariant feature module;

[0009] Step S3: Collect low-light images as a low-light dataset and input them into the network model;

[0010] Step S4: Extract illumination-invariant features using illumination invariance constraints, and use loss constraints to guide the learning of the network model modules;

[0011] Step S5: Train the network model, obtain the trained network model, and perform target detection on the low-light image in real time to obtain the final position and size of the target.

[0012] To better implement the present invention, step S1 further includes:

[0013] Cross-color features are constructed using any two channels from the R, G, and B channels of the two-dimensional image through positive and negative convolution operations and log operations;

[0014] All the constructed cross-color features are stacked and added together to form the cross-color feature module, which includes three sets of positive and negative convolutional kernel layers with the same structure.

[0015] To better implement this invention, the method for constructing cross-color features using any two channels of the R, G, and B channels of the two-dimensional image through positive and negative convolution operations and log operations further includes: when using the R and B channels, through convolution... The logarithmic operation eliminates lighting-related attributes while increasing learnable parameters and feature diversity, as shown below:

[0016]

[0017] in, This represents a learnable convolutional kernel. Indicates a positive kernel convolutional layer. This indicates a negative kernel convolutional layer;

[0018] use Indicates the use of convolution kernel The cross-color features extracted from image I are represented as follows:

[0019]

[0020] Here, [·] indicates that features are stacked along the channel dimension.

[0021] To better realize the present invention, step S2 further includes:

[0022] The illumination-invariant feature module includes 16 parallel cross-color feature processing modules;

[0023] The network model includes the illumination-invariant feature module and a target detector connected in series with it.

[0024] To better realize the present invention, step S3 further includes:

[0025] Select a low-light photo P1 from the low-light dataset, input the low-light photo P1 into the network model, and the illumination invariant feature module sends the low-light photo P1 to 16 different cross-color feature processing modules inside the module, and stacks the 16 output results on the channel;

[0026] The cross-color feature processing module includes three identical positive and negative convolutional kernel layers. Each set of positive and negative convolutional kernels is preceded by a log operation and followed by an addition operation. The output includes a stacked operation in the channel dimension of the three sets of convolutional kernel layers. The three sets of convolutional kernel layers are parallel operations and process pairwise channel combinations between different R, G, and B channels. The illumination-invariant feature module includes the first to sixteenth cross-color feature processing modules. The 16 cross-color feature processing modules are processed in parallel, and the output includes a stacked operation in the channel dimension.

[0027] Next, information unrelated to illumination is extracted from the low-light image P1, and this information is fused with the original low-light image P1 before being sent to the target detector.

[0028] To better realize the present invention, step S4 further includes:

[0029] Select a low-light photo P2 from the low-light dataset, perform Gamma transformation on the low-light photo P2, randomly brighten or darken it, and then send it to the cross-color feature processing module to obtain a set of feature maps P21.

[0030] The low-light image P2 is then passed to the weight-sharing cross-color feature processing module without any transformation to obtain another set of feature maps P22.

[0031] Using Smooth L1 constraint to keep feature map P21 and feature map P22 consistent further ensures the illumination invariance of the features, and is expressed as:

[0032]

[0033] Where σ represents the gamma brightness transformation function, Let represent the convolution kernel, where the subscript i indicates that the convolution kernel is the i-th one, and β represents the hyperparameters during training.

[0034] To better implement the present invention, step S5 further includes:

[0035] The network model is trained on the low-light dataset for a regular object detection task. Before starting training, the network is loaded with pre-trained open-source weights based on normal lighting data. During training, a subset of low-light images are fine-tuned.

[0036] This invention also provides a target detection system based on illumination-invariant features.

[0037] The present invention also provides an electronic device comprising a processor and a memory; the processor includes the target detection system based on illumination-invariant features described in the second aspect above.

[0038] The present invention also provides a computer-readable storage medium comprising instructions that, when executed on an electronic device described in the third aspect, cause the electronic device to perform the method described in the first aspect.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] (1) This invention provides a target detection method, system, device, and medium based on illumination-invariant features. Unlike previous studies, we propose to solve this challenging problem from the perspective of feature learning.

[0041] (2) This invention provides a target detection method, system, device and medium based on illumination-invariant features. We use deep neural networks to extract illumination-invariant features to perform target detection tasks in low-light scenes. Experimental verification shows that our method can significantly improve performance in low-light object detection tasks. Attached Figure Description

[0042] The present invention will be further described in conjunction with the following drawings and embodiments. All inventive concepts of the present invention should be considered as disclosed content and within the scope of protection of the present invention.

[0043] Figure 1 A schematic diagram of the network model in a target detection method, system, device, and medium based on illumination-invariant features provided in this application embodiment;

[0044] Figure 2 A schematic diagram of the structure of a cross-color feature processing module in a target detection method, system, device, and medium based on illumination-invariant features provided in this application embodiment;

[0045] Figure 3 A schematic diagram of the structure of an illumination-invariant feature module in a target detection method, system, device, and medium based on illumination-invariant features provided in this application embodiment;

[0046] Figure 4 Image a1 illustrates the effect of a target detection method, system, device, and medium based on illumination-invariant features provided in this application embodiment.

[0047] Figure 5 Image a2 illustrates the effect of a target detection method, system, device, and medium based on illumination-invariant features provided in this application embodiment;

[0048] Figure 6 Image b shows the effect of a target detection method, system, device, and medium based on illumination-invariant features provided in this application embodiment. Detailed Implementation

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments, and therefore should not be regarded as a limitation on the scope of protection. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set up," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0051] Example 1:

[0052] This embodiment presents a target detection method, system, device, and medium based on illumination-invariant features. We propose an illumination-invariant feature-based target detection method specifically designed for target detection tasks in low-light conditions. Unlike previous research, we propose addressing this challenging problem from the perspective of feature learning. Specifically, we propose learning illumination-invariant features using a Lambert illumination model. We observe that, under the Lambert assumption, the illumination-invariant feature map can be approximated using the relationships between adjacent color channels and spatially adjacent pixels. By adding additional constraints, these relationships can be represented in the form of convolutional kernels, enabling them to be trained within the network. Therefore, we introduce a novel module specifically for extracting illumination-invariant features from low-light images, coupled with a newly designed loss function to enhance the consistency of features across different illumination levels. Both modules can be easily integrated into existing target detection frameworks, and our empirical results demonstrate a significant performance improvement in low-light object detection tasks.

[0053] like Figure 1 As shown, the model structure consists of two parts: an illumination-invariant feature extraction module (IIM) and a cascaded object detector. The detector can be any commercially available object detection model with any network structure. During training, a low-light image is brightened, resulting in a low-light image and a brightened image. These two images are fed into the IIM module, and their outputs are used to construct an illumination-invariant constraint (II Loss). The IIM module with this constraint is then trained together with the subsequent detector. During inference, a low-light image is fed into the IIM module without brightening it, and the input of the IIM module is passed to the detector to obtain the bounding box.

[0054] Example 2:

[0055] This embodiment is a further optimization based on Embodiment 1, such as... Figure 2 As shown, construct the cross-color feature module.

[0056] A two-dimensional image in the RGB domain has three channels: red, green, and blue. Cross-color features can be constructed from images of any two channels. Taking the red and blue color channels R and B as an example, we can use convolution... By using logarithmic operations, we can eliminate lighting-related properties. Simultaneously, we can increase the number of learnable parameters, thereby increasing the diversity of features.

[0057]

[0058] in This represents a learnable convolutional kernel. Indicates a positive kernel convolutional layer. This represents a negative kernel convolutional layer. Similarly, besides RB, we can also construct cross-color features of RG and GB. We use... Indicates the use of convolution kernel Cross-color features extracted from image I:

[0059]

[0060] Here, [·] indicates that features are stacked along the channel dimension.

[0061] The other parts of this embodiment are the same as those in Embodiment 1, so they will not be described again.

[0062] Example 3:

[0063] This embodiment is a further optimization based on the above embodiment 1 or 2. In constructing the illumination invariant feature module, the illumination invariant feature module first accepts a low-light image as input, and sends it to 16 different cross-color processing modules inside the module, and stacks the 16 output results on the channel.

[0064] like Figure 2 As shown, the cross-color processing module includes three identical positive and negative convolutional kernel layers. Each set of positive and negative convolutional kernels is preceded by a log operation, followed by an addition operation. The output consists of a stacked operation along the channel dimension of the three sets of convolutional kernel layers. The three sets of convolutional kernel layers operate in parallel and process different pairwise RGB channel combinations.

[0065] like Figure 3 As shown, the illumination-invariant feature module includes the first to sixteenth cross-color feature processing modules. The 16 processing modules are connected in parallel, and the output includes a stacked operation with channel dimensions.

[0066] Next, the illumination-invariant feature extraction (IIM) module extracts illumination-independent information from the low-light image and fuses this information with the original image to generate an enhanced image suitable for object detection. This enhanced image is then fed into the detector to obtain the final target location and size.

[0067] The other parts of this embodiment are the same as any one of the embodiments 1-2 above, so they will not be described again.

[0068] Example 4:

[0069] This embodiment is a further optimization based on any one of embodiments 1-3 above, with an illumination invariant loss constraint:

[0070] Low-light images undergo a Gamma transformation, randomly brightening or darkening them, and are then fed into the Inter-Color Feature Processing (IIM) module to obtain a set of feature maps. Similarly, low-light images are not transformed in any way, and then...

[0071]

[0072] The weights are passed to the cross-color feature processing module (IIM module) for weight sharing.

[0073] We obtain another set of feature maps. These two sets of feature maps should be consistent. We use Smooth L1 constraint to keep these two sets of feature maps consistent, further ensuring the illumination invariance of the features.

[0074] Where σ represents the gamma brightness transformation function, Let represent the convolution kernel, where the subscript i indicates that the convolution kernel is the i-th one, and β represents the hyperparameters during training.

[0075] The other parts of this embodiment are the same as any one of the embodiments 1-3 above, so they will not be described again.

[0076] Example 5:

[0077] This embodiment further optimizes any one of embodiments 1-4 above. During network training, through the preceding feature extraction operation and loss constraints, the network is forced to learn to extract illumination-invariant features. Based on this, we train the network on a regular object detection task on a low-light dataset. Before starting training, we load the network with pre-trained open-source weights from normal lighting data. During training, we fine-tune the network using a small number of low-light images.

[0078] like Figures 4-6 The image shown is a demonstration of the effect. Figures 4-6 The left side shows the visualization of edge features, the right side shows the effect of illumination-invariant features, the first column shows the effect of a conventional detector, and the second column shows the effect after adding our method.

[0079] By comparing the results of conventional detectors, the effects of adding our method, and the magnified region, we can see that our proposed method outperforms edge features and the features we extract. The visualization of edge features and the effects of illumination-invariant features demonstrate that our extracted features are more diverse.

[0080] The other parts of this embodiment are the same as any one of the embodiments 1-4 above, so they will not be described again.

[0081] Example 6:

[0082] The present invention also provides a target detection system based on illumination-invariant features that matches the method, including a model building unit, a data acquisition unit, a model learning unit, and a model training unit.

[0083] The present invention also provides an electronic device, which includes a processor and a memory; the processor includes the target detection system based on illumination-invariant features described above.

[0084] Example 7:

[0085] The present invention also provides a computer-readable storage medium comprising instructions; when the instructions are executed on the electronic device described in the above embodiments, the electronic device causes the electronic device to perform the methods described in the above embodiments. Optionally, the computer-readable storage medium may be a memory.

[0086] The processor involved in the embodiments of this application can be a chip. For example, it can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0087] The memory involved in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0088] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0089] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0090] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0092] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located on one device or distributed across multiple devices. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] In addition, the functional modules in the various embodiments of this application can be integrated into one device, or each module can exist physically separately, or two or more modules can be integrated into one device.

[0094] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

[0095] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications or equivalent changes made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the present invention.

Claims

1. A target detection method based on illumination-invariant features, characterized in that, Includes the following steps: Step S1: Construct a cross-color feature processing module in a two-dimensional image in the RGB domain; Step S2: Construct an illumination-invariant feature module based on the cross-color feature module, and build a network model based on the illumination-invariant feature module; Step S3: Collect low-light images as a low-light dataset and input them into the network model; Step S4: Extract illumination-invariant features using illumination invariance constraints, and use loss constraints to guide the learning of the network model modules; Step S5: Train the network model, obtain the trained network model, and input the low-light image in real time for target detection to obtain the final position and size of the target; Step S1 includes: using any two channels of the R, G, and B channels of the two-dimensional image to construct cross-color features through positive and negative convolution operations and log operations; stacking and adding all the constructed cross-color features by channels to form the cross-color feature module, which includes three sets of positive and negative convolution kernel layers with the same structure; The method for constructing cross-color features using any two channels from the R, G, and B channels of the two-dimensional image through positive and negative convolution operations and log operations includes: when using the R and B channels, through convolution... The logarithmic operation eliminates lighting-related attributes while increasing learnable parameters and feature diversity, as shown below: ； in, This represents a learnable convolutional kernel. Indicates a positive kernel convolutional layer. This indicates a negative kernel convolutional layer; use Indicates the use of convolution kernel To the image The extracted cross-color features are represented as follows: ； in, This indicates that features are stacked along the channel dimension.

2. The target detection method based on illumination-invariant features according to claim 1, characterized in that, Step S2 includes: the illumination-invariant feature module includes 16 parallel cross-color feature processing modules; The network model includes the illumination-invariant feature module and a target detector connected in series with it.

3. The target detection method based on illumination-invariant features according to claim 1, characterized in that, Step S3 includes: selecting a low-light photo P1 from the low-light dataset, inputting the low-light photo P1 into the network model, and having the illumination-invariant feature module send the low-light photo P1 to 16 different cross-color feature processing modules within the module, and stacking the 16 output results on the channel. The cross-color feature processing module includes three identical positive and negative convolutional kernel layers. Each set of positive and negative convolutional kernels is preceded by a log operation and followed by an addition operation. The output includes a stacked operation in the channel dimension of the three sets of convolutional kernel layers. The three sets of convolutional kernel layers are parallel operations and process pairwise channel combinations between different R, G, and B channels. The illumination-invariant feature module includes the first to sixteenth cross-color feature processing modules. The 16 cross-color feature processing modules are processed in parallel, and the output includes a stacked operation in the channel dimension. Next, information unrelated to illumination is extracted from the low-light image P1, and this information is fused with the original low-light image P1 before being sent to the target detector.

4. The target detection method based on illumination-invariant features according to claim 1, characterized in that, Step S4 includes: Select a low-light photo P2 from the low-light dataset, perform Gamma transformation on the low-light photo P2, randomly brighten or darken it, and then send it to the cross-color feature processing module to obtain a set of feature maps P21. The low-light image P2 is then passed to the weight-sharing cross-color feature processing module without any transformation to obtain another set of feature maps P22. Using Smooth L1 constraint to keep feature map P21 and feature map P22 consistent further ensures the illumination invariance of the features, and is expressed as: ； in, This represents the gamma brightness transformation function. This represents a learnable convolutional kernel, where the subscript i indicates that the kernel is the i-th kernel. This represents the hyperparameters used during the training process. (·) indicates that the i-th convolutional kernel is used. A function to extract cross-color features from an input image.

5. The target detection method based on illumination-invariant features according to claim 1, characterized in that, Step S5 includes: training the network model on the low-light dataset for a regular object detection task; before starting training, loading the network with pre-trained open-source weights under normal lighting data; and fine-tuning a portion of the low-light images during training.

6. A target detection system based on illumination-invariant features, used to implement the method according to any one of claims 1-5; characterized in that, It includes a model building unit, a data acquisition unit, a model learning unit, and a model training unit, among which: The model building unit is used to construct a cross-color feature processing module in a two-dimensional image in the RGB domain; construct an illumination-invariant feature module based on the cross-color feature processing module; and construct a network model based on the illumination-invariant feature module. The data acquisition unit is used to collect low-light images as a low-light dataset and input them into the network model; The model learning unit is used to extract illumination-invariant features using illumination invariance constraints and to guide the learning of the network model module using loss constraints. The model training unit is used to train the network model, obtain the trained network model, and perform target detection in real time by inputting low-light images to obtain the final position and size of the target.

7. An electronic device, characterized in that, It includes a processor and a memory; the processor is used to run the target detection system based on illumination-invariant features as described in claim 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions; when executed, the instructions perform the method as described in any one of claims 1-5.

Images