Target detection method, electronic device and storage medium

By extracting and fusion of infrared and visible light images, the problems of insufficient light and background interference in vehicle detection at night are solved, and the accuracy and visibility of the detection are improved.

CN118691804BActive Publication Date: 2024-12-06HANGZHOU HUACHENG SOFTWARE TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411200788.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-06
Estimated Expiration
2044-08-29

AI Technical Summary

Technical Problem

Night vehicle detection is difficult to accurately detect under factors such as insufficient light, background interference and motion blur, resulting in false alarms and missed alarms.

Method used

The pre-trained generative adversarial model is used to feature extraction of infrared and visible images and the features of the two are fused to improve the accuracy of object detection.

Benefits of technology

By fusing infrared and visible image features, the accuracy of vehicle detection at night is significantly improved, false alarms and missed alarms are reduced, and visibility of vehicle boundaries is enhanced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691804B_ABST
    Figure CN118691804B_ABST
Patent Text Reader

Abstract

The present application discloses a target detection method, an electronic device and a storage medium. The target detection method comprises: acquiring an infrared image and a visible light image acquired by an image acquisition device; extracting features of the visible light image using a pre-trained first generative adversarial model to obtain visible light image features; and extracting features of the infrared image using a pre-trained second generative adversarial model to obtain infrared image features; fusing the visible light image features and the infrared image features to obtain target features; and obtaining target detection results of the infrared image and the visible light image based on the target features. The above scheme can improve the accuracy of target detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a target detection method, an electronic device, and a storage medium. Background Art

[0002] Vehicle detection is mainly used to detect and identify vehicles on the road. Accurate vehicle detection can improve traffic management efficiency, reduce traffic accidents, etc. However, the particularity of the night environment and various challenging factors make night vehicle detection a challenging task.

[0003] First, the lighting conditions in nighttime environments are usually limited. Insufficient lighting makes the appearance and texture of the vehicle blurred, and the boundaries and shapes of the vehicle are difficult to detect accurately. Due to the interference of the background environment, the contrast between the vehicle and the background is low, and the color and brightness difference between the vehicle area and the surrounding environment is not obvious, which can easily lead to false positives and missed positives in vehicle detection. Secondly, there are various strong light sources in nighttime environments, such as street lights, car lights, and reflective signs. These light sources will produce halos, reflections, and other interferences, making the visibility of vehicle boundaries even lower, increasing the difficulty of vehicle detection. Finally, vehicle motion blur will also have a certain impact on nighttime vehicle detection. Summary of the invention

[0004] The present application at least provides a target detection method, an electronic device and a storage medium.

[0005] The present application provides a target detection method, including: acquiring an infrared image and a visible light image acquired by an image acquisition device; performing feature extraction on the visible light image using a pre-trained first generative adversarial model to obtain visible light image features; and performing feature extraction on the infrared image using a pre-trained second generative adversarial model to obtain infrared image features; fusing the visible light image features and the infrared image features to obtain target features; and obtaining target detection results of the infrared image and the visible light image based on the target features.

[0006] The present application provides a target detection device, including: an image acquisition module, a feature extraction module, a feature fusion module and a target detection module; the image acquisition module is used to acquire an infrared image and a visible light image acquired by an image acquisition device; the feature extraction module is used to perform feature extraction on the visible light image using a pre-trained first generative adversarial model to obtain visible light image features; and to perform feature extraction on the infrared image using a pre-trained second generative adversarial model to obtain infrared image features; the feature fusion module is used to fuse the visible light image features and the infrared image features to obtain target features; the target detection module is used to obtain target detection results of the infrared image and the visible light image based on the target features.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned target detection method.

[0008] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned target detection method is implemented.

[0009] The above scheme obtains an infrared image and a visible light image, and then uses the first generative adversarial model to extract features from the visible light image and uses the second generative adversarial model to extract features from the infrared image, and then fuses the extracted infrared image features and visible light image features, so that the target detection results obtained subsequently based on the fused target features are more accurate than the results obtained using only visible light detection.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 It is a flowchart of an embodiment of the target detection method of the present application;

[0013] Figure 2 This is a schematic diagram of the structure of the first feature extraction module in an embodiment of the target detection method of the present application;

[0014] Figure 3 It is a structural diagram of the second feature extraction module in an embodiment of the target detection method of the present application;

[0015] Figure 4 is a schematic diagram of the structure of a detection head in an embodiment of a target detection method of the present application;

[0016] Figure 5 It is a schematic diagram of a first training process for a first generative adversarial model and a second generative adversarial model in an embodiment of a target detection method of the present application;

[0017] Figure 6 It is a schematic diagram of a second training process for the first generative adversarial model and the second generative adversarial model in an embodiment of the target detection method of the present application;

[0018] Figure 7 It is a structural schematic diagram of an embodiment of the target detection device of the present application;

[0019] Figure 8It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0020] Fig. 9 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0021] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0022] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0023] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0024] In some application scenarios, the target detection method provided by the present application can be applied to the computer field, etc. The execution subject for implementing the target detection method described in the present application can be a target detection device, an electronic device, etc. For example, the target detection device can be set in a terminal device or a server or other processing device, wherein the terminal device can be an electronic device, a device (User Equipment, UE), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the target detection method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0025] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of the target detection method of the present application. Figure 1 As shown, the target detection method provided by the embodiment of the present disclosure may include the following steps:

[0026] Step S11: Acquire the infrared image and the visible light image captured by the image acquisition device.

[0027] The image acquisition device may be a device with an image acquisition function such as a camera or a webcam. The infrared image and the visible light image may be acquired from the same area. Exemplarily, the image acquisition device includes a visible light acquisition module and an infrared acquisition module, and the visible light acquisition module and the infrared acquisition module may be acquired from the same road area.

[0028] Step S12: using the pre-trained first generative adversarial model to extract features from the visible light image to obtain visible light image features; and using the pre-trained second generative adversarial model to extract features from the infrared image to obtain infrared image features.

[0029] The first generative adversarial model and the second generative adversarial model are pre-trained. Optionally, the first generative adversarial model and the second generative adversarial model can extract common features in the visible light image and the infrared image.

[0030] Step S13: Fusing the visible light image features and the infrared image features to obtain target features.

[0031] The fusion method includes but is not limited to directly splicing the visible light image features with the infrared image features, or further processing the visible light image and the infrared image separately and then splicing them, or fusing the visible light image features with the infrared image features through a feature fusion network, etc.

[0032] Step S14: Based on the target features, the target detection results of the infrared image and the visible light image are obtained.

[0033] Exemplarily, target features may be input into a detection head to obtain target detection results.

[0034] The above scheme obtains an infrared image and a visible light image, and then uses the first generative adversarial model to extract features from the visible light image and uses the second generative adversarial model to extract features from the infrared image, and then fuses the extracted infrared image features and visible light image features, so that the target detection results obtained subsequently based on the fused target features are more accurate than the results obtained using only visible light detection.

[0035] In some embodiments, the first generative adversarial model includes a first backbone network and a first neck network, and the second generative adversarial model includes a second backbone network and a second neck network. The method of obtaining the visible light image features in the above step S12 can be: using the first backbone network to fuse the shallow features extracted from the visible light image by the first backbone network with the shallow features extracted from the infrared image by the second backbone network to obtain a first fusion result. The first fusion result is used as the input of the first neck network to obtain the visible light image features.

[0036] Optionally, the target detection method can be specifically performed using a target detection model, and the first generative adversarial model and the second generative adversarial model can be part of it. Exemplarily, the target detection model as a whole is based on the YOLO-Lite framework, and is combined with a generative adversarial model (GAN) for inter-modal feature fusion. Specifically, the framework of the target detection model can be divided into four parts: a backbone network (backbone), a neck network (neck), a detection head (head), and a generative adversarial model (GAN), wherein the backbone extracts features from the input image, the neck performs feature fusion on the extracted features, the head outputs the detection results, and the GAN is used for adaptive fusion of features between different modalities. Specifically, the first backbone network and the first neck network are used as generators of the first generative adversarial model, and the second backbone network and the second neck network can be used as generators of the second generative adversarial model. Optionally, the structure of the first backbone network may be the same as or different from the structure of the second backbone network.

[0037] Optionally, the first fusion result includes a multi-scale first fusion feature map.

[0038] Exemplarily, the first backbone network can extract multi-scale features of the visible light image, and the multi-scale first fused feature map can be obtained by fusing the multi-scale features of the visible light image with the multi-scale features of the infrared image. The specific fusion method can be to fuse features of the same scale, so that the number and scale distribution of the first fused feature map are the same as the number and scale distribution of the multi-scale features of the visible light image, or in other embodiments, it can be a fusion of features of different scales, so that the number and scale distribution of the first fused feature map are different from the number and scale distribution of the multi-scale features of the visible light image.

[0039] In the above step S12, the infrared image features can be obtained by: using the second backbone network to fuse the shallow features extracted from the infrared image by the second backbone network with the shallow features extracted from the visible light image by the first backbone network to obtain a second fusion result. The second fusion result is used as the input of the second neck network to obtain the infrared image features. Optionally, the second fusion result includes a multi-scale second fusion feature map.

[0040] Exemplarily, the second backbone network can extract multi-scale features of the infrared image, and the multi-scale second fused feature map can be obtained by fusing the multi-scale features of the infrared image with the multi-scale features of the visible light image. The specific fusion method can be to fuse features of the same scale, so that the number and scale distribution of the second fused feature maps are the same as the number and scale distribution of the multi-scale features of the infrared image, or in other embodiments, it can be a fusion of features of different scales, so that the number and scale distribution of the second fused feature maps are different from the number and scale distribution of the multi-scale features of the infrared image.

[0041] In some embodiments, the first backbone network includes a first feature extraction module and a first feature fusion module after the first feature extraction module, and the second backbone network includes a second feature extraction module and a second feature fusion module after the second feature extraction module. On this basis, the above-mentioned method of fusing the shallow features extracted from the visible light image by the first backbone network with the shallow features extracted from the infrared image by the second backbone network to obtain the first fusion result can be: using the first feature extraction module to extract features from the visible light image to obtain shallow features. Then, the shallow features output by the first feature extraction module are fused with the shallow features output by the second feature extraction module using the first feature fusion module to obtain the first fusion result.

[0042] Similarly, the above-mentioned method of fusing the shallow features extracted from the infrared image by the second backbone network with the shallow features extracted from the visible light image by the first backbone network to obtain the second fusion result can be: using the second feature extraction module to extract features from the infrared image to obtain shallow features. Then, using the second feature fusion module to fuse the shallow features output by the first feature extraction module with the shallow features output by the second feature extraction module to obtain the second fusion result.

[0043] Optionally, the structures of the first feature extraction modules in the first backbone network and the second backbone network may be the same or different. Considering the subsequent model deployment application, two parallel backbone networks are built using a lightweight network to extract multi-scale features of infrared images and visible light images respectively. The two backbone networks may be the same or different. For the sake of simplicity, ShuffleNet-v2 0.5x can be used as the backbone network to extract features of infrared images and visible light images. Exemplarily, the number of first feature extraction modules in the first backbone network may be one or more. This scheme takes three first feature extraction modules arranged in series as an example. The number of second feature extraction modules in the second backbone network may be one or more. This scheme takes three second feature extraction modules arranged in series as an example. The structure of the first feature extraction module can refer to Figure 2For example, the input of the first first feature extraction module can be a visible light image. The input visible light image is input into two channels for processing respectively, and then the features obtained from the two channels are merged, and then the channels are rearranged to obtain the output of the first feature extraction module. The input of the second first feature extraction module can be the output of the previous first feature extraction module.

[0044] Optionally, the first feature fusion module can be located after any first feature extraction module, and the second feature fusion module can be located after any second feature extraction module. As mentioned above, the first feature extraction module and the second feature extraction module may include multiple ones. If the first feature fusion module is located after the last first feature extraction module in the serial setting, the output of the first feature fusion module will be used as the input of the first neck network. If the first feature fusion module is located after the non-last first feature extraction module, the output of the first feature fusion module will be used as the input of the next first feature extraction module, and the output of the last first feature extraction module will be used as the input of the first neck network. Similarly, if the second feature fusion module is located after the last second feature extraction module in the serial setting, the output of the second feature fusion module will be used as the input of the second neck network. If the second feature fusion module is located after the non-last second feature extraction module, the output of the second feature fusion module will be used as the input of the next second feature extraction module, and the output of the last second feature extraction module will be used as the input of the second neck network. The structure of the second feature extraction module can refer to Figure 3 . Exemplarily, the input of the first second feature extraction module can be an infrared image, which is divided into two channels for separate processing, and then the features obtained from the two channels are merged, and then the channels are rearranged to obtain the output of the second feature extraction module, and the input of the second second feature extraction module can be the output of the previous second feature extraction module.

[0045] The above scheme takes into account that the infrared image and visible light image used are from the same scene, and the low-frequency information of the two modalities contains common information, such as background and large-scale environmental features. Therefore, when extracting shallow features in the initial stage of the two backbone networks, two feature fusion modules are used to promote information interaction and fusion between different modalities. The two feature fusion modules can be bidirectional fusion modules (Bi-FM).

[0046] As described above, the first fusion result includes a multi-scale first fusion feature map. And the second fusion result includes a multi-scale second fusion feature map. On this basis, the above method of using the first fusion result as the input of the first neck network to obtain the visible light image feature can be: using the first neck network to fuse the first fusion feature maps to obtain the visible light image feature. On this basis, the above method of using the second fusion result as the input of the second neck network to obtain the infrared image feature can be: using the second neck network to fuse the second fusion feature maps to obtain the infrared image feature.

[0047] The neck network mainly performs feature fusion on the multi-scale feature maps output by the backbone network to further extract high-level semantic information. Optionally, the backbone network and the neck network provided in this embodiment can be an FPN-PAN structure. Feature Pyramid Network (FPN): The main function of FPN is to transfer the semantic information in the deep feature map to the shallow layer through upsampling, so that the shallow feature map can obtain richer semantic information, which is conducive to identifying the target object. This structure helps to improve the accuracy when detecting small targets. Path Aggregation Network (PAN): In contrast to FPN, PAN transfers the shallow position information to the deep layer through downsampling, so that the deep feature map can obtain more accurate position information, which is conducive to accurately locating the target object. This structure helps to improve the positioning accuracy when detecting large targets.

[0048] Optionally, in this embodiment, a variant convolution (Ghost module) module is introduced into the original FPN-PAN structure. The position of the Ghost module in the first neck network and the second neck network is not limited. The specific structure of the Ghost module is not specifically limited here.

[0049] In some embodiments, the above step S13 may include the following steps: inputting the visible light image features into the shared fully connected layer to obtain advanced visible light image features, and then fusing the advanced visible light image features with the advanced infrared image features to obtain the target features.

[0050] In order to extract the features of the infrared image and the visible light image in the common space, a fully connected layer can be shared at the end of the two generators. The fusion method can be to directly splice the advanced visible light image and the advanced infrared image features to obtain the target features, or it can be fused by other weighted fusion methods, which are not specifically limited here.

[0051] Optionally, the target detection result includes the position of the target object in the infrared image and the visible light image, the category of the target object, and the confidence. Optionally, the target object can be any object that needs to be detected, such as a vehicle, an animal, etc. The above-mentioned method of obtaining the target detection results of the infrared image and the visible light image based on the target features can be: inputting the target features into the first channel in the detection head and processing the target features using the first compact convolution layer in the first channel. Then, based on the target features processed by the first compact convolution layer, the category of the target object is obtained. And, inputting the target features into the second channel in the detection head and processing the target features using the second compact convolution layer in the second channel. Then, based on the target features processed by the second compact convolution layer, the position and confidence of the target object in the infrared image and the visible light image are obtained.

[0052] In the object detection problem, regression tasks and detection tasks are mutually exclusive. Based on this, this proposal does not use the original Yolov5-Lite detection head, but further improves it, separates the regression task and the detection task, and adjusts the original Yolov5-Lite detection head. For example, the structure of the detection head can refer to Figure 4 ,On the one hand, the decoupling is divided into three parts, for regression and detection tasks respectively, and on the other hand, compact ,convolution with a larger kernel size is introduced to expand the ,receptive field.

[0053] In some embodiments, the first generative adversarial model and the second generative adversarial model are obtained through game adversarial training, and the method further includes a training step of the first generative adversarial model and the second generative adversarial model. Figure 5 and Figure 6 , the training steps include:

[0054] Step S21: Acquire a sample infrared image and a sample visible light image.

[0055] Capture videos containing vehicles through cameras or other image acquisition devices to ensure that the captured videos cover a variety of different scenes, lighting conditions, and target states. Convert the captured data into images by extracting frames and clean them to remove redundant data. Annotate the target boxes and categories of the obtained sample visible light images of vehicles and the sample infrared images of their corresponding scenes, divide the annotated data set into training sets, validation sets, and test sets according to a certain ratio, and perform data preprocessing operations on the training sets and validation sets.

[0056] Step S22: using the first generative adversarial model to obtain sample visible light image features corresponding to the sample visible light image, and using the second generative adversarial model to obtain sample infrared image features corresponding to the sample infrared image.

[0057] The specific method of obtaining the visible light image features of the sample can refer to the above method of obtaining the visible light image features, and the method of obtaining the infrared image features of the sample can refer to the above method of obtaining the infrared image features, which will not be described in detail here.

[0058] Step S23: obtaining true samples of the first discriminator based on the sample visible light image features and false samples of the first discriminator based on the sample infrared image features; and obtaining false samples of the second discriminator based on the sample visible light image features and true samples of the second discriminator based on the sample infrared image features.

[0059] Among them, the sample visible light image feature can be input into the shared fully connected layer to obtain the first feature, and the sample infrared image feature can be input into the shared fully connected layer to obtain the second feature. On this basis, the above-mentioned method of obtaining the real sample of the first discriminator based on the sample visible light image feature and the method of obtaining the false sample of the first discriminator based on the sample infrared image feature can be: the first feature is spliced ​​with the sample visible light image feature to obtain the real sample of the first discriminator, and the second feature is spliced ​​with the sample visible light image feature to obtain the false sample of the first discriminator.

[0060] The above-mentioned method of obtaining the false sample of the second discriminator based on the sample visible light image feature and obtaining the real sample of the second discriminator based on the sample infrared image feature can be: splicing the first feature with the sample infrared image feature to obtain the false sample of the second discriminator, and splicing the second feature with the sample infrared image feature to obtain the real sample of the second discriminator.

[0061] For example, a fully connected layer is shared at the end of the two modal generators, and the feature output of the RGB image and the IR image after necking is: and , the features after the shared fully connected layer are recorded as and . Generator The corresponding first discriminator ,by and The concatenated features of are real samples. and The concatenated features of are false samples; the generator The corresponding second discriminator ,by and The concatenated features of are real samples. and The concatenated features of are false samples.

[0062] Step S24: Using the discrimination results of the first discriminator and the second discriminator, adjust the parameters in the first generative adversarial model and the second generative adversarial model.

[0063] Optionally, the first discriminator and the second discriminator respectively obtain the probability that each real sample and false sample belongs to the real sample. The goal of the discriminator is to minimize its classification error rate, that is, to maximize the probability of correctly judging the real sample as true and the false sample as false. Therefore, the loss function is calculated by the classification results of the real sample and the false sample, and the parameters in the first generative adversarial model and the second generative adversarial model are updated accordingly, including updating the parameters of the generator in the first generative adversarial model and the second generative adversarial model.

[0064] During model training, the two generative adversarial models conduct game-style adversarial learning to explore the potential common features between various modalities, promote the mutual transfer of features between different modalities, and promote further integration of features between modalities.

[0065] In some embodiments, the parameters in the target detection model can also be adjusted according to the target detection results of the sample visible light image and the sample infrared image obtained by the detection head. Specifically, the parameters of the generators in the first generative adversarial model and the second generative adversarial model, the shared fully connected layer, and the parameters in the detection head can be adjusted. Optionally, the first discriminator and the second discriminator may not be deployed during the application of the target detection model, or the first discriminator and the second discriminator may be deployed but not used to participate in target detection.

[0066] Exemplarily, the target detection result includes the coordinates, category and confidence of the target object. Based on the prior box and the target real box, positive and negative samples are allocated according to the aspect ratio matching strategy, and then the cross entropy loss function is used to calculate the category prediction loss, the binary cross entropy loss function is used to calculate the confidence prediction loss, and the CIOU loss function is used to calculate the border regression prediction loss. The original loss function in the field of generative adversarial models is used to promote the fusion of features between different modalities, so that and are closer. Use gradient descent to update the weight parameters of each node of the network model until the loss converges. Then save the optimal high-precision network model weights, further quantize the model according to the equipment requirements, convert the quantized network model into an offline model and deploy it on the device side.

[0067] See also Figure 7 , Figure 7It is a structural diagram of an embodiment of the target detection device of the present application. The target detection device 50 can execute the above-mentioned target detection method. The target detection device 50 includes an image acquisition module 51, a feature extraction module 52, a feature fusion module 53 and a target detection module 54; the image acquisition module 51 is used to acquire the infrared image and the visible light image acquired by the image acquisition device; the feature extraction module 52 is used to extract features of the visible light image using a pre-trained first generative adversarial model to obtain visible light image features; and to extract features of the infrared image using a pre-trained second generative adversarial model to obtain infrared image features; the feature fusion module 53 is used to fuse the visible light image features and the infrared image features to obtain target features; the target detection module 54 is used to obtain target detection results of the infrared image and the visible light image based on the target features.

[0068] The above scheme obtains an infrared image and a visible light image, and then uses the first generative adversarial model to extract features from the visible light image and uses the second generative adversarial model to extract features from the infrared image, and then fuses the extracted infrared image features and visible light image features, so that the target detection results obtained subsequently based on the fused target features are more accurate than the results obtained using only visible light detection.

[0069] Among them, the functions of each module can be found in the target detection method embodiment, and will not be repeated here.

[0070] See also Figure 8 , Figure 8 6 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in any of the above target detection method embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a carrier device such as a laptop computer and a tablet computer, which is not limited here.

[0071] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above-mentioned target detection method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0072] The above scheme obtains an infrared image and a visible light image, and then uses the first generative adversarial model to extract features from the visible light image and uses the second generative adversarial model to extract features from the infrared image, and then fuses the extracted infrared image features and visible light image features, so that the target detection results obtained subsequently based on the fused target features are more accurate than the results obtained using only visible light detection.

[0073] See also Fig. 9 , Fig. 9 The computer-readable storage medium 70 stores program instructions 701, which implement the steps in any of the above target detection method embodiments when executed by a processor.

[0074] The above scheme obtains an infrared image and a visible light image, and then uses the first generative adversarial model to extract features from the visible light image and uses the second generative adversarial model to extract features from the infrared image, and then fuses the extracted infrared image features and visible light image features, so that the target detection results obtained subsequently based on the fused target features are more accurate than the results obtained using only visible light detection.

[0075] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0076] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0077] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0078] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A target detection method, characterized in that: include: Acquire infrared images and visible light images captured by an image acquisition device; Using a pre-trained first generative adversarial model to extract features from the visible light image to obtain visible light image features; and using a pre-trained second generative adversarial model to extract features from the infrared image to obtain infrared image features; fusing the visible light image features and the infrared image features to obtain target features; Based on the target features, obtaining target detection results of the infrared image and the visible light image; The first generative adversarial model includes a first backbone network and a first neck network, the second generative adversarial model includes a second backbone network and a second neck network, and the use of the pre-trained first generative adversarial model to extract features from the visible light image to obtain visible light image features includes: using the first backbone network to fuse shallow features extracted from the visible light image by the first backbone network with shallow features extracted from the infrared image by the second backbone network to obtain a first fusion result; using the first fusion result as an input to the first neck network to obtain the visible light image features; The method uses a pre-trained second generative adversarial model to extract features from the infrared image to obtain infrared image features, including: using the second backbone network to fuse shallow features extracted from the infrared image by the second backbone network with shallow features extracted from the visible light image by the first backbone network to obtain a second fusion result; and using the second fusion result as an input to the second neck network to obtain the infrared image features.

2. The method according to claim 1, characterized in that The first backbone network includes a first feature extraction module and a first feature fusion module behind the first feature extraction module, the second backbone network includes a second feature extraction module and a second feature fusion module behind the second feature extraction module, and the shallow features extracted from the visible light image by the first backbone network and the shallow features extracted from the infrared image by the second backbone network are fused using the first backbone network to obtain a first fusion result, including: Using the first feature extraction module to extract features from the visible light image to obtain shallow features; Using the first feature fusion module to fuse the shallow features output by the first feature extraction module with the shallow features output by the second feature extraction module to obtain the first fusion result; And / or, using the second backbone network to fuse the shallow features extracted from the infrared image by the second backbone network with the shallow features extracted from the visible light image by the first backbone network to obtain a second fusion result, including: Using the second feature extraction module to extract features from the infrared image to obtain shallow features; The second feature fusion module is used to fuse the shallow features output by the first feature extraction module with the shallow features output by the second feature extraction module to obtain the second fusion result.

3. The method according to claim 1, characterized in that The first fusion result includes a multi-scale first fusion feature map, and the first fusion result is used as the input of the first neck network to obtain the visible light image feature, including: Using the first neck network to fuse the first fused feature maps to obtain the visible light image feature; And / or, the second fusion result includes a multi-scale second fusion feature map, and the second fusion result is used as the input of the second neck network to obtain the infrared image feature, including: The second fusion feature maps are fused using the second neck network to obtain the infrared image features.

4. The method according to claim 1, characterized in that: The fusing the visible light image features and the infrared image features to obtain target features includes: Inputting the visible light image features into a shared fully connected layer to obtain advanced visible light image features, and inputting the infrared image features into the shared fully connected layer to obtain advanced infrared image features; The advanced visible light image feature and the advanced infrared image feature are fused to obtain the target feature.

5. The method according to any one of claims 1 to 4, characterized in that: The target detection result includes the position of the target object in the infrared image and the visible light image, the category of the target object, and the confidence level. The target detection results of the infrared image and the visible light image are obtained based on the target features, including: Inputting the target feature into a first channel in a detection head and processing the target feature using a first compact convolutional layer in the first channel; Obtaining a category of the target object based on the target features processed by the first compact convolutional layer; and, inputting the target feature into a second channel in the detection head and processing the target feature using a second compact convolutional layer in the second channel; Based on the target features processed by the second compact convolutional layer, the position and confidence of the target object in the infrared image and the visible light image are obtained.

6. The method according to any one of claims 1 to 4, characterized in that: The first generative adversarial model and the second generative adversarial model are obtained through game adversarial training. The method further includes a training step of the first generative adversarial model and the second generative adversarial model, and the training step includes: Acquire a sample infrared image and a sample visible light image; Using the first generative adversarial model to obtain a sample visible light image feature corresponding to the sample visible light image, and using the second generative adversarial model to obtain a sample infrared image feature corresponding to the sample infrared image; Obtaining true samples of a first discriminator based on the sample visible light image features and obtaining false samples of the first discriminator based on the sample infrared image features; and obtaining false samples of a second discriminator based on the sample visible light image features and obtaining true samples of the second discriminator based on the sample infrared image features; The parameters in the first generative adversarial model and the second generative adversarial model are adjusted using the discrimination results of the first discriminator and the second discriminator.

7. The method according to claim 6, characterized in that The method further comprises: Inputting the sample visible light image feature into a shared fully connected layer to obtain a first feature, and inputting the sample infrared image feature into the shared fully connected layer to obtain a second feature; The method of obtaining a true sample of the first discriminator based on the sample visible light image feature and obtaining a false sample of the first discriminator based on the sample infrared image feature includes: splicing the first feature with the sample visible light image feature to obtain a true sample of the first discriminator, and splicing the second feature with the sample visible light image feature to obtain a false sample of the first discriminator; The method of obtaining a false sample of the second discriminator based on the visible light image feature of the sample and obtaining a true sample of the second discriminator based on the infrared image feature of the sample includes: The first feature is concatenated with the sample infrared image feature to obtain a false sample of the second discriminator, and the second feature is concatenated with the sample infrared image feature to obtain a true sample of the second discriminator.

8. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.