Infrared image unknown target detection method and device based on feature migration and high-order semantics

Abstract

The application discloses an infrared image unknown target detection method and device based on feature migration and high-order semantics, and belongs to the technical field of multi-modal artificial intelligence, which comprises the following steps: step one, based on the input infrared original image, the feature domain is converted through the generative adversarial network to obtain the visible light domain image feature; step two, the visible light domain image feature output in step one is used to extract high-order semantic features through a multi-modal large model, unknown target detection positioning is carried out based on the infrared original image and the high-order semantic features, and based on the position information of the target output by the model, an image generation tool is used to output the infrared image result with a detection marking box. The application can quickly and accurately complete the infrared image unknown target detection task, has the advantages of high precision and strong generalization ability, can be applied in the fields of security monitoring, disaster prevention and reduction, remote sensing and telemetry, and has a wide industrial application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal artificial intelligence, and more particularly to an infrared image unknown target detection method and device based on feature migration and high-order semantics. BACKGROUND

[0002] The infrared image unknown target detection method has profound practical needs and broad application prospects. With the development of generative artificial intelligence technology, the zero-shot unknown target detection technology based on visible light images has been fully developed, and using large-scale visual language models for target detection or semantic segmentation of visible light images has achieved high accuracy. However, its idea cannot be directly "reused" in the field of infrared images. The reason is that infrared images do not have obvious visual features such as color and shape contour. At present, the unknown target detection technology for infrared images is still in the stage of traditional deep neural network models or hybrid models, and the accuracy is not high, so it is necessary to reform the traditional algorithm combined with the latest technology to improve the accuracy of infrared image unknown target detection. SUMMARY

[0003] The present application aims to overcome the shortcomings of the prior art and provide an infrared image unknown target detection method and device based on feature migration and high-order semantics, which can quickly and accurately complete the infrared image unknown target detection task, has the advantages of high precision and strong generalization ability, and can be applied in the fields of security monitoring, disaster prevention and reduction, remote sensing and telemetry, etc., and has broad industrial application prospects.

[0004] The purpose of the present application is achieved by the following scheme:

[0005] An infrared image unknown target detection method based on feature migration and high-order semantics, comprising:

[0006] Step one, based on the input infrared original image, convert the feature domain through the generative adversarial network to obtain the visible light domain image feature;

[0007] Step two, use the visible light domain image feature output by step one and extract the high-order semantic feature through the multi-modal large model, perform unknown target detection positioning based on the infrared original image and the high-order semantic feature, and use the image generation tool to output the infrared image result with the detection mark box based on the position information of the target output by the model.

[0008] Further, in step one, the infrared original image is converted into a visible light domain image feature through the generative adversarial network, and the specific steps include:

[0009] S1.1, use a visible light and infrared image aligned dataset to construct a global feature set, and train the generative adversarial network to enable it to complete the missing modality feature completion;

[0010] S1.2, using the trained generative adversarial network to derive the input untrained infrared raw image, generating the corresponding visible light image features.

[0011] Further, in step S1.1, the ability to complete the missing modal feature completion is specifically to generate another modal image corresponding to the input single modal image, thereby realizing the mutual conversion of visible light and infrared images.

[0012] Further, in step S1.1, in the process of training the generative adversarial network, the following loss function is used:

[0013] ;

[0014] wherein, represents the total loss of the network, represents the loss when generating infrared image features, is the loss when generating visible light image features, is the loss when discriminating infrared features, is the loss when discriminating visible light features, is the comprehensive weighting average coefficient of the network, is the infrared feature extraction function, is the migration function to the visible light domain, is the infrared raw data, is the visible light raw data, is the visible light feature extraction function, is the migration function to the infrared domain, is the infrared feature value extracted based on the infrared raw data, is the feature value extracted based on the visible light raw data.

[0015] Further, in step S1.2, the use of the trained generative adversarial network to derive the input untrained infrared raw image, generating the corresponding visible light image features, specifically includes the following sub-steps: setting the lower limit of the discrimination threshold to 0.95.

[0016] Further, the comprehensive weighting average coefficient of the network .

[0017] Further, in step two, the visible light domain image features output by step one are used to extract high-order semantic features through a multi-modal large model, unknown target detection and positioning are performed based on the infrared raw image and the high-order semantic features, and based on the position information of the target in the image output by the model, an infrared image result with a detection marking box is output using an image generation tool. Specifically, the following sub-steps are included:

[0018] S2.1, for multi-modal large model construction thinking chain, first generate the prompt word of the picture-text understanding task, analyze the visible light image features of the input, and extract the semantic description of foreground and background separation;

[0019] S2.2, according to the initial semantic features, generate entity extraction task prompt words, and deeply mine high-order semantics;

[0020] S2.3, after extracting the entity in step S2.2, the entity information and the infrared original image are input into the model to generate the target detection task prompt word, and the specific position information of the entity in the image is obtained;

[0021] S2.4, visualize the information, and use the image generation tool to output the infrared image with target marking box as the final result.

[0022] Further, in step S2.1, the semantic description of foreground and background separation is the initial semantic.

[0023] An infrared image unknown target detection device based on feature migration and high-order semantics, comprising a processor and a memory, the memory stores a computer program, when the computer program is loaded by the processor, the method as claimed in any one of the above is executed.

[0024] The beneficial effects of the present application include:

[0025] The present application has the advantages of high precision and strong generalization ability. Specifically, the image feature domain migration is realized by using the generative adversarial network, thereby avoiding the use of a large amount of labeled data for training in the infrared feature domain; high-order semantic extraction is realized by migrating features, entity extraction is realized without labeled data, and the encoding amount during data processing is compressed; the multi-modal large model is used to directly detect and mark the infrared unknown target according to the high-order semantic information, and the visible light domain and infrared domain features are aligned by fully utilizing the pre-training results of the large model. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0027] Figure 1 The new infrared image unknown target detection block diagram in the embodiment of the present application;

[0028] Figure 2 The infrared image feature migration process schematic diagram in the embodiment of the present application;

[0029] Figure 3 FIG. 1 is a schematic diagram of the construction of a thinking chain based on high-order semantic extraction and unknown target detection of a multimodal large model in an embodiment of the present application;

[0030] Figure 4 FIG. 4 is a workflow diagram of the detection method in an embodiment of the present application. DETAILED DESCRIPTION

[0031] All features disclosed in the specification or in any document incorporated by reference can be combined in any combination, except where such combination is not technically feasible.

[0032] The specific implementation process of the present application is as follows:

[0033] In the inventive concept, especially in the field of multi-modal artificial intelligence technology, a detection scheme for unknown targets in infrared images is developed based on generative adversarial networks (GAN) and multi-modal large models (MLLM). The detection scheme uses generative adversarial networks to transfer infrared image features to the visible light feature domain, and realizes the detection and labeling of zero-shot unknown targets based on the high-order semantic features of visible light images. In a preferred embodiment, an infrared image unknown target detection method based on feature transfer and high-order semantics is provided, and the overall principle is as shown in FIG. 1. This embodiment method only needs to input unlabeled infrared images, and after processing by the generative adversarial network and the multi-modal large model, the infrared image with target bounding box is finally output. More specifically, the steps include: Figure 1

[0034] Step one, based on the input infrared original image, convert the feature domain through the generative adversarial network to obtain the easily recognizable visible light domain image features. Referring to FIG. 2, the specific implementation process of this step is as follows: Figure 2

[0035] S1.1, use the visible-infrared image alignment dataset to construct a global feature set, train the generative adversarial network, so that it can complete the missing modality feature completion, that is: input a single modality image to generate another modality image corresponding to it, realize the mutual conversion of visible-infrared images, and train the generative adversarial network as shown in FIG. 3. The loss function used in this step is as follows: Figure 2

[0036]

[0037] wherein, Ltotal represents the total loss of the network, Lir represents the loss when generating infrared image features, Lvis represents the loss when generating visible light image features, Lir_d represents the loss when judging infrared features, Lvis_d represents the loss when judging visible light features,​​​ is the comprehensive weighted average coefficient of the network of the present application.

[0038] S1.2, using a pre-trained adversarial generative network to derive an untrained infrared image of the input, generating a visible light image feature corresponding thereto, setting the lower limit of the discrimination threshold to 0.95, and generating a visible light image with a high degree of approximation.

[0039] Step two, using the visible light domain image feature output in step one, high-order semantic features are extracted through a multi-modal large model, unknown target detection and positioning are performed based on the infrared original image and the high-order semantic features, and based on the position information of the target output by the model, an infrared image result with a detection marking box is output using an image generation tool. Reference Figure 3 The specific implementation process of this step is as follows:

[0040] S2.1, constructing an automatic thinking thought chain for the multi-modal large model, first generating prompt words for a picture-text understanding task, analyzing the input visible light image feature, and extracting semantic descriptions for foreground-background separation (primary semantic), such as: there is an animal in the middle of the image, and the background is a forest;

[0041] S2.2, generating entity extraction task prompt words automatically according to the primary semantic features, and deeply mining high-order semantics, such as: asking which animal, answering that it is a cervid animal, and specifically which deer, the information is insufficient to determine;

[0042] S2.3, by extracting entities in the previous step, the entity information (entity in the example in the previous step is "deer") and the infrared original image are input into the model together to automatically generate target detection task prompt words, and obtain specific position information of the entity in the image;

[0043] S2.4, visualizing the information, i.e. using an image generation tool to output an infrared image with a target marking box as the final result of the algorithm.

[0044] More specifically, in other embodiments, based on the above embodiments, an "end-to-end" idea is designed, a hybrid model architecture is adopted, and a pre-training result is used to directly detect zero-shot unknown targets, which is better than the traditional algorithm using labeled data for supervised learning. The overall workflow is as follows: Figure 4 , specifically including the following steps:

[0045] T1, inputting an infrared image to be detected;

[0046] T2, performing feature transfer based on an adversarial generative network to the visible light feature domain;

[0047] T3, extracting high-order semantic features containing entity information based on the visible light features;

[0048] T4 uses a multimodal large model to perform object detection based on the original image and semantic features;

[0049] T5 outputs the detection results with the target bounding boxes.

[0050] This method offers wider applicability and higher robustness compared to traditional auxiliary information-based models, compatibility judgment models, and hybrid models, representing a new generation of inductive models. Furthermore, this method is an "end-to-end" approach, demonstrating high maturity and applicability across multiple scenarios, including:

[0051] 1) Field survey: Field surveys widely use fixed-position infrared imaging equipment to take long-term nighttime pictures of a certain area, resulting in a large amount of data that needs to be processed. The solution of this invention can help image analysts in this field improve the efficiency of target identification.

[0052] 2) Fire and rescue field: Because infrared imaging equipment can penetrate the obstruction of coverings, firefighters often use infrared scanning equipment to analyze the fire situation in fire scenes. Rescue personnel also use infrared equipment to search for survivors in disaster scenes such as earthquakes and tsunamis. In these scenarios, they often face unknown situations. The solution of this invention can improve the accuracy of target detection.

[0053] 3) Industrial monitoring: In some industrial scenarios that require high temperatures, infrared imaging equipment is widely used to monitor the production process and alarm for potential temperature changes. The solution of this invention can be applied to the industrial monitoring field to shorten the early warning detection time and achieve immediate detection and early warning.

[0054] 4) Aerospace field: Aircraft use infrared imaging equipment to perceive the air and ground situation during night flight to avoid risks. Space launch missions also need to use infrared equipment for tracking and telemetry. The solution of this invention is also applicable to the aerospace field, using situational awareness to assist pilots or operators in handling emergencies.

[0055] In summary, this invention, based on existing artificial intelligence technology and integrating innovative algorithm structures, proposes a novel infrared image unknown target detection scheme that can be applied in various industrial scenarios.

[0056] It should be noted that, within the scope of protection defined in the claims of this invention, the following embodiments can be combined and / or extended or replaced in any logical manner from the above specific embodiments, such as the disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0057] Example 1

[0058] An infrared image unknown target detection method based on feature transfer and higher-order semantics includes:

[0059] Step 1: Based on the input raw infrared image, the feature domain is transformed through an adversarial generative network to obtain the visible light domain image features;

[0060] Step 2: Using the visible light domain image features output in Step 1 and extracting high-order semantic features through a multimodal large model, unknown target detection and localization are performed based on the original infrared image and high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image result with detection bounding boxes.

[0061] Example 2

[0062] Based on Example 1, in step one, the visible light domain image features are obtained by transforming the feature domain of the input raw infrared image through an adversarial generative network, specifically including the following sub-steps:

[0063] S1.1, Construct a global feature set using a dataset aligned with visible light and infrared images, and train an adversarial generative network to complete missing modal features;

[0064] S1.2, using a trained adversarial generative network to deduce the input untrained raw infrared image and generate the corresponding visible light image features.

[0065] Example 3

[0066] Based on Example 2, in step S1.1, the ability to complete missing modal features specifically means that a single modal image can be input to generate another corresponding modal image, thereby enabling the mutual conversion between visible light and infrared images.

[0067] Example 4

[0068] Based on Example 2, in step S1.1, the following loss function is used during the training of the adversarial generative network:

[0069] ;

[0070] in, Indicates the total network loss. This represents the loss during infrared image feature generation. It is the loss during the generation of visible light image features. This is the loss during infrared feature discrimination. This is the loss during visible light feature discrimination. It is the network comprehensive weighted average coefficient. It is an infrared feature extraction function. It is a transfer function to the visible light domain. It is raw infrared data. It is raw visible light data. It is a visible light feature extraction function. It is a transfer function to the infrared domain. It is based on infrared feature values ​​extracted from raw infrared data. These are feature values ​​extracted from raw visible light data.

[0071] Example 5

[0072] Based on Example 4, in step S1.2, the step of using a trained adversarial generative network to deduce the input untrained original infrared image and generate the corresponding visible light image features specifically includes the sub-step of setting the lower limit of the discrimination threshold to 0.95.

[0073] Example 6

[0074] Based on Example 4, the network comprehensive weighted average coefficient .

[0075] Example 7

[0076] Based on Example 1, in step two, the visible light domain image features output in step one are used to extract high-order semantic features through a multimodal large model. Unknown target detection and localization are performed based on the original infrared image and the high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image result with detection bounding boxes. Specifically, this includes the following sub-steps:

[0077] S2.1, Construct a thought chain for the multimodal large model. First, generate prompts for the image and text comprehension task, analyze the features of the input visible light image, and extract semantic descriptions for foreground and background separation.

[0078] S2.2, Based on the initial semantic features, generate entity extraction task prompts and further explore higher-order semantics;

[0079] S2.3 After extracting entities through step S2.2, the entity information and the original infrared image are combined to form a text-image pair input model, generating target detection task prompt words, and obtaining the specific location information of the entity in the image;

[0080] S2.4 Visualize the information by using an image generation tool to output an infrared image with target bounding boxes as the final result.

[0081] Example 8

[0082] Based on Example 7, in step S2.1, the semantic description of the foreground-background separation is a primary semantic.

[0083] Example 9

[0084] An infrared image unknown target detection device based on feature transfer and higher-order semantics includes a processor and a memory. The memory stores a computer program, which, when loaded by the processor, executes the method described in any one of Embodiments 1 to 8.

[0085] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0086] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0087] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. A method for detecting unknown targets in infrared images based on feature transfer and higher-order semantics, characterized in that, include: Step 1: Based on the input raw infrared image, the feature domain is transformed through an adversarial generative network to obtain the visible light domain image features; Step two involves using the visible light domain image features output from step one and extracting high-order semantic features through a multimodal large model. Based on the original infrared image and the high-order semantic features, unknown target detection and localization are performed. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image with detection bounding boxes. This specifically includes the following sub-steps: S2.1, Construct a thought chain for the multimodal large model. First, generate prompts for the image and text comprehension task, analyze the features of the input visible light image, and extract semantic descriptions for foreground and background separation. S2.2, Based on the initial semantic features, generate entity extraction task prompts and further explore higher-order semantics; S2.3 After extracting entities through step S2.2, the entity information and the original infrared image are combined to form a text-image pair input model, generating target detection task prompt words, and obtaining the specific location information of the entity in the image; S2.4 Visualize the information by using an image generation tool to output an infrared image with target bounding boxes as the final result; In step S2.1, the semantic description of the foreground-background separation is a primary semantic.

2. The infrared image unknown target detection method based on feature transfer and higher-order semantics according to claim 1, characterized in that, In step one, the input raw infrared image is transformed into visible light domain image features through an adversarial generative network, specifically including the following sub-steps: S1.1, Construct a global feature set using a dataset aligned with visible light and infrared images, and train an adversarial generative network to complete missing modal features; S1.2, using a trained adversarial generative network to deduce the input untrained raw infrared image and generate the corresponding visible light image features.

3. The infrared image unknown target detection method based on feature transfer and higher-order semantics according to claim 2, characterized in that, In step S1.1, the ability to complete missing modal features specifically means that a single modal image can be input to generate another corresponding modal image, thereby enabling the mutual conversion between visible light and infrared images.

4. The infrared image unknown target detection method based on feature transfer and higher-order semantics according to claim 2, characterized in that, In step S1.1, the following loss function is used during the training of the adversarial generative network: ； in, Indicates the total network loss. This represents the loss during infrared image feature generation. It is the loss during the generation of visible light image features. This is the loss during infrared feature discrimination. This is the loss during visible light feature discrimination. It is the network comprehensive weighted average coefficient. It is an infrared feature extraction function. It is a transfer function to the visible light domain. It is raw infrared data. It is raw visible light data. It is a visible light feature extraction function. It is a transfer function to the infrared domain. It is based on infrared feature values ​​extracted from raw infrared data. These are feature values ​​extracted from raw visible light data.

5. The infrared image unknown target detection method based on feature transfer and higher-order semantics according to claim 4, characterized in that, In step S1.2, the trained adversarial generative network is used to deduce the input untrained raw infrared image and generate corresponding visible light image features. Specifically... This includes the following sub-step: setting the lower limit of the discrimination threshold to 0.

95.

6. The infrared image unknown target detection method based on feature transfer and higher-order semantics according to claim 4, characterized in that, Network comprehensive weighted average coefficient .

7. An infrared image unknown target detection device based on feature transfer and higher-order semantics, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 6.

Images