Infrared image unknown target detection method and device based on feature migration and high-order semantics

Abstract

The invention discloses an infrared image unknown target detection method and device based on feature migration and high-order semantics, and belongs to the technical field of multi-modal artificial intelligence, and the method comprises the steps: 1, converting a feature domain through an adversarial generative network based on an input infrared original image, and obtaining a visible light domain image feature; step 2, extracting high-order semantic features by using the visible light domain image features output in the step 1 and through a multi-modal large model, detecting and positioning an unknown target based on the infrared original image and the high-order semantic features, and determining the position information of the target in the image based on the position information of the target output by the model; and outputting an infrared image result with the detection mark box by using an image generation tool. The method can quickly and accurately complete the unknown target detection task of the infrared image, has the advantages of high precision, strong generalization ability and the like, can be applied to the fields of security monitoring, disaster prevention and reduction, remote sensing and telemetering and the like, and has a wide industrial application prospect.

Description

Technical Field

[0001] This invention relates to the field of multimodal artificial intelligence technology, and more specifically, to a method and apparatus for detecting unknown targets in infrared images based on feature transfer and higher-order semantics. Background Technology

[0002] Infrared image unknown target detection methods have profound practical needs and broad application prospects. With the development of generative artificial intelligence technology, zero-shot unknown target detection technology based on visible light images has been fully developed, and high accuracy has been achieved in target detection or semantic segmentation of visible light images using large-scale visual language models. However, the ideas cannot be directly "reused" in the field of infrared images. The reason is that infrared images lack obvious visual features such as color and shape contours. Currently, unknown target detection technology for infrared images is still in the stage of traditional deep neural network models or hybrid models, with low accuracy. Therefore, it is necessary to combine the latest technologies to modify traditional algorithms to improve the accuracy of unknown target detection in infrared images. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and device for detecting unknown targets in infrared images based on feature transfer and higher-order semantics. This method and device can quickly and accurately complete the task of detecting unknown targets in infrared images. It has the advantages of high accuracy and strong generalization ability and can be applied in fields such as security monitoring, disaster prevention and mitigation, and remote sensing and telemetry, and has broad industrial application prospects.

[0004] The objective of this invention is achieved through the following solution: An infrared image unknown target detection method based on feature transfer and higher-order semantics includes: Step 1: Based on the input raw infrared image, the feature domain is transformed through an adversarial generative network to obtain the visible light domain image features; Step 2: Using the visible light domain image features output in Step 1 and extracting high-order semantic features through a multimodal large model, unknown target detection and localization are performed based on the original infrared image and high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image result with detection bounding boxes.

[0005] Further, in step one, the visible light domain image features are obtained by transforming the feature domain of the input raw infrared image through an adversarial generative network, specifically including the following sub-steps: S1.1, Construct a global feature set using a dataset aligned with visible light and infrared images, and train an adversarial generative network to complete missing modal features; S1.2, using a trained adversarial generative network to deduce the input untrained raw infrared image and generate the corresponding visible light image features.

[0006] Furthermore, in step S1.1, the ability to complete missing modal features specifically means that a single modal image can be input to generate another corresponding modal image, thereby enabling the mutual conversion between visible light and infrared images.

[0007] Furthermore, in step S1.1, during the training of the adversarial generative network, the following loss function is used: ; in, Indicates the total network loss. This represents the loss during infrared image feature generation. It is the loss during the generation of visible light image features. This is the loss during infrared feature discrimination. This is the loss during visible light feature discrimination. It is the network comprehensive weighted average coefficient. It is an infrared feature extraction function. It is a transfer function to the visible light domain. It is raw infrared data. It is raw visible light data. It is a visible light feature extraction function. It is a transfer function to the infrared domain. It is based on infrared feature values ​​extracted from raw infrared data. These are feature values ​​extracted from raw visible light data.

[0008] Further, in step S1.2, the step of using the trained adversarial generative network to deduce the input untrained original infrared image and generate the corresponding visible light image features specifically includes the sub-step of setting the lower limit of the discrimination threshold to 0.95.

[0009] Furthermore, the network comprehensive weighted average coefficient .

[0010] Further, in step two, the visible light domain image features output in step one are used to extract high-order semantic features through a multimodal large model. Unknown target detection and localization are performed based on the original infrared image and the high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image result with detection bounding boxes. This specifically includes the following sub-steps: S2.1, Construct a thought chain for the multimodal large model. First, generate prompts for the image and text comprehension task, analyze the features of the input visible light image, and extract semantic descriptions for foreground and background separation. S2.2, Based on the initial semantic features, generate entity extraction task prompts and further explore higher-order semantics; S2.3 After extracting entities through step S2.2, the entity information and the original infrared image are combined to form a text-image pair input model, generating target detection task prompt words, and obtaining the specific location information of the entity in the image; S2.4 Visualize the information by using an image generation tool to output an infrared image with target bounding boxes as the final result.

[0011] Further, in step S2.1, the semantic description of the foreground-background separation is a primary semantic.

[0012] An infrared image unknown target detection device based on feature transfer and higher-order semantics includes a processor and a memory, wherein the memory stores a computer program that executes the method described in any of the preceding methods when the computer program is loaded by the processor.

[0013] The beneficial effects of this invention include: This invention boasts advantages such as high accuracy and strong generalization ability. Specifically, it employs a generative adversarial network to achieve image feature domain transfer, thereby avoiding the need for extensive training with labeled data in the infrared feature domain; it uses transferred features for high-order semantic extraction, enabling entity extraction from unlabeled data and compressing the amount of encoding required for data processing; and it utilizes a multimodal large model to directly detect and label unknown infrared targets based on high-order semantic information, fully leveraging the pre-training results of the large model to achieve feature alignment between the visible and infrared domains. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a block diagram of a novel infrared image unknown target detection method in an embodiment of the present invention; Figure 2 This is a schematic diagram of the infrared image feature migration process in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the construction of the thought chain for high-order semantic extraction and unknown target detection based on a multimodal large model in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the detection method of an embodiment of the present invention. Detailed Implementation

[0016] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0017] The specific implementation process of this invention is as follows: The invention specifically relates to the field of multimodal artificial intelligence technology, developing a detection scheme for unknown targets in infrared images based on Generative Adversarial Networks (GANs) and Multimodal Large Models (MLLMs). This detection scheme utilizes GANs to transfer infrared image features to the visible light feature domain, achieving zero-sample detection and labeling of unknown targets based on high-order semantic features of the visible light image. In a preferred embodiment, a method for detecting unknown targets in infrared images based on feature transfer and high-order semantics is provided, the overall principle of which is as follows: Figure 1 As shown, this embodiment of the method only requires an unlabeled infrared image as input. After processing by a generative adversarial network and a multimodal large model, the final output is an infrared image with target bounding boxes. More specifically, it includes the following steps: Step one: Based on the input raw infrared image, an adversarial generative network is used to transform the feature domain, obtaining easily identifiable visible light domain image features. (Reference) Figure 2 The specific implementation process of this step is as follows: S1.1, a global feature set is constructed using a dataset aligned with visible light and infrared images. An adversarial generative network is trained to complete missing modal features; that is, an input image of a single modality can generate a corresponding image of another modality, achieving mutual conversion between visible light and infrared images. The training is as follows: Figure 2 The adversarial generative network shown uses the following loss function:

[0018] in, Indicates the total network loss. This represents the loss during infrared image feature generation. It is the loss during the generation of visible light image features. This is the loss during infrared feature discrimination. This is the loss during visible light feature discrimination. It is the comprehensive weighted average coefficient of the network of this invention.

[0019] S1.2, use a pre-trained adversarial generative network to deduce the input untrained infrared image, generate the corresponding visible light image features, set the lower limit of the discrimination threshold to 0.95, and generate a visible light image with a high degree of approximation.

[0020] Step two: Using the visible light domain image features output from step one, high-order semantic features are extracted through a multimodal large model. Unknown target detection and localization are then performed based on the original infrared image and the high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image with detection bounding boxes. (Reference) Figure 3 The specific implementation process of this step is as follows: S2.1, Construct an automatic thinking chain for the multimodal large model. First, generate prompts for the image and text comprehension task, analyze the features of the input visible light image, and extract semantic descriptions of foreground and background separation (basic semantics), such as: There is an animal in the middle of the image and the background is a forest. S2.2, Based on the initial semantic features, automatically generate entity extraction task prompts and delve deeper into higher-order semantics. For example, if the question is about what kind of animal it is, the answer is a deer, but there is insufficient information to determine which type of deer it is. S2.3, by extracting entities in the previous step, the entity information (the entity in the previous example is "deer") and the original infrared image are combined to form a text-image pair input model, automatically generating target detection task prompts and obtaining the specific location information of the entity in the image; S2.4 Visualize the information, that is, use an image generation tool to output an infrared image with target bounding boxes as the final result of the algorithm.

[0021] More specifically, in other embodiments, based on the above embodiments, an "end-to-end" design is adopted, using a hybrid model architecture to directly detect zero-shot unknown targets based on pre-training results. This approach is superior to traditional algorithms using labeled data for supervised learning. The overall workflow is referenced... Figure 4 Specifically, it includes the following steps: T1, input the infrared image to be detected; T2 uses a generative adversarial network to perform feature transfer, transferring the feature to the visible light feature domain. T3 extracts high-order semantic features containing entity information based on visible light features; T4 uses a multimodal large model to perform object detection based on the original image and semantic features; T5 outputs the detection results with the target bounding boxes.

[0022] This method offers wider applicability and higher robustness compared to traditional auxiliary information-based models, compatibility judgment models, and hybrid models, representing a new generation of inductive models. Furthermore, this method is an "end-to-end" approach, demonstrating high maturity and applicability across multiple scenarios, including: 1) Field survey: Field surveys widely use fixed-position infrared imaging equipment to take long-term nighttime pictures of a certain area, resulting in a large amount of data that needs to be processed. The solution of this invention can help image analysts in this field improve the efficiency of target identification. 2) Fire and rescue field: Because infrared imaging equipment can penetrate the obstruction of coverings, firefighters often use infrared scanning equipment to analyze the fire situation in fire scenes. Rescue personnel also use infrared equipment to search for survivors in disaster scenes such as earthquakes and tsunamis. In these scenarios, they often face unknown situations. The solution of this invention can improve the accuracy of target detection. 3) Industrial monitoring: In some industrial scenarios that require high temperatures, infrared imaging equipment is widely used to monitor the production process and alarm for potential temperature changes. The solution of this invention can be applied to the industrial monitoring field to shorten the early warning detection time and achieve immediate detection and early warning. 4) Aerospace field: Aircraft use infrared imaging equipment to perceive the air and ground situation during night flight to avoid risks. Space launch missions also need to use infrared equipment for tracking and telemetry. The solution of this invention is also applicable to the aerospace field, using situational awareness to assist pilots or operators in handling emergencies. In summary, based on existing artificial intelligence technology, this invention integrates innovative algorithm structures to propose a novel infrared image unknown target detection scheme that can be applied in various industrial scenarios.

[0023] It should be noted that, within the scope of protection defined in the claims of this invention, the following embodiments can be combined and / or extended or replaced in any logical manner from the above specific embodiments, such as the disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0024] Example 1 An infrared image unknown target detection method based on feature transfer and higher-order semantics includes: Step 1: Based on the input raw infrared image, the feature domain is transformed through an adversarial generative network to obtain the visible light domain image features; Step 2: Using the visible light domain image features output in Step 1 and extracting high-order semantic features through a multimodal large model, unknown target detection and localization are performed based on the original infrared image and high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image result with detection bounding boxes.

[0025] Example 2 Based on Example 1, in step one, the visible light domain image features are obtained by transforming the feature domain of the input raw infrared image through an adversarial generative network, specifically including the following sub-steps: S1.1, Construct a global feature set using a dataset aligned with visible light and infrared images, and train an adversarial generative network to complete missing modal features; S1.2, using a trained adversarial generative network to deduce the input untrained raw infrared image and generate the corresponding visible light image features.

[0026] Example 3 Based on Example 2, in step S1.1, the ability to complete missing modal features specifically means that a single modal image can be input to generate another corresponding modal image, thereby enabling the mutual conversion between visible light and infrared images.

[0027] Example 4 Based on Example 2, in step S1.1, the following loss function is used during the training of the adversarial generative network: ; in, Indicates the total network loss. This represents the loss during infrared image feature generation. It is the loss during the generation of visible light image features. This is the loss during infrared feature discrimination. This is the loss during visible light feature discrimination. It is the network comprehensive weighted average coefficient. It is an infrared feature extraction function. It is a transfer function to the visible light domain. It is raw infrared data. It is raw visible light data. It is a visible light feature extraction function. It is a transfer function to the infrared domain. It is based on infrared feature values ​​extracted from raw infrared data. These are feature values ​​extracted from raw visible light data.

[0028] Example 5 Based on Example 4, in step S1.2, the step of using a trained adversarial generative network to deduce the input untrained original infrared image and generate the corresponding visible light image features specifically includes the sub-step of setting the lower limit of the discrimination threshold to 0.95.

[0029] Example 6 Based on Example 4, the network comprehensive weighted average coefficient .

[0030] Example 7 Based on Example 1, in step two, the visible light domain image features output in step one are used to extract high-order semantic features through a multimodal large model. Unknown target detection and localization are performed based on the original infrared image and the high-order semantic features. Based on the target's position information in the image output by the model, an image generation tool is used to output an infrared image result with detection bounding boxes. Specifically, this includes the following sub-steps: S2.1, Construct a thought chain for the multimodal large model. First, generate prompts for the image and text comprehension task, analyze the features of the input visible light image, and extract semantic descriptions for foreground and background separation. S2.2, Based on the initial semantic features, generate entity extraction task prompts and further explore higher-order semantics; S2.3 After extracting entities through step S2.2, the entity information and the original infrared image are combined to form a text-image pair input model, generating target detection task prompt words, and obtaining the specific location information of the entity in the image; S2.4 Visualize the information by using an image generation tool to output an infrared image with target bounding boxes as the final result.

[0031] Example 8 Based on Example 7, in step S2.1, the semantic description of the foreground-background separation is a primary semantic.

[0032] Example 9 An infrared image unknown target detection device based on feature transfer and higher-order semantics includes a processor and a memory. The memory stores a computer program, which, when loaded by the processor, executes the method described in any one of Embodiments 1 to 8.

[0033] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0034] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0035] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. An infrared image unknown target detection method based on feature migration and high-order semantics, characterized in that, The method comprises the following steps: Step one, based on the input infrared original image, the feature domain is converted through the generative adversarial network to obtain the visible light domain image feature; Step two, using the visible light domain image feature output by step one and through the multimodal large model to extract high-order semantic features, based on the infrared original image and the high-order semantic features to detect and locate the unknown target, based on the position information of the target in the image output by the model, using the image generation tool to output the infrared image result with the detection mark box.

2. The method of claim 1, wherein, In step one, based on the input infrared original image, the feature domain is converted through the generative adversarial network to obtain the visible light domain image feature, which specifically comprises the following sub-steps: S1.1, using the visible light and infrared image aligned dataset to construct the global feature set, training the generative adversarial network to enable it to complete the missing modal feature completion; S1.2, using the trained generative adversarial network to derive the input untrained infrared original image to generate the corresponding visible light image feature.

3. The method of claim 2, wherein the method further comprises: In step S1.1, the ability to complete the missing modal feature completion specifically refers to the ability to generate another modal image corresponding to the input single modal image, thereby realizing the mutual conversion of visible light and infrared images.

4. The method of claim 2, wherein the method further comprises: In step S1.1, the following loss function is used in the process of training the generative adversarial network: ； wherein, represents the total loss of the network, represents the loss during generation of infrared image features, is the loss during generation of visible light image features, is the loss during infrared feature discrimination, is the loss during visible light feature discrimination, is the comprehensive weighted average coefficient of the network, is an infrared feature extraction function, is a migration function to the visible light domain, is infrared raw data, is visible light raw data, is a visible light feature extraction function, is a migration function to the infrared domain, is an infrared feature value extracted based on the infrared raw data, is a feature value extracted based on the visible light raw data.

5. The method of claim 4, wherein, In step S1.2, using the trained generative adversarial network to derive the input untrained infrared original image to generate the corresponding visible light image feature, specifically comprises the following sub-steps: setting the lower limit of the discrimination threshold to 0.

95.

6. The method of claim 4, wherein the method further comprises: Network comprehensive weighted average coefficient .

7. The method of claim 1, wherein the method further comprises: In step two, using the visible light domain image feature output by step one and through the multimodal large model to extract high-order semantic features, based on the infrared original image and the high-order semantic features to detect and locate the unknown target, based on the position information of the target in the image output by the model, using the image generation tool to output the infrared image result with the detection mark box, specifically comprising the following sub-steps: S2.1, constructing a thought chain for the multimodal large model, first generating the prompt words of the image-text understanding task, analyzing the input visible light image feature, and extracting the semantic description of foreground and background separation; S2.2, generating the entity extraction task prompt word according to the primary semantic feature, and deeply mining the high-order semantic; S2.3, after extracting the entity through step S2.2, inputting the entity information and the infrared original image into the model to generate the target detection task prompt word, and obtaining the specific position information of the entity in the image; S2.4, visualizing the information, and using the image generation tool to output the infrared image with the target mark box as the final result.

8. The method of claim 7, wherein the method further comprises: In step S2.1, the semantic description of foreground and background separation is the primary semantic.

9. An infrared image unknown target detection device based on feature migration and high-order semantics, characterized in that, The device comprises a processor and a memory, and the memory stores a computer program which, when loaded by the processor, executes the method of any one of claims 1-8.

Images