Infrared image confrontation generation method based on physical constraint and semantic shape inference

Abstract

The invention relates to the technical field of artificial intelligence image generation, in particular to an infrared image confrontation generation method based on physical constraint and semantic shape inference, and the method comprises the steps: inputting a semantic mask into a semantic-to-target instance conversion module, generating a corresponding semantic instance mask, screening out a target instance of a high-value category, and storing the target instance in a semantic-to-target instance conversion module; calculating a bounding box of the target instance of the high-value category so as to obtain a cutting parameter of the target instance of the high-value category; inputting the semantic instance mask and the cutting parameter of the target instance of the high-value category into a semantic shape inference module of a physical constraint composed of a cross attention network and a temperature and emissivity look-up table unit, and generating the physical constraint; and inputting the random noise, the semantic mask and the generated physical constraint into a guide generation model obtained through adversarial learning pre-training, and finally generating an infrared image which is accurate in structural details and physical characteristics by the guide generation model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence image generation technology, and in particular to an infrared image adversarial generation method based on physical constraints and semantic shape inference. Background Art

[0002] As a key form of visual information, infrared images are widely used in fields such as security monitoring, industrial inspection, and medical imaging. Infrared images are generated by detecting an object's thermal radiation. Therefore, they can effectively capture a target's thermal signature even in low-light or even no-light environments. Compared to visible light images, infrared images possess unique thermal radiation properties, clearly displaying the target's temperature distribution and thermal characteristics. This makes them valuable for tasks such as target detection, thermal anomaly identification, and temperature monitoring.

[0003] Currently, generating highly realistic infrared images still faces numerous challenges. Although deep learning techniques, particularly generative adversarial networks (GANs), have demonstrated impressive performance in image generation tasks, their application to high-quality infrared image synthesis still faces challenges such as thermal feature modeling and detail reproduction. Currently proposed methods generally employ the GAN framework, leveraging its powerful distributional learning capabilities to fit the feature distribution of real infrared images, thereby generating simulated images that closely resemble real images in both visual perception and thermal characteristics. The specific process involves: first, infrared images are preprocessed, such as through denoising and contrast enhancement, to improve the representation of key thermal features and construct clean data input. Next, a generator generates an initial infrared image based on a random input vector, and a discriminator determines its authenticity. As training progresses, the generator continuously optimizes its ability to fit the image's thermal features, while the discriminator improves its accuracy in distinguishing between real and synthetic images. Through a game-playing training mechanism between the generator and discriminator, the model ultimately generates high-quality image samples that closely resemble real infrared images in both thermal characteristics and structural texture.

[0004] However, GAN-based infrared image generation methods have two shortcomings. First, the generated infrared images lack physical consistency. GAN-based infrared image generation methods focus primarily on visual realism, but due to a lack of constraints on physical laws, the generated infrared images may not accurately reflect actual physical properties, which limits their reliability in engineering applications. Second, there is insufficient detail fidelity. Due to insufficient or unevenly distributed training data, GAN-generated infrared images may exhibit blurring or distortion in detailed features such as edges and textures. This is particularly serious in complex scenes, making the generated infrared images difficult to meet the requirements of high-precision simulation.

[0005] In order to ensure that the generated infrared images can accurately reflect the actual physical properties and ensure their scientificity and reliability in engineering applications, a research team has proposed a method to couple image generation and physical quantities through a simulation data set with physical quantity parameters, providing physical quantities and semantics as guidance for GAN training, thereby guiding the generator to generate infrared images according to physical quantities and semantics during forward propagation, and using the discriminator to evaluate and provide feedback on the physical characteristics and semantic layout of the generated infrared images. During the training process, the generator and discriminator are continuously optimized through adversarial games. The discriminator improves its ability to determine whether an image has accurate physical characteristics by extracting deep features and combining them with physical semantic information. The generator gradually learns the physical and semantic characteristics of real infrared images, thereby generating more realistic infrared images.

[0006] However, this method still has two shortcomings. First, it is heavily dependent on simulated datasets, that is, it needs to rely on multiple information sources such as temperature, material semantics, and material emissivity. If the generative model is applied to measured data with only rough semantic shapes, the model performance will be significantly reduced. Second, the generated content is still not realistic enough. Because this type of model mainly relies on simulated datasets for physical quantity and semantic guidance without introducing measured data, the generated infrared images may not be completely consistent with the measured infrared images in terms of detailed features such as edges and textures. Especially in complex scenes, the generated infrared images have difficulty simulating the complex content changes of the measured data. Summary of the Invention

[0007] In view of this, an embodiment of the present application proposes an infrared image adversarial generation method based on physical constraints and semantic shape inference, which effectively solves the problem of difficulty in accurately characterizing the physical radiation characteristics of the generated target due to the lack of physical properties and material semantic information in the measured data. It can generate high-quality infrared images with structural details and physical laws to meet the actual needs of various applications.

[0008] In the first aspect, an embodiment of the present application proposes an infrared image adversarial generation method based on physical constraints and semantic shape inference, which is suitable for high-quality generation of infrared images. The method comprises the following steps: inputting a semantic mask into a semantic to target instance conversion module to generate a corresponding semantic instance mask, screening out target instances of high-value categories, and calculating the bounding box of the target instances of high-value categories, thereby obtaining the cropping parameters of the target instances of high-value categories; inputting the semantic instance mask and the cropping parameters of the target instances of high-value categories into a physical constraint semantic shape inference module composed of a cross-attention network and a temperature and emissivity lookup table unit, and through the cross-attention network, the semantic instance mask is converted into a target instance. The example mask and the cropping parameters of the target instance of the high-value category are used as the query set, and a small number of semantic contours and their structural information of the same category are used as the support set. The semantic information of the structural material is inferred based on the query set and the support set. The structural material semantic information is mapped through the temperature and emissivity lookup table unit to obtain the average emissivity matrix and temperature matrix, which are used as the generated physical constraints; random noise, semantic mask and generated physical constraints are input into the guided generation model obtained by adversarial learning pre-training. The guided generation model injects the generated physical constraints, semantic mask and random noise in sequence through multi-layer guidance, and finally generates an infrared image that is accurate in both structural details and physical properties.

[0009] Optionally, infrared images and their semantic segmentation maps in the MSRS dataset are selected as measured datasets for training and testing, and cars, pedestrians, and bicycles in the measured dataset are defined as target instances of high-value categories, and curves, parking spaces, guardrails, color cones, and bumps in the measured dataset are defined as target instances of low-value categories.

[0010] Optionally, inputting the semantic mask into a semantic-to-target-instance conversion module to generate a corresponding semantic instance mask includes:

[0011] The semantic segmentation map in the measured dataset is input into the semantic to target instance conversion module as a semantic mask M(x,y), M(x,y)∈{1,2,…,C}, C represents the total number of categories, (x,y) represents the coordinates of the pixel point;

[0012] For each category in the semantic mask, the semantic to target instance conversion module generates a corresponding binary mask B c (x,y), B c The definition of (x,y) is as follows:

[0013]

[0014] Among them, M(x,y)=c means that M(x,y) is of category c, and M(x,y)≠c means that M(x,y) is not of category c;

[0015] Binarization mask B c (x,y) performs connected component analysis to obtain the binary mask B c (x,y) corresponds to the instance region B of category c c,i ,i∈{1,2,…,N c}, i represents the i-th instance under category c, N c represents the total number of instances under category c;

[0016] Connected component analysis is expressed as:

[0017]

[0018] In the process of connected component analysis, each connected region R c,i Each instance mask I(x,y) is assigned a globally unique instance number g that increases continuously. By concatenating the instance mask I(x,y) with the semantic mask M(x,y), the corresponding semantic instance mask L(x,y) is generated, where L(x,y) = [M(x,y), I(x,y)].

[0019] Optionally, screening out target instances of high-value categories and calculating bounding boxes of target instances of high-value categories to obtain cropping parameters of target instances of high-value categories includes:

[0020] Extract the instance number and cropping parameters of the specified high-value category in each image;

[0021] Assume that the high value category is C h , then for each category c h ∈C h , extract the corresponding binary mask according to the semantic mask M(x,y) When the category of M(x,y) is c h At this time The value is I(x,y), otherwise it is -1;

[0022] Then according to Each pixel instance number i in further extracts its binary mask Then pass To calculate the bounding box of the target instance of the high-value category, and thus obtain the cropping parameters of the target instance of the high-value category, the process is expressed by the following formula:

[0023]

[0024] P=[c h ,i,h s ,h e ,ws ,w e ,H,W];

[0025] Among them, h s and h e are the starting and ending coordinates of the target instance of the high-value category in the vertical direction, w s and w e are the starting and ending coordinates of the target instance of the high-value category in the horizontal direction, H and W are the height and width of the current image, respectively, and P represents the cth h The cropping parameters for the i-th target instance of a high-value category;

[0026] Finally, we get the cropping parameter set P of all target instances of all high-value categories in the current image. k , P k It is expressed by the formula:

[0027]

[0028] Among them, N k is the total number of all object instances of all high-value categories in the k-th image;

[0029] Finally, P of all images k Pad to a maximum length to ensure a fixed-size tensor in subsequent processing.

[0030] Optionally, the cross attention network is specifically composed of a shape encoder, a mask encoder, a cross attention unit, a decoder and an output layer. The cross attention network takes the semantic instance mask and the cropping parameters of the target instance of the high-value category as the query set, performs feature extraction through the shape encoder, and obtains the query shape feature F Q , a small number of semantic contours of the same category and their structural information are used as support sets, and feature extraction is performed through shape encoder and mask encoder respectively to obtain support shape features F K and support mask feature F V , query shape feature F through cross attention unit and decoder Q , support shape feature F K and support mask feature F V The output layer finally infers the structural material semantic information S.

[0031] Optionally, query the shape feature F Q Indicates the target feature that needs to be inferred, that is, the shape features that have never been seen in most of the same categories, supporting the shape feature F K and support mask feature F VRepresents the context information provided for reference, that is, a small amount of structural information of the same category. The cross attention unit is based on the cross attention mechanism and converts the query shape feature F Q , support shape feature F K and support mask feature F V Projected to query space, key space and value space through linear transformation, the process of linear transformation projection is expressed by formula:

[0032] Q=W Q F Q ;

[0033] K=W K F K ;

[0034] V=W V F V ;

[0035] Among them, W Q 、W K and W V All are learnable projection matrices, Q, K, and V are query vectors, key vectors, and value vectors generated by linear transformations, respectively;

[0036] Calculate the dot product of the query vector Q and the key vector K to get the similarity distribution and divide it by Scaling is performed, and finally the scaled similarity distribution is converted into the weight distribution α between the query feature and the support feature through the softmax function. The calculation formula of this process is as follows:

[0037]

[0038] Among them, d k is the dimension, softmax(·) represents the softmax function;

[0039] Finally, the weight distribution α is multiplied by the value vector V to obtain the final cross-attention output, which is further processed by the decoder to generate decoding features, and then the decoding features are mapped to inferred structural material semantic information S through the output layer.

[0040] Optionally, the guided generation model sequentially injects generated physical constraints, semantic masks, and random noise through a multi-layer guidance method, and finally generates an infrared image that is accurate in both structural details and physical properties, including: the guided generation model first maps the semantic mask to the initial feature space through a fully convolutional layer to generate an initial feature map, then a backbone network is formed by multiple generator residual blocks with spatially adaptive normalization modules, and each generator residual block is injected with physical constraints through the spatially adaptive normalization module; the generator residual block adopts a staged injection method to sequentially inject physical constraints and semantic masks, and the physical constraints first independently guide the generator residual block to synthesize an image that conforms to the physical constraints and has structural features, and then the semantic mask and random noise are jointly used as guidance information and injected into the generator residual block through a spatially adaptive modulation activation mechanism to achieve semantic texture embedding; the number of channels of the generator residual block decreases layer by layer, and each time it passes through a generator residual block, the feature map will be gradually amplified through the upsampling module, thereby forming a top-down layer-by-layer guided generation process; finally, the guided generation model generates an infrared image that is accurate in both structural details and physical properties.

[0041] Optionally, in the process of training the guided generative model through adversarial learning, the guided generative model is used as the generator of the adversarial generative network, and the real infrared image or the generated infrared image is input into the discriminator of the adversarial generative network. The discriminator extracts image features through the encoding and decoding structure, and finally maps them into N semantic category probabilities and true and false probabilities of each pixel value through convolution. c+1 The feature maps of each channel are obtained, and then the hinge loss, object-level CLIP style loss and class adversarial loss are calculated in sequence. Finally, the discriminator and generator are repeatedly trained and the weights of the adversarial generation network are updated until the preset convergence conditions are met.

[0042] This application proposes an infrared image adversarial generation method based on physical constraints and semantic shape inference. It designs a semantic-to-target instance conversion module. By re-encoding the semantic mask and combining it with connected component analysis, the rough semantic mask outline in the measured data is refined into specific instances of each category, so that the global semantic layout is further refined to instance-level targets, allowing the guided generation model to better focus on target generation. A physical constraint semantic shape inference module is designed that introduces a cross-attention mechanism to use the semantic instance mask and the cropping parameters of the target instance of the high-value category as the query set, and the semantic outlines and structural information of a small number of similar categories as the support set. The structural material semantic information is inferred based on the semantic shape, and then mapped to physical constraints based on the temperature and emissivity lookup table unit. Random noise, semantic mask and generated physical constraints are sequentially injected into the guided generation model based on disentanglement guidance and multi-task discrimination, so that the guided generation model generates infrared images with accurate structural details and physical properties. Compared with traditional infrared image generation technology, the method proposed in this application effectively solves the problem of difficulty in accurately characterizing the physical radiation characteristics of the generated target due to the lack of physical properties and material semantic information in the measured data. It can generate high-quality infrared images with structural details and physical laws to meet the actual needs of various applications.

[0043] In a second aspect, an embodiment of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute an infrared image adversarial generation method based on physical constraints and semantic shape inference as described in the first aspect above.

[0044] In a third aspect, an embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement an infrared image adversarial generation method based on physical constraints and semantic shape inference as described in the first aspect above.

[0045] It can be understood that the beneficial effects of the second to third aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related technologies, the following is a brief introduction to the drawings required for use in the embodiments of the present application or the description of the related technologies. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a flowchart of an infrared image adversarial generation method based on physical constraints and semantic shape inference, provided by an embodiment of the present application;

[0048] Figure 2 This is a modular schematic diagram of infrared image adversarial generation provided by an embodiment of the present application;

[0049] Figure 3 An embodiment of the present application provides a modular detail map generated by infrared image adversarial generation;

[0050] Figure 4 This is a schematic diagram of the results of a physical constraint semantic shape inference module provided by an embodiment of the present application;

[0051] Figure 5 This is a schematic diagram of a cross-attention mechanism provided by an embodiment of the present application;

[0052] Figure 6 This is an example of an embodiment of the present application. The results of the method proposed in this application are compared with those of other algorithms. Figure 1 ;

[0053] Figure 7 This is an example of an embodiment of the present application. The results of the method proposed in this application are compared with those of other algorithms. Figure 2 ;

[0054] Figure 8 It is a structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the various embodiments of the present application, many technical details are proposed to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can be implemented. The division of the following embodiments is only for the convenience of description and should not constitute any limitation on the specific implementation of the present application. The various embodiments can be combined with each other and referenced to each other under the premise of no contradiction.

[0056] An embodiment of the present application proposes an infrared image adversarial generation method based on physical constraints and semantic shape inference, which is suitable for high-quality generation of infrared images and is applied to electronic devices, wherein the electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is explained using the server as an example. The implementation details of the infrared image adversarial generation method based on physical constraints and semantic shape inference proposed in this embodiment are specifically described below. The following content is only the implementation details provided for easy understanding and is not necessary for the implementation of this solution.

[0057] The specific process of the infrared image adversarial generation method based on physical constraints and semantic shape inference proposed in this embodiment can be as follows: Figure 1 Shown, including:

[0058] In step 101, the semantic mask is input into the semantic to target instance conversion module to generate the corresponding semantic instance mask, filter out the target instances of the high-value category, and calculate the bounding box of the target instances of the high-value category, thereby obtaining the cropping parameters of the target instances of the high-value category.

[0059] In the specific implementation, the semantic mask cannot be used directly and needs to be processed by the semantic to target instance conversion module. The server inputs the semantic mask into the semantic to target instance conversion module, and the semantic to target instance conversion module generates the corresponding semantic instance mask, and then filters out the target instances of the high-value category and calculates the bounding box of the target instance of the high-value category, and finally obtains the cropping parameters of the target instance of the high-value category.

[0060] In one example, the adversarial generation flow of infrared images can be as follows Figure 2 The specific details can be as shown in Figure 3 As shown in the figure, the semantic mask is first input into the semantic to target instance conversion module, then processed by the physical constraint semantic shape inference module, and finally the infrared image is generated by the guided generation model.

[0061] In one example, the server selects infrared images and their semantic segmentation maps in the MSRS dataset as the measured dataset for training and testing, as input to the semantic to target instance conversion module. The MSRS dataset is the first RGB-Thermal urban scene image dataset with pixel-level annotations. The MSRS dataset contains 1,569 images, of which 820 were taken during the day and 749 were taken at night. The MSRS dataset marks eight common objects in car driving, namely cars, people, bicycles, curves, parking spaces, guardrails, color cones, and bumps. This embodiment defines cars, pedestrians, and bicycles in the measured dataset as target instances of high-value categories, and defines curves, parking spaces, guardrails, color cones, and bumps in the measured dataset as target instances of low-value categories.

[0062] In one example, the semantic segmentation map in the measured dataset is input into the semantic-to-target instance conversion module as a semantic mask M(x,y), M(x,y)∈{1,2,…,C}, where C represents the total number of categories and (x,y) represents the coordinates of the pixel point, which also serves as the instance number.

[0063] For each category in the semantic mask, the semantic to target instance conversion module generates a corresponding binary mask B c (x,y), B c The definition of (x,y) is as follows:

[0064]

[0065] Here, M(x, y)=c means that M(x, y) is of category c, and M(x, y)≠c means that M(x, y) is not of category c.

[0066] In getting B c (x,y), then B c (x,y) performs connected component analysis and obtains B c (x,y) corresponds to the instance region B of category c c,i ,i∈{1,2,…,N c}, i represents the i-th instance under category c, N c represents the total number of instances under category c.

[0067] Connected component analysis is expressed as:

[0068]

[0069] In the process of connected component analysis, each connected region R c,i Each instance mask I(x,y) is assigned a globally unique instance number g that increases continuously. By concatenating the instance mask I(x,y) with the semantic mask M(x,y), the corresponding semantic instance mask L(x,y) is generated, where L(x,y) = [M(x,y), I(x,y)].

[0070] After obtaining the semantic instance mask L(x,y), the semantic to target instance conversion module extracts the instance number and cropping parameters of the specified high-value category in each image.

[0071] Assume that the high value category is C h , then for each category c h ∈C h , extract the corresponding binary mask according to the semantic mask M(x,y) When the category of M(x,y) is ch At this time The value of is I(x,y), otherwise it is -1.

[0072] Subsequently, according to Each pixel instance number i in further extracts its binary mask Then pass To calculate the bounding box of the target instance of the high-value category, and thus obtain the cropping parameters of the target instance of the high-value category, the process is expressed by the following formula:

[0073]

[0074] P=[c h ,i,h s ,h e ,w s ,w e ,H,W];

[0075] Among them, h s and h e are the starting and ending coordinates of the target instance of the high-value category in the vertical direction, w s and w e are the starting and ending coordinates of the target instance of the high-value category in the horizontal direction, H and W are the height and width of the current image, respectively, and P represents the cth h The cropping parameters for the i-th object instance of the high-value category.

[0076] Finally, we get the cropping parameter set P of all target instances of all high-value categories in the current image. k , P k It is expressed by the formula:

[0077]

[0078] Among them, N k is the total number of all object instances of all high-value categories in the k-th image.

[0079] Finally, P of all images k Pad to a maximum length to ensure a fixed-size tensor in subsequent processing.

[0080] In step 102, the semantic instance mask and the cropping parameters of the target instance of the high-value category are input into the semantic shape inference module of the physical constraint composed of the cross-attention network and the temperature and emissivity lookup unit. Through the cross-attention network, the semantic instance mask and the cropping parameters of the target instance of the high-value category are used as the query set, and a small number of semantic contours and their structural information of the same category are used as the support set. Based on the query set and the support set, the structural material semantic information is inferred. Through the temperature and emissivity lookup unit, the structural material semantic information is mapped by lookup table to obtain the average emissivity matrix and temperature matrix, which are used as the generated physical constraints.

[0081] In its implementation, the ultimate goal of the physical constraint semantic shape inference module is to generate physical constraints. This module consists of a cross-attention network and a temperature and emissivity lookup table unit. The semantic instance mask and the cropping parameters of the target instance in the high-value category are fed into the cross-attention network. The cross-attention network then uses these as the query set, along with a small number of semantic contours and their structural information from the same category, as the support set. This module then infers the structural material semantic information based on the query and support sets. After obtaining the structural material semantic information, the temperature and emissivity lookup table unit performs a table lookup mapping on the structural material semantic information, resulting in an average emissivity matrix and a temperature matrix, which serve as the generated physical constraints.

[0082] In one example, the specific structure of the physical constraint semantic shape inference module is as follows: Figure 4 As shown in Figure 1, the physical constraint semantic shape inference module is specifically composed of a cross-attention network and a temperature and emissivity lookup table unit. The cross-attention network is specifically composed of a shape encoder, a mask encoder, a cross-attention unit, a decoder, and an output layer. The cross-attention network takes the semantic instance mask and the cropping parameters of the target instance of the high-value category as the query set, and extracts features through the shape encoder to obtain the query shape feature F Q , a small number of semantic contours of the same category and their structural information are used as support sets, and feature extraction is performed through shape encoder and mask encoder respectively to obtain support shape features F K and support mask feature F V , query shape feature F through cross attention unit and decoder Q , support shape feature F K and support mask feature F V The output layer finally infers the structural material semantic information S.

[0083] In one example, the principle of the cross attention mechanism is as follows Figure 5The cross-attention mechanism originates from the self-attention mechanism proposed in Transformer. It is an extension of the self-attention mechanism. Unlike the self-attention mechanism, which only calculates feature correlation within the same feature set, the cross-attention mechanism is used to capture the correlation between two different feature sets.

[0084] Query shape feature F Q Indicates the target feature that needs to be inferred, that is, the shape features that have never been seen in most of the same categories, supporting the shape feature F K and support mask feature F V Represents the context information provided for reference, that is, a small amount of structural information of the same category. The cross attention unit is based on the cross attention mechanism and converts the query shape feature F Q , support shape feature F K and support mask feature F V Projected to query space, key space and value space through linear transformation, the process of linear transformation projection is expressed by formula:

[0085] Q=W Q F Q ;

[0086] K=W K F K ;

[0087] V=W V F V ;

[0088] Among them, W Q 、W K and W V are all learnable projection matrices, Q, K, and V are the query vector, key vector, and value vector generated by linear transformation, respectively.

[0089] Then calculate the dot product of the query vector Q and the key vector K to get the similarity distribution and divide it by Scaling is performed, and finally the scaled similarity distribution is converted into the weight distribution α between the query feature and the support feature through the softmax function. The calculation formula of this process is as follows:

[0090]

[0091] Among them, d k is the dimension, and softmax(·) represents the softmax function.

[0092] Finally, the weight distribution α is multiplied by the value vector V to obtain the final cross-attention output, which is further processed by the decoder to generate decoding features, and then the decoding features are mapped to inferred structural material semantic information S through the output layer.

[0093] The cross attention mechanism processes F layer by layer Q and F K 、F V In the first layer, F Q As the query feature input to the cross attention mechanism, each subsequent layer’s output of the previous layer’s cross attention is used as the input of the next layer. K and F V It is reused as a supporting feature in each layer. Structural material semantic information S = {s1, s2, ..., s n}, where s i Represents the material semantic label of the i-th part in the target instance.

[0094] In getting s i Then, it is input into the temperature and emissivity lookup table module, which generates the corresponding average emissivity matrix according to the average emissivity and temperature of different material semantics. and temperature matrix Then combine it with s i are input into the guided generative model together.

[0095] In step 103, random noise, semantic masks, and generated physical constraints are input into a guided generative model pre-trained through adversarial learning. The guided generative model sequentially injects the generated physical constraints, semantic masks, and random noise through multi-layer guidance, ultimately generating an infrared image that is accurate in both structural details and physical properties.

[0096] In the specific implementation, the guided generative model is obtained through adversarial learning pre-training. Random noise, semantic masks and generated physical constraints are input into the guided generative model. The guided generative model injects the generated physical constraints, semantic masks and random noise in sequence through multi-layer guidance, and finally generates an infrared image that is accurate in both structural details and physical properties.

[0097] In one example, the specific structure of the guided generation model is as follows Figure 3 As shown in the figure, the guided generation model first maps the semantic mask into the initial feature space through a fully convolutional layer to generate an initial feature map. Next, the backbone network is composed of multiple generator residual blocks with spatially adaptive normalization modules. Each generator residual block is injected with physical constraints through the spatially adaptive normalization module.

[0098] The generator residual block adopts a staged injection method, injecting physical constraints and semantic masks in sequence. The physical constraints first independently guide the generator residual block to synthesize an image that conforms to the physical constraints and has structural features. Then the semantic mask and random noise are jointly used as guiding information and injected into the generator residual block through a spatially adaptive modulation activation mechanism to achieve the embedding of semantic texture.

[0099] The number of channels of the generator residual block decreases layer by layer. After each generator residual block, the feature map will be gradually enlarged through the upsampling module, thus forming a top-down layer-by-layer guided generation process.

[0100] Ultimately, the guided generative model produces infrared images that are accurate in both structural details and physical properties.

[0101] It should be noted that in the process of training the guided generative model through adversarial learning, the guided generative model is used as the generator of the adversarial generative network, and the real infrared image or the generated infrared image is input into the discriminator of the adversarial generative network. The discriminator extracts image features through the encoding and decoding structure, and finally maps them into N semantic category probabilities and true and false probabilities of each pixel value through convolution. c+1 The feature maps of each channel are obtained, and then the hinge loss, object-level CLIP style loss and class adversarial loss are calculated in sequence. Finally, the discriminator and generator are repeatedly trained and the weights of the adversarial generation network are updated until the preset convergence conditions are met.

[0102] This embodiment proposes an infrared image adversarial generation method based on physical constraints and semantic shape inference. A semantic-to-target instance conversion module is designed. By re-encoding the semantic mask and combining it with connected component analysis, the coarse semantic mask outlines in the measured data are refined into specific instances of each category. This further refines the global semantic layout to instance-level targets, allowing the guided generation model to better focus on target generation. A physically constrained semantic shape inference module is designed that incorporates a cross-attention mechanism. This module uses the semantic instance mask and the cropping parameters of target instances in high-value categories as the query set, and the semantic outlines and structural information of a small number of similar categories as the support set. This module infers structural material semantic information from the semantic shape, and then maps it to physical constraints based on temperature and emissivity lookup table units. Random noise, semantic masks, and generated physical constraints are sequentially injected into a guided generation model based on disentanglement guidance and multi-task discrimination, enabling the guided generation model to generate infrared images with accurate structural details and physical properties. Compared with traditional infrared image generation technology, the method proposed in this embodiment effectively solves the problem of difficulty in accurately characterizing the physical radiation characteristics of the generated target due to the lack of physical properties and material semantic information in the measured data. It can generate high-quality infrared images with structural details and physical laws to meet the actual needs of various applications.

[0103] The steps of the various methods described above are divided for clarity of description only. During implementation, they can be combined into a single step, or some steps can be split into multiple steps. As long as they contain the same logical relationships, they are all within the scope of protection of this application. Adding minor modifications or introducing minor designs to the algorithm or process, but not changing the core design of the algorithm or process, are also within the scope of protection of this application.

[0104] In one embodiment, in order to verify the effectiveness of the proposed infrared image adversarial generation method based on physical constraints and semantic shape inference (hereinafter referred to as OURS), we conducted comparative experiments on OURS with Pix2pix, Pix2pixHD, SPADE, OAISIS, SCGAN, CC-FPSE, SACGAN, etc., using PSNR, SSIM, LIPIS and FID as evaluation indicators. The experimental results of the comparative experiments are shown in Table 1. Figure 6 and Figure 7 shown.

[0105] Table 1: Comparative experimental results analysis

[0106] Generative Network PSNR SSIM LIPIS FID Pix2pix 15.06 0.4051 0.4143 98.85 Pix2pixHD 16.25 0.4152 0.4254 109.14 SPADE 16.61 0.4074 0.3946 130.48 OAISIS 16.78 0.4172 0.4055 139.13 SCGAN 16.69 0.3911 0.4555 202.99 CC-FPSE 16.34 0.4085 0.4107 129.99 SACGAN 15.98 0.4259 0.4075 95.64 OURS 17.18 0.4175 0.3970 89.10

[0107] From Table 1, Figure 6 and Figure 7 It can be seen that the infrared image adversarial generation method based on physical constraints and semantic shape inference proposed in this application has good performance.

[0108] Another embodiment of the present application provides an electronic device, the specific structure of which is as follows: Figure 8 As shown, it includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to execute an infrared image adversarial generation method based on physical constraints and semantic shape inference as described in the above method embodiment.

[0109] The memory and processor can be connected using a bus, which can include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and will not be described further in this article. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. Data processed by the processor is transmitted over a wireless medium via an antenna. Furthermore, the antenna also receives data and transmits it to the processor.

[0110] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0111] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement an infrared image adversarial generation method based on physical constraints and semantic shape inference as described in the above method embodiment.

[0112] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by instructing the relevant hardware through a program, and the program is stored in a storage medium and includes a number of instructions for causing a device (such as a single-chip microcomputer, chip) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk, and other media that can store program code.

[0113] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for adversarial generation of infrared images based on physical constraints and semantic shape inference, suitable for high-quality generation of infrared images, characterized by: The method comprises: The semantic mask is input into the semantic to object instance conversion module to generate the corresponding semantic instance mask, filter out the object instances of the high-value category, and calculate the bounding box of the object instances of the high-value category, thereby obtaining the cropping parameters of the object instances of the high-value category; The semantic instance mask and the cropping parameters of the target instance of the high-value category are input into the physical constraint semantic shape inference module composed of a cross-attention network and a temperature and emissivity lookup unit. Through the cross-attention network, the semantic instance mask and the cropping parameters of the target instance of the high-value category are used as the query set, and a small number of semantic contours and their structural information of the same category are used as the support set. The structural material semantic information is inferred based on the query set and the support set. The structural material semantic information is mapped to a table through the temperature and emissivity lookup unit to obtain the average emissivity matrix and temperature matrix, which are used as the generated physical constraints. Random noise, semantic masks and generated physical constraints are input into the guided generative model pre-trained by adversarial learning. The guided generative model injects the generated physical constraints, semantic masks and random noise in sequence through multi-layer guidance, and finally generates an infrared image that is accurate in both structural details and physical properties.

2. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to claim 1, characterized in that: The infrared images and semantic segmentation maps in the MSRS dataset are selected as the measured datasets for training and testing. Cars, pedestrians, and bicycles in the measured dataset are defined as target instances of high-value categories, and curves, parking spaces, guardrails, color cones, and bumps in the measured dataset are defined as target instances of low-value categories.

3. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to claim 2, characterized in that: The step of inputting the semantic mask into the semantic to target instance conversion module to generate the corresponding semantic instance mask includes: The semantic segmentation map in the measured dataset is input into the semantic to target instance conversion module as a semantic mask M(x,y), M(x,y)∈{1,2,…,C}, C represents the total number of categories, (x,y) represents the coordinates of the pixel point; For each category in the semantic mask, the semantic to target instance conversion module generates a corresponding binary mask B c (x,y), B c The definition of (x,y) is as follows: Among them, M(x,y)=c means that M(x,y) is of category c, and M(x,y)≠c means that M(x,y) is not of category c; Binarization mask B c (x,y) performs connected component analysis to obtain the binary mask B c (x,y) corresponds to the instance region B of category c c,i ,i∈{1,2,…,N c }, i represents the i-th instance under category c, N c represents the total number of instances under category c; Connected component analysis is expressed as: In the process of connected component analysis, each connected region R c,i Each instance mask I(x,y) is assigned a globally unique instance number g that increases continuously. By concatenating the instance mask I(x,y) with the semantic mask M(x,y), the corresponding semantic instance mask L(x,y) is generated, where L(x,y) = [M(x,y), I(x,y)].

4. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to claim 3, characterized in that: The step of screening out target instances of high-value categories and calculating bounding boxes of target instances of high-value categories, thereby obtaining cropping parameters of target instances of high-value categories, includes: Extract the instance number and cropping parameters of the specified high-value category in each image; Assume that the high value category is C h , then for each category c h ∈C h , extract the corresponding binary mask according to the semantic mask M(x,y) When the category of M(x,y) is c h At this time The value is I(x,y), otherwise it is -1; Then according to Each pixel instance number i in further extracts its binary mask Then pass To calculate the bounding box of the target instance of the high-value category, and thus obtain the cropping parameters of the target instance of the high-value category, the process is expressed by the following formula: P=[c h ,i,h s ,h e ,w s ,w e ,H,W]; Among them, h s and h e are the starting and ending coordinates of the target instance of the high-value category in the vertical direction, w s and w e are the starting and ending coordinates of the target instance of the high-value category in the horizontal direction, H and W are the height and width of the current image, respectively, and P represents the cth h The cropping parameters for the i-th target instance of a high-value category; Finally, we get the cropping parameter set P of all target instances of all high-value categories in the current image. k , P k It is expressed by the formula: Among them, N k is the total number of all object instances of all high-value categories in the k-th image; Finally, P of all images k Pad to a maximum length to ensure a fixed-size tensor in subsequent processing.

5. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to claim 4, characterized in that: The crisscross attention network is specifically composed of a shape encoder, a mask encoder, a crisscross attention unit, a decoder, and an output layer. The crisscross attention network takes the semantic instance mask and the cropping parameters of the target instance of the high-value category as the query set, and extracts features through the shape encoder to obtain the query shape feature F Q , a small number of semantic contours of the same category and their structural information are used as support sets, and feature extraction is performed through shape encoder and mask encoder respectively to obtain support shape features F K and support mask feature F V , query shape feature F through cross attention unit and decoder Q , support shape feature F K and support mask feature F V The output layer finally infers the structural material semantic information S.

6. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to claim 5, characterized in that: Query shape feature F Q Indicates the target feature that needs to be inferred, that is, the shape features that have never been seen in most of the same categories, supporting the shape feature F K and support mask feature F V Represents the context information provided for reference, that is, a small amount of structural information of the same category. The cross attention unit is based on the cross attention mechanism and converts the query shape feature F Q , support shape feature F K and support mask feature F V Projected to query space, key space and value space through linear transformation, the process of linear transformation projection is expressed by formula: Q=W Q F Q ; K=W K F K ; V=W V F V ; Among them, W Q 、W K and W V All are learnable projection matrices, Q, K, and V are query vectors, key vectors, and value vectors generated by linear transformations, respectively; Calculate the dot product of the query vector Q and the key vector K to get the similarity distribution and divide it by Scaling is performed, and finally the scaled similarity distribution is converted into the weight distribution α between the query feature and the support feature through the softmax function. The calculation formula of this process is as follows: Among them, d k is the dimension, softmax(·) represents the softmax function; Finally, the weight distribution α is multiplied by the value vector V to obtain the final cross-attention output, which is further processed by the decoder to generate decoding features, and then the decoding features are mapped to inferred structural material semantic information S through the output layer.

7. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to claim 6, characterized in that: The guided generative model uses multi-layer guidance to sequentially inject generated physical constraints, semantic masks, and random noise, ultimately generating infrared images that are accurate in both structural details and physical properties, including: The guided generative model first maps the semantic mask into the initial feature space through a fully convolutional layer to generate an initial feature map. Next, the backbone network is composed of multiple generator residual blocks with spatially adaptive normalization modules. Each generator residual block is injected with physical constraints through the spatially adaptive normalization module. The generator residual block adopts a staged injection method, sequentially injecting physical constraints and semantic masks. The physical constraints first independently guide the generator residual block to synthesize an image that conforms to the physical constraints and has structural features. Then, the semantic mask and random noise are jointly used as guiding information and injected into the generator residual block through a spatially adaptive modulation activation mechanism to achieve the embedding of semantic texture. The number of channels of the generator residual block decreases layer by layer. After each generator residual block, the feature map will be gradually enlarged through the upsampling module, thus forming a top-down layer-by-layer guided generation process; Ultimately, the guided generative model produces infrared images that are accurate in both structural details and physical properties.

8. The infrared image adversarial generation method based on physical constraints and semantic shape inference according to any one of claims 1 to 7, characterized in that: In the process of training the guided generative model through adversarial learning, the guided generative model is used as the generator of the adversarial generative network, and the real infrared image or the generated infrared image is input into the discriminator of the adversarial generative network. The discriminator extracts image features through the encoding and decoding structure, and finally maps them into N semantic category probabilities and true and false probabilities of each pixel value through convolution. c+1 The feature maps of each channel are obtained, and then the hinge loss, object-level CLIP style loss and class adversarial loss are calculated in sequence. Finally, the discriminator and generator are repeatedly trained and the weights of the adversarial generation network are updated until the preset convergence conditions are met.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an infrared image adversarial generation method based on physical constraints and semantic shape inference as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it can implement an infrared image adversarial generation method based on physical constraints and semantic shape inference as claimed in any one of claims 1 to 8.

Images