Infrared image and text oriented unified semantic representation learning model construction method

Abstract

This application relates to the fields of artificial intelligence and computer vision, and provides a method for constructing a unified semantic representation learning model for infrared images and text. This method extracts feature vectors from infrared images and text using an infrared image encoder and a text encoder, respectively. The extracted feature vectors are then mapped to a unified semantic embedding space. A contrastive learning loss function incorporating infrared semantic consistency constraints is used for model training, updating the parameters of the infrared image encoder and the text encoder to obtain the trained model. By introducing physical priors and semantic constraints, this method achieves a deep understanding of the inherent thermodynamic semantics of infrared images and precise cross-modal alignment. Furthermore, this method is highly efficient and does not rely on large-scale manually labeled data.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a method for constructing a unified semantic representation learning model for infrared images and text. Background Technology

[0002] Infrared imaging technology can capture the thermal radiation information of objects, offering unique advantages at night, in adverse weather conditions, or against complex backgrounds, and is widely used in fields such as security monitoring, remote sensing reconnaissance, and medical diagnosis. With the development of artificial intelligence technology, the demand for intelligent tasks based on infrared images is increasing, among which cross-modal interaction between infrared images and natural language text has become one of the core requirements.

[0003] In recent years, multimodal learning has made groundbreaking progress in the visible light domain by jointly understanding images and text. For example, the multimodal pre-trained model CLIP (Contrastive Language-Image Pre-training) has achieved a universal representation of vision and language. However, directly applying such techniques to infrared images faces significant challenges, namely the "infrared-text modal gap": on the one hand, infrared images focus on the distribution of thermal radiation intensity, lacking the detailed information such as texture and color found in visible light images, and generally suffer from problems such as blurred edges, low contrast, and noise interference; on the other hand, text descriptions are discrete sequences of semantic symbols, and their semantic expression depends on grammatical structure and contextual relationships. Therefore, multimodal learning based on infrared images and text still faces many technical difficulties.

[0004] In other words, traditional single-modal models (such as classification models for infrared images only, and semantic models for text only) cannot achieve cross-modal semantic association and alignment. Existing cross-modal models are mostly designed for visible light images and text, without fully considering the unique modal characteristics of infrared images, resulting in low semantic matching accuracy and poor robustness in infrared-text cross-modal tasks.

[0005] Therefore, there is an urgent need for a unified representation learning method that can specifically target the characteristics of infrared images and achieve deep integration with natural language semantics, in order to support higher-level cross-modal understanding and reasoning tasks. Summary of the Invention

[0006] In view of this, embodiments of this application provide a method for constructing a unified semantic representation learning model for infrared images and text, in order to solve the problem in the prior art that it is difficult to accurately match the features of infrared images and text in multimodal learning based on infrared images and text.

[0007] A first aspect of this application provides a method for constructing a unified semantic representation learning model for infrared images and text, including:

[0008] Infrared images and corresponding text descriptions are acquired to form training sample pairs;

[0009] The infrared image is input into the infrared image encoder to obtain the infrared image feature vector; the text description is input into the text encoder to obtain the text feature vector.

[0010] Map infrared image feature vectors and text feature vectors to a unified semantic embedding space;

[0011] Calculate the improved contrastive learning loss function; the improved contrastive learning loss function includes at least an infrared semantic consistency constraint term;

[0012] The parameters of the infrared image encoder and text encoder are updated based on the improved contrastive learning loss function until the model converges.

[0013] A second aspect of this application provides an apparatus for constructing a unified semantic representation learning model for infrared images and text, comprising:

[0014] The acquisition module is configured to acquire infrared images and corresponding text descriptions to form training sample pairs.

[0015] The encoding module is configured to input infrared images into an infrared image encoder to obtain infrared image feature vectors; and to input text descriptions into a text encoder to obtain text feature vectors.

[0016] The mapping module is configured to map infrared image feature vectors and text feature vectors to a unified semantic embedding space;

[0017] The computation module is configured to compute an improved contrastive learning loss function; the improved contrastive learning loss function includes at least an infrared semantic consistency constraint term.

[0018] The iterative module is configured to update the parameters of the infrared image encoder and the text encoder based on the improved contrastive learning loss function until the model converges.

[0019] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0020] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0021] The beneficial effects of this application's embodiments compared to existing technologies are as follows: This application's embodiments extract infrared image feature vectors and text feature vectors using an infrared image encoder and a text encoder, respectively, and map the extracted infrared image feature vectors and text feature vectors to a unified semantic embedding space. A contrastive learning loss function containing an infrared semantic consistency constraint term is used for model training, updating the parameters of the infrared image encoder and text encoder to obtain the trained model. This method, by introducing physical priors and semantic constraints, achieves a deep understanding of the inherent thermodynamic semantics of infrared images and accurate cross-modal alignment. Furthermore, this method does not rely on large-scale manually labeled data, achieving high efficiency. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating the method for constructing a unified semantic representation learning model for infrared images and text provided in this application embodiment.

[0024] Figure 2 This is a flowchart illustrating another method for constructing a unified semantic representation learning model for infrared images and text, provided in an embodiment of this application.

[0025] Figure 3 This is a flowchart of the channel hot weight attention mechanism provided in the embodiments of this application.

[0026] Figure 4 This is a schematic diagram of a unified semantic representation learning model construction device for infrared images and text provided in an embodiment of this application.

[0027] Figure 5 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0029] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for constructing a unified semantic representation learning model for infrared images and text according to embodiments of this application.

[0030] As mentioned above, traditional unimodal models cannot achieve cross-modal semantic association and alignment, while existing cross-modal models are mostly designed for visible light images and text, without fully considering the unique modal characteristics of infrared images. At the same time, due to the existence of the "infrared-text modal gap," it is difficult to directly apply technologies such as CLIP to multimodal learning based on infrared images and text.

[0031] In view of this, the embodiments of this application provide a method for constructing a unified semantic representation learning model for infrared images and text. By introducing physical priors and semantic constraints, it achieves a deep understanding of the inherent thermodynamic semantics of infrared images and accurate cross-modal alignment. Moreover, the implementation of this method does not rely on large-scale manually labeled data, thus achieving high efficiency.

[0032] Figure 1 This is a flowchart illustrating a method for constructing a unified semantic representation learning model for infrared images and text, as provided in an embodiment of this application. Figure 1 As shown, the method includes the following steps:

[0033] In step S101, infrared images and corresponding text descriptions are acquired to form training sample pairs.

[0034] In step S102, the infrared image is input into the infrared image encoder to obtain the infrared image feature vector; the text description is input into the text encoder to obtain the text feature vector.

[0035] In step S103, the infrared image feature vector and the text feature vector are mapped to a unified semantic embedding space.

[0036] In step S104, the improved contrastive learning loss function is calculated.

[0037] The improved contrastive learning loss function includes at least an infrared semantic consistency constraint term.

[0038] In step S105, the parameters of the infrared image encoder and the text encoder are updated according to the improved contrastive learning loss function until the model converges.

[0039] In some embodiments of this application, the method may be executed by a server or by a terminal device with certain processing capabilities.

[0040] In some embodiments of this application, infrared images and corresponding text descriptions can be acquired to form training sample pairs. For example, images and their corresponding bounding boxes and category labels can be acquired from infrared datasets (such as the publicly available dataset FLIR ADAS), and the labels can be converted into simple text descriptions using predefined templates.

[0041] For example, given an image labeled "person" and "car," the text "An infrared image containing a person and a car" can be generated. This allows us to obtain weakly supervised training sample pairs. ,in Represents an infrared image. This refers to the text description corresponding to the infrared image.

[0042] In some embodiments of this application, an infrared image can be input into an infrared image encoder to obtain an infrared image feature vector; at the same time, a text description can be input into a text encoder to obtain a text feature vector.

[0043] In other words, infrared images can be used. Input infrared image encoder Input the text description T into the text encoder. To extract infrared image features and text features respectively.

[0044] in, The standard BERT model can be used. An improved Vision Transformer can be used. In one example, the infrared image can be segmented into several small blocks and linearly embedded before being fed into a Vision Transformer. Feature extraction is performed using the Transformer layer. The Vision Transformer is a deep learning model that applies the Transformer architecture to the field of computer vision.

[0045] In some embodiments of this application, the infrared image feature vector and the text feature vector can be mapped to a unified semantic embedding space. That is, and The output feature vector can be passed through independent linear projection layers. and Mapped to the same The normalized vector is obtained from the semantic embedding space. and . For semantic embedding space dimension.

[0046] In some embodiments of this application, an improved contrastive learning loss function can be calculated, and the parameters of the infrared image encoder and text encoder can be updated according to the improved contrastive learning loss function until the model converges. The improved contrastive learning loss function includes at least an infrared semantic consistency constraint term.

[0047] According to the technical solution provided in this application, infrared image feature vectors and text feature vectors are extracted using an infrared image encoder and a text encoder, respectively. The extracted infrared image feature vectors and text feature vectors are then mapped to a unified semantic embedding space. A contrastive learning loss function containing an infrared semantic consistency constraint term is used for model training, updating the parameters of the infrared image encoder and the text encoder to obtain the trained model. This method, by introducing physical priors and semantic constraints, achieves a deep understanding of the inherent thermodynamic semantics of infrared images and accurate cross-modal alignment. Furthermore, this method does not rely on large-scale manually labeled data, thus achieving high efficiency.

[0048] As mentioned above, in some embodiments of this application, the infrared image encoder can be implemented based on the VisionTransformer architecture. Furthermore, to enable the model to learn the prior knowledge of "which thermal radiation regions of the image should be focused on" while learning the image-text alignment task, a channel thermal weight attention mechanism can be added to the multi-head self-attention module of the VisionTransformer architecture. By incorporating a learnable prior mask into the attention calculation, the infrared image encoder can adaptively focus on regions with significant thermal radiation. These regions can be areas where the thermal radiation exceeds a preset thermal radiation threshold.

[0049] The initial value of the learnable prior mask is generated based on the global thermal distribution of the infrared image and is used as the parameter update of the infrared image encoder during model training.

[0050] In the channel-based hot-weighted attention mechanism, the attention weight of each channel is calculated as follows: ;in, , and These are the query matrix, key matrix, and value matrix for this channel, respectively. Dimension factor The learnable coefficients, and For scalar parameters, For learnable prior masks, It is the transpose symbol. It is a normalized exponential function.

[0051] In other words, the infrared image encoder of this application embodiment improves upon the classic Vision Transformer architecture by introducing a channel hot weight attention mechanism at layer L. The improved Vision Transformer architecture operates as follows:

[0052] First, calculate the overall thermal distribution histogram of the infrared image and normalize it to obtain the prior thermal map. .

[0053] Then to Downsampling to match the size of the attention head yields a learnable prior mask. .

[0054] At the same time, during attention calculation, This is added as a bias term to the calculation of the attention weight matrix A, so that the model can pay extra attention to areas with abnormal or significant thermal radiation when establishing the association between image patches.

[0055] Unlike binary hard masks generated based on fixed thresholds in related technologies, the learnable prior mask provided in this application embodiment... A soft mask is typically a continuous-valued tensor (ranging from 0 to 1 or wider) that represents the degree of soft attention paid to different regions of an image. This type of soft mask can retain more information and is more suitable for integration with attention mechanisms.

[0056] The initial value can be determined based on the global thermal distribution of the infrared image. For example, the infrared image can be normalized, and then larger initial values ​​can be assigned to areas with higher thermal radiation intensity. However, this is only the initial state, not the final form.

[0057] The mask is a trainable parameter in the model, and its value is updated with the gradient of the loss function during backpropagation. The values ​​are continuously optimized and updated during model training based on the task objective (such as contrastive learning loss).

[0058] at the same time, and Having the same dimensions, and In attention calculations, it appears as an additive bias term, guiding attention scores to receive higher weights in regions of significant thermal radiation. Furthermore, the learnable coefficients... The influence of the mask is further controlled, and this coefficient also participates in training. This design allows the model to utilize prior knowledge of infrared thermal distribution while also adaptively adjusting the prior according to the requirements of downstream tasks, making it more flexible and expressive than a fixed threshold mask.

[0059] By adding a channel thermal weight attention mechanism to the multi-head self-attention module of an infrared image encoder, a learnable prior mask is integrated into the attention calculation, enabling the model to learn the optimal thermal radiation saliency prior for specific tasks. Unlike traditional methods of manually designing masks or using fixed thresholds, the learnable mask mechanism proposed in this application can be deeply integrated with the attention module, achieving end-to-end adaptive optimization while preserving the thermal distribution prior.

[0060] In some embodiments of this application, the initial values ​​of the learnable prior mask can be generated based on the global thermal distribution of a typical infrared image, and a universal prior mask can be generated after the model training is completed. The typical infrared image can be an infrared image of multiple preset objects under multiple preset scenes.

[0061] In some other embodiments of this application, the initial value of the learnable prior mask can be generated based on the global thermal distribution of the infrared images in the training sample pair, and a proprietary prior mask for the training sample pair can be generated after the model training is completed.

[0062] In other words, the learnable prior mask provided in the embodiments of this application It can be a sample-independent general mask or a sample-related special mask.

[0063] Sample-independent These are globally shared, learnable parameters, independent of the input samples. The model learns a general prior for the saliency of thermal radiation, applicable to all infrared images.

[0064] Sample correlation It can be dynamically generated from the infrared image itself through a small network (such as a convolutional layer). In this case, each sample has its own corresponding mask, and the network parameters for generating the mask are learnable.

[0065] Regardless of type The learnable properties all indicate that the parameter is optimized through training data rather than being manually designed or determined using fixed rules.

[0066] In some embodiments of this application, the infrared semantic consistency constraint term can be determined in the following manner:

[0067] For each pair of matched infrared image feature vectors in the same batch of training data and text feature vectors ,calculate With all text features in this batch Similarity distribution ,as well as Features of all images in this batch Similarity distribution ;

[0068] Determine separately relatively The first KL divergence, and relatively The second KL divergence is calculated, and the mean of the first and second KL divergences is calculated.

[0069] Confirm all of this batch and The sum of the divergence mean values ​​of the pairs is the infrared semantic consistency constraint term.

[0070] Furthermore, the improved contrastive learning loss function provided in this application embodiment also includes an InfoNCE loss function, used to narrow the distance between matching infrared image feature vector and text feature vector pairs in the embedding space, and to widen the distance between mismatched infrared image feature vector and text feature vector pairs in the embedding space. This improved contrastive learning loss function is a weighted sum of the InfoNCE loss function and the infrared semantic consistency constraint term; wherein the weighting coefficients apply to the infrared semantic consistency constraint term.

[0071] In other words, for a given set containing N matching pairs The improved comparative loss in that batch. The calculation is as follows:

[0072] First, calculate the standard InfoNCE loss. : Where sim(,) is the cosine similarity. For temperature coefficient, This represents the batch size during training.

[0073] Then calculate the consistency constraint loss. For each positive sample pair ,calculate Similarity vector with all vectors t in the batch ,as well as Similarity vectors with all v Constrain these two distributions to be as consistent as possible: ;in, This is the KL divergence (relative entropy). It is a normalized exponential function.

[0074] The final loss is: , This is a hyperparameter.

[0075] In some embodiments of this application, the gradient descent algorithm can be used to minimize ,renew , , , The parameters are used for iterative training until convergence.

[0076] Furthermore, after training is complete, the trained model can be used to query new infrared images or text to generate unified representation vectors with rich semantic information for subsequent applications.

[0077] Figure 2 This is a flowchart illustrating another method for constructing a unified semantic representation learning model for infrared images and text, provided in an embodiment of this application. Figure 2 As shown, you can first input an infrared dataset and process it. The processing includes generating text descriptions based on target labels and outputting weakly labeled image-text pairs.

[0078] Next, feature extraction is performed. On one hand, infrared image features can be extracted using an infrared image encoder. This infrared image encoder is implemented based on an improved Vision Transformer architecture, which introduces a channel-based hot weighted attention mechanism. Using this improved Vision Transformer architecture, infrared features can be extracted and then input into an embedding spatial projection.

[0079] On the other hand, a text encoder can be used for text feature extraction. The text encoder can adopt the standard BERT model architecture, and the extracted text features are also input into the aforementioned embedding space projection.

[0080] Through the phenomenon projection layer and The extracted infrared image features and text features are processed to obtain and output normalized embedding vectors. and This normalized embedding vector can be used as a basis for... and Loss calculation is performed, including calculating the standard InfoNCE loss and semantic consistency constraints. The final loss is obtained by weighted summation of the two types of losses, and the final loss is used for model optimization.

[0081] During model optimization, all trainable parameters are updated through backpropagation. After each update, it is determined whether the model has converged. If not, the updated encoder is used to re-extract features and iteratively execute the relevant steps of model training. If the model converges, the iteration ends and a pre-trained model is obtained.

[0082] Figure 3 This is a flowchart illustrating the channel hot weighting attention mechanism provided in an embodiment of this application. Figure 3As shown, the original infrared image and the output features of the (L-1)th layer of the Vision Transformer architecture can be used as inputs to perform the following operations in parallel: First, the global heat distribution is calculated using the original infrared image to generate a learnable prior mask with an initial state. Second, the output features of the (L-1)th layer are linearly projected to generate a query Q, key K, and value V matrix. Then, attention weights can be calculated based on the learnable prior mask and the query Q, key K, and value V matrix. Combined with V, the attention weight output is obtained, resulting in the output weighted features. Finally, the enhanced features of the Lth layer can be obtained through a feedforward network and residual connections.

[0083] The technical solution provided in this application uses target detection labels in publicly available infrared datasets to automatically generate corresponding text descriptions through predefined templates, constructing weak label training pairs of "infrared image-text", thus solving the problem of scarce paired data.

[0084] Simultaneously, a dual-tower model consisting of an infrared image encoder and a text encoder was constructed. The infrared image encoder is based on the Vision Transformer architecture, and a channel thermal weight attention mechanism is introduced into its multi-head self-attention module. This mechanism incorporates a learnable prior mask generated based on the global thermal distribution of the image into the attention calculation, enabling the model to adaptively focus on regions with significant thermal radiation.

[0085] Next, the feature vectors extracted by the dual-tower encoder are mapped to a unified semantic embedding space, and an improved contrastive learning loss function is computed for joint optimization. This loss function, based on the standard InfoNCE loss, adds an infrared semantic consistency constraint, which ensures that the embedding space maintains the thermodynamic semantic structure of the infrared image by forcing the matched image and text features to have a consistent similarity distribution with the negative sample group.

[0086] Finally, the model trained using the aforementioned loss function can encode input infrared images or text queries into a compact, unified semantic vector. This vector can directly support downstream tasks such as infrared image retrieval, zero-shot classification, and automatic report generation, significantly improving the level of intelligence.

[0087] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0088] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0089] Figure 4 This is a schematic diagram of a device for constructing a unified semantic representation learning model for infrared images and text, provided in an embodiment of this application. Figure 4As shown, the device includes:

[0090] The acquisition module 401 is configured to acquire infrared images and corresponding text descriptions to form training sample pairs.

[0091] The encoding module 402 is configured to input an infrared image into an infrared image encoder to obtain an infrared image feature vector; and to input a text description into a text encoder to obtain a text feature vector.

[0092] The mapping module 403 is configured to map infrared image feature vectors and text feature vectors to a unified semantic embedding space.

[0093] The computation module 404 is configured to compute an improved contrastive learning loss function; the improved contrastive learning loss function includes at least an infrared semantic consistency constraint term.

[0094] Iteration module 405 is configured to update the parameters of the infrared image encoder and the text encoder according to the improved contrastive learning loss function until the model converges.

[0095] According to the technical solution provided in this application, infrared image feature vectors and text feature vectors are extracted using an infrared image encoder and a text encoder, respectively. The extracted infrared image feature vectors and text feature vectors are then mapped to a unified semantic embedding space. A contrastive learning loss function containing an infrared semantic consistency constraint term is used for model training, updating the parameters of the infrared image encoder and the text encoder to obtain the trained model. This device, by introducing physical priors and semantic constraints, achieves a deep understanding of the inherent thermodynamic semantics of infrared images and precise cross-modal alignment. Furthermore, this method does not rely on large-scale manually labeled data, achieving high efficiency.

[0096] In some implementations, the infrared image encoder is based on the Vision Transformer architecture, and a channel thermal weight attention mechanism is added to the multi-head self-attention module of the Vision Transformer architecture. By incorporating a learnable prior mask into the attention calculation, the infrared image encoder adaptively focuses on the region with significant thermal radiation. The initial value of the learnable prior mask is generated based on the global thermal distribution of the infrared image and is updated as the parameters of the infrared image encoder during model training.

[0097] In some implementations, the attention weight of each channel in the channel hot weight attention mechanism is calculated as follows: ;in, , and These are the query matrix, key matrix, and value matrix for this channel, respectively. Dimension factor The learnable coefficient, For learnable prior masks, It is the transpose symbol. It is a normalized exponential function.

[0098] In some implementations, the initial values ​​of the learnable prior mask are generated based on the global thermal distribution of a typical infrared image, and a universal prior mask is generated after the model training is completed.

[0099] In some implementations, the initial values ​​of the learnable prior mask are generated based on the global thermal distribution of the infrared images in the training sample pairs, and a proprietary prior mask for the training sample pairs is generated after the model training is completed.

[0100] In some implementations, the infrared semantic consistency constraint is determined as follows: for each pair of matched infrared image feature vectors in the same batch of training data and text feature vectors ,calculate With all text features in this batch Similarity distribution ,as well as Features of all images in this batch Similarity distribution ; determine respectively relatively The first KL divergence, and relatively The second KL divergence is calculated, and the mean of the first and second KL divergences is calculated; all of this batch are determined. and The sum of the divergence mean values ​​of the pairs is the infrared semantic consistency constraint term.

[0101] In some implementations, the improved contrastive learning loss function further includes an InfoNCE loss function, used to narrow the distance between matching infrared image feature vector and text feature vector pairs in the embedding space and widen the distance between mismatched infrared image feature vector and text feature vector pairs in the embedding space; the improved contrastive learning loss function is a weighted sum of the InfoNCE loss function and the infrared semantic consistency constraint term; wherein the weighting coefficients are applied to the infrared semantic consistency constraint term.

[0102] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0103] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 5As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various method embodiments described above. Alternatively, when the processor 501 executes the computer program 503, it implements the functions of each module / unit in the various device embodiments described above.

[0104] Electronic device 5 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 5 may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or different components.

[0105] The processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0106] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM of the electronic device 5. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 5. The memory 502 can also include both internal and external storage units of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0109] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for constructing a unified semantic representation learning model for infrared images and text, characterized in that, include: Infrared images and corresponding text descriptions are acquired to form training sample pairs; The infrared image is input into an infrared image encoder to obtain an infrared image feature vector; the text description is input into a text encoder to obtain a text feature vector. Map the infrared image feature vector and the text feature vector to a unified semantic embedding space; Calculate the improved contrastive learning loss function; the improved contrastive learning loss function includes at least an infrared semantic consistency constraint term; The parameters of the infrared image encoder and the text encoder are updated according to the improved contrastive learning loss function until the model converges.

2. The method according to claim 1, characterized in that, The infrared image encoder is implemented based on the VisionTransformer architecture, and a channel thermal weight attention mechanism is added to the multi-head self-attention module of the VisionTransformer architecture. By incorporating a learnable prior mask into the attention calculation, the infrared image encoder can adaptively focus on the area with significant thermal radiation. The initial value of the learnable prior mask is generated based on the global thermal distribution of the infrared image and is updated as infrared image encoder parameters during model training.

3. The method according to claim 2, characterized in that, In the aforementioned channel-based hot-weighted attention mechanism, the attention weight of each channel is calculated as follows: ； in, , and These are the query matrix, key matrix, and value matrix for this channel, respectively. Dimension factor The learnable coefficient, For learnable prior masks, It is the transpose symbol. It is a normalized exponential function.

4. The method according to claim 2, characterized in that, The initial values ​​of the learnable prior mask are generated based on the global thermal distribution of a typical infrared image, and a universal prior mask is generated after the model training is completed.

5. The method according to claim 2, characterized in that, The initial value of the learnable prior mask is generated based on the global thermal distribution of the infrared images in the training sample pair, and a proprietary prior mask for the training sample pair is generated after the model training is completed.

6. The method according to claim 1, characterized in that, The infrared semantic consistency constraint term is determined in the following manner: For each pair of matched infrared image feature vectors in the same batch of training data and text feature vectors ,calculate With all text features in this batch Similarity distribution ,as well as Features of all images in this batch Similarity distribution ; Determine separately relatively The first KL divergence, and relatively The second KL divergence is calculated, and the mean of the first and second KL divergences is calculated. Confirm all of this batch and The sum of the divergence mean values ​​of the pairs is the infrared semantic consistency constraint term.

7. The method according to claim 1, characterized in that, The improved contrastive learning loss function also includes an InfoNCE loss function, used to bring in the distance between matching infrared image feature vector and text feature vector pairs in the embedding space, and to push away the distance between mismatched infrared image feature vector and text feature vector pairs in the embedding space. The improved contrastive learning loss function is a weighted sum of the InfoNCE loss function and the infrared semantic consistency constraint term; wherein the weighting coefficients are applied to the infrared semantic consistency constraint term.

8. A device for constructing a unified semantic representation learning model for infrared images and text, characterized in that, include: The acquisition module is configured to acquire infrared images and corresponding text descriptions to form training sample pairs. The encoding module is configured to input the infrared image into an infrared image encoder to obtain an infrared image feature vector; and to input the text description into a text encoder to obtain a text feature vector. The mapping module is configured to map the infrared image feature vector and the text feature vector to a unified semantic embedding space; The computation module is configured to compute an improved contrastive learning loss function; the improved contrastive learning loss function includes at least an infrared semantic consistency constraint term. An iterative module is configured to update the parameters of the infrared image encoder and the text encoder according to the improved contrastive learning loss function until the model converges.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images