A text-guided spectral image description method, system, device and storage medium
By employing spectral tokenization and joint representation, the problems of insufficient precision and lack of detail in image description are solved, thereby improving the accuracy and detail richness of image description and optimizing the modality fusion effect.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHEASTERN UNIV CHINA
- Filing Date
- 2026-05-20
- Publication Date
- 2026-07-24
Smart Images
Figure CN122265805B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a text-guided spectral image description method, system, device, and storage medium. Background Technology
[0002] Image captioning is a key research direction at the intersection of computer vision and natural language processing. It can automatically generate natural language descriptions that fit the content based on the input image. It has practical application needs in scenarios such as intelligent retrieval, human-computer interaction, and assistive vision. The expressive accuracy and detail richness of related technologies directly affect the system's performance.
[0003] Current image captioning methods mostly employ an encoder-decoder architecture, with the Transformer architecture as the core technology. This involves a visual encoding module extracting spatial domain features from the image, followed by a language decoding module generating descriptive text word by word based on these visual features. Some techniques introduce similar textual cues to aid semantic understanding, or adopt a progressive decoding approach to first generate an initial description and then optimize it. All these methods are based on traditional visual representation frameworks for modeling.
[0004] Existing technologies largely rely on spatial domain information for image modeling, with limited utilization of structural, edge, and texture information in the frequency domain, and relatively simple feature fusion methods. Furthermore, text guidance and spectral enhancement mechanisms are not deeply integrated, making it difficult for single-stage decoding to balance language fluency and detail accuracy. Image features, spectral features, and guiding text are also difficult to model collaboratively within a unified architecture. Therefore, existing technologies suffer from insufficient image description accuracy, lack of detail representation, and weak modal fusion effects. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention provides a text-guided spectral image description method, system, device, and storage medium. First, spectral tagging is used to achieve consistency between frequency domain information and the visual tagging space. Then, image-spectral joint representation is used to retrieve guiding text, and the guiding text participates in the relational attention calculation of image-spectral joint features. Finally, a refinement module performs gating selection based on the visual-spectral context and the candidate text context, thus forming a closed-loop synergistic enhancement of spectral, visual, and textual information. This approach avoids simply stacking spectral features, guiding text, and two-stage decoding side-by-side, thereby solving the technical problems of insufficient image description accuracy, lack of detail expression, and weak modal fusion effects in existing technologies.
[0006] The technical means adopted in this invention are as follows: Firstly, embodiments of this invention provide a text-guided spectral image description method, comprising:
[0007] Acquire the target image, preprocess the target image to obtain an image tensor; The image tensor is subjected to image visual encoding processing to obtain an image visual label sequence; Generate a spectral feature tensor based on the image tensor; Convert the spectral feature tensor into a spectral label sequence; A joint global representation is constructed based on the image visual label sequence and the spectral label sequence, and the guiding text is retrieved based on the joint global representation; The guiding text is processed by text encoding to obtain a text guiding vector; The image visual label sequence and the spectral label sequence are fused to obtain a fused vector. The fused vector and the text guidance vector are then subjected to text guidance spectral-visual relationship enhancement processing to obtain a joint representation after relationship enhancement. The enhanced joint representation of the relationship is subjected to initial description generation processing to obtain an initial image description; The guiding text is concatenated with the initial image description to form candidate text; The candidate text is encoded to obtain a candidate text representation. The enhanced joint representation and the candidate text representation are then combined for descriptive refinement to obtain an image description result.
[0008] Furthermore, the step of generating a spectral feature tensor based on the image tensor includes: processing the image tensor based on the four-step phase shift principle of single-pixel imaging to obtain a complex spectrum, and sequentially performing amplitude extraction, logarithmic compression, spectrum centering, center cropping, standardization, and channel construction processing on the complex spectrum to obtain a spectral tensor.
[0009] Furthermore, the formula for the complex spectrum is:
[0010] in, For the frequency domain coordinates obtained by calculation Complex spectral values at that location, The imaginary unit, These are the projected phase shifts. for , , and When a sinusoidal mask pattern is applied, the total light intensity value is obtained by a single-pixel detector. and The frequency variable in the frequency domain space is set.
[0011] Further, the step of converting the spectral feature tensor into a spectral label sequence includes: processing the spectral feature tensor through input shaping, low-frequency and high-frequency partitioning, low-frequency convolution, high-frequency convolution, label construction, and projection normalization to obtain a spectral label sequence with the same dimension as the image visual label sequence.
[0012] Furthermore, the formula for dividing low and high frequencies is as follows:
[0013] in, Given a one-dimensional sequence with a flattened input spectrum, Low frequency ratio, and These are low-frequency sequence and high-frequency sequence tensors, respectively. For batch size, Input the number of channels. and These are the corresponding sequence lengths. It is the set of real numbers.
[0014] Further, the process of fusing the image visual label sequence and the spectral label sequence to obtain a fused vector, and then performing text-guided spectral-visual relation enhancement processing on the fused vector and the text guidance vector to obtain a relation-enhanced joint representation, includes: The image visual label sequence and the spectral label sequence are aligned by label number, and the aligned image visual label sequence and the spectral label sequence are weighted and fused to obtain an initial joint feature; Based on the initial joint features and the text guidance vector, a query matrix, a key matrix, and a value matrix are constructed. Based on the query matrix and the key matrix, a relation attention weight matrix is constructed. Based on the relation attention weight matrix and the value matrix, an attention-enhanced relation feature representation is constructed. The attention-enhanced relation feature representation and the initial joint feature are used to obtain the relation-enhanced joint representation.
[0015] Further, the step of performing descriptive refinement processing on the joint representation enhanced by the relationship and the candidate text representation to obtain the image description result includes: Attention is calculated on the joint representation after relation enhancement based on the current refined decoding hidden state to obtain visual-spectral context features; Attention is calculated on the candidate text representation based on the current refined decoding hidden state to obtain the candidate text context features; The visual-spectral context features and the candidate text context features are fused through a gating mechanism to obtain refined context features; Based on the refined context features, a generation-copy hybrid distribution is used to obtain the output word probability distribution; The final image description result is obtained by correcting each word based on the output word probability distribution.
[0016] Secondly, embodiments of the present invention also provide a text-guided spectral image description system, comprising: The image input and preprocessing module is used to acquire the target image, preprocess the target image, and obtain an image tensor; An image visual encoding module is used to perform image visual encoding processing on the image tensor to obtain an image visual label sequence; A spectral feature construction module is used to generate a spectral feature tensor based on the image tensor. A spectral feature encoding module is used to convert the spectral feature tensor into a spectral label sequence; The guided text retrieval module is used to construct a joint global representation based on the image visual label sequence and the spectral label sequence, and retrieve guided text based on the joint global representation; The text encoding module is used to encode the guiding text to obtain a text guiding vector. The text-guided spectrum-visual relation encoding module is used to fuse the image visual label sequence and the spectrum label sequence to obtain a fused processing vector, and to perform text-guided spectrum-visual relation enhancement processing on the fused processing vector and the text-guided vector to obtain a joint representation after relation enhancement. An initial description generation module is used to perform initial description generation processing on the joint representation after relationship enhancement to obtain an initial image description; A candidate text construction module is used to concatenate the guiding text with the initial image description to form candidate text; The description refinement module is used to encode the candidate text to obtain a candidate text representation, and then perform description refinement processing on the joint representation after relationship enhancement and the candidate text representation to obtain an image description result.
[0017] Thirdly, embodiments of the present invention also provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the text-guided spectral image description method described in the first aspect.
[0018] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the text-guided spectral image description method described in the first aspect.
[0019] Compared with the prior art, the present invention has the following advantages: This invention simultaneously extracts the visual marker sequence and spectral marker sequence of the target image and constructs a joint global representation. It introduces guiding text to enhance the spectral-visual relationship under text guidance. Through step-by-step processing of initial generation and candidate text refinement, it can improve the ability of image description to capture detailed information to a certain extent, optimize the fusion effect of different modal features, help alleviate the problems of insufficient image description accuracy, lack of detail expression and weak modal fusion effect in the prior art, and improve the accuracy and detail richness of the final image description result.
[0020] Based on the above reasons, this invention can be widely applied in fields such as image processing. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0022] Figure 1 This is a flowchart of a text-guided spectral image description method according to the present invention; Figure 2 This is a flowchart illustrating the spectral feature construction process of a text-guided spectral image description method according to the present invention. Figure 3 This is a schematic diagram illustrating the alignment and fusion of image visual features and spectral features in a text-guided spectral image description method according to the present invention; Figure 4 This is a schematic diagram of the structure of a text-guided spectral image description system according to the present invention. Detailed Implementation
[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0024] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification and claims of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.
[0025] Image captioning is a key research direction at the intersection of computer vision and natural language processing. It can automatically generate natural language descriptions that fit the content based on the input image. It has practical application needs in scenarios such as intelligent retrieval, human-computer interaction, and assistive vision. The expressive accuracy and detail richness of related technologies directly affect the system's performance.
[0026] Current image captioning methods mostly employ an encoder-decoder architecture, with the Transformer architecture as the core technology. This involves a visual encoding module extracting spatial domain features from the image, followed by a language decoding module generating descriptive text word by word based on these visual features. Some techniques introduce similar textual cues to aid semantic understanding, or adopt a progressive decoding approach to first generate an initial description and then optimize it. All these methods are based on traditional visual representation frameworks for modeling.
[0027] First, existing technologies mostly rely on image spatial domain information to complete modeling, and do not make sufficient use of the overall structural information, edge information, texture information and different frequency band features contained in the image frequency domain. This leads to problems such as inaccurate description or missing details when the model processes images with complex textures, significant edges or rich local details. As a result, the utilization of structural, edge and texture information in the frequency domain is limited, and the feature fusion method is relatively simple.
[0028] Secondly, existing methods typically aim to achieve both logical fluency and accurate, rich detail in language expression during a single-stage decoding process. However, due to the semantic gap between the visual and linguistic modalities, it is difficult for models to simultaneously meet both requirements in a single generation process. Consequently, the generated results often suffer from fluent sentences but insufficient detail, or some detail coverage but an overall unnatural expression. This prevents deep integration of text guidance and spectral enhancement mechanisms, making it difficult for single-stage decoding to balance linguistic fluency and detail accuracy.
[0029] Furthermore, existing technologies employ methods that first generate a relatively logically and grammatically complete initial description, and then refine the initial description with additional textual information or candidate text. However, existing technologies struggle to achieve collaborative modeling within a unified framework of image features, spectral features, and guiding text.
[0030] Therefore, existing technologies suffer from technical problems such as insufficient image description accuracy, lack of detail expression, and weak modal fusion effect.
[0031] This invention provides a text-guided spectral image description method, system, device, and storage medium to solve the technical problems of insufficient image description accuracy, lack of detail expression, and weak modal fusion effect in the prior art.
[0032] The embodiments of the present invention will now be described with reference to the accompanying drawings.
[0033] Please see Figure 1 , Figure 1 This is a flowchart of a text-guided spectral image description method according to the present invention.
[0034] This application provides a text-guided spectral image description method, including the following steps: Step 101: Obtain the target image and preprocess it to obtain the image tensor.
[0035] The target image is preferably a three-channel color image. The target image undergoes resizing, normalization, and tensor quantization preprocessing to obtain a preprocessed image tensor. Preprocessing the target image unifies its input specifications, reduces the interference of image differences on subsequent visual feature extraction and spectral feature construction, and improves the adaptability and stability of the image tensor in the encoding process.
[0036] The formula for an image tensor is:
[0037] Where I is the preprocessed image input tensor, Let B be a set of real numbers, B be the batch size, 3 be the number of RGB color channels in the image, and H and W be the height and width dimensions of the image, respectively. Preferably, the values of H and W are 384, but they can also be 224, 256, 448, or other dimensions suitable for the visual encoder input.
[0038] Step 102: Perform image visual encoding on the image tensor to obtain the image visual label sequence.
[0039] Image visual encoding employs a Transformer-based visual encoder to extract semantic representations from the image's spatial domain. The visual encoder may include patch embedding layers, positional encoding layers, and multiple self-attention encoding layers. After visual encoding, the image yields a sequence of visual features. By performing image visual encoding on the image tensor, global semantic information and target positional relationships in the spatial domain can be mined from the image tensor. This allows the image visual label sequence to more comprehensively reflect the visual content of the input image, contributing to improved expressive effects in subsequent joint feature modeling.
[0040] The formula for the image visual label sequence is:
[0041] Among them, H v The extracted image visual label sequence, I represents the preprocessed image tensor, and the image visual coding module... This is a visual feature extraction network based on a transformer architecture, where B is the batch size and N is the number of elements. v Where D is the number of visual markers in the image, and D is the feature dimension. The preferred value for D is 768, but it can also be 512, 1024, or other dimensions.
[0042] The image visual coding module is preferably a transformer-based visual encoder, including image patch embedding, positional coding, and a multi-layer self-attention coding layer, used to extract global semantic and spatial relationship features of the image. This encoder can effectively capture the overall content information of the input image and the positional association between the target, making the extracted image visual label sequence more consistent with the actual semantics of the image, which helps to improve the accuracy of subsequent feature fusion and description generation.
[0043] Please see Figure 2 , Figure 2 This is a flowchart illustrating the spectral feature construction process of a text-guided spectral image description method according to the present invention.
[0044] Step 103: Generate a spectral feature tensor from the image tensor. This step extracts frequency domain structure, edge, and texture information from the image tensor, enabling the spectral feature tensor to better reflect the frequency domain distribution characteristics of the image and helping to enrich the detailed expression of subsequent joint features.
[0045] Specifically, by spatially modulating and probing the image tensor, a spectral feature tensor is constructed based on the four-step phase shift principle of single-pixel imaging.
[0046] In some embodiments, generating a spectral feature tensor from an image tensor includes: processing the image tensor based on the four-step phase-shift principle of single-pixel imaging to obtain a complex spectrum; and sequentially performing amplitude extraction, logarithmic compression, spectrum centering, center cropping, standardization, and channel construction on the complex spectrum to obtain a spectral tensor. This allows for the extraction of high- and low-frequency differences, edge and texture features in the frequency domain from the image tensor, while simultaneously compressing the dynamic range of the spectrum and unifying the feature scale, making the spectral tensor more suitable for subsequent spectral feature encoding steps.
[0047] In some embodiments, the formula for the complex spectrum is:
[0048] in, For the frequency domain coordinates obtained by calculation Complex spectral values at that location, The imaginary unit, These are the projected phase shifts. for , , and When a sinusoidal mask pattern is applied, the total light intensity value is obtained by a single-pixel detector. and The frequency variable in the frequency domain space is set.
[0049] The formula for the single-step light intensity response value in the above formula is:
[0050] Among them, I k (u,v) represents the single-step light intensity response value, I(x,y) represents the physical image information (such as reflectivity or transmittance) of the target in the spatial domain at coordinates (x,y), u and v are the set frequency variables in the frequency domain, H and W are the height and width dimensions of the detection field of view (or reconstructed image), respectively, A is the DC background bias of the mask pattern to ensure that the light intensity is non-negative, and B is the modulation amplitude (contrast) of the mask pattern. k This represents the projected phase shift.
[0051] The formula for extracting the amplitude of a complex spectrum is:
[0052] Where M(u,v) is the calculated spectral amplitude, F(u,v) is the complex spectrum, and |·| is the operation of taking the complex modulus.
[0053] The formula for logarithmic compression is:
[0054] Where S(u,v) is the logarithmically compressed spectral feature value, M(u,v) is the spectral amplitude, and log represents the logarithmic operation.
[0055] After amplitude extraction and logarithmic compression of the complex spectrum, further spectrum centering, center clipping, and standardization are performed. The optimal spectrum size is 32×32 or 64×64.
[0056] The formula for the spectral feature tensor is:
[0057] Where S is the final output spectral feature tensor, B is the batch size, and C is the value of S. s For the number of spectrum channels (e.g., single channel, three channels, or high / medium / low frequency channels), C s Preferably 3, H s and W s These represent the height and width of the spectral feature, respectively.
[0058] Step 104: Convert the spectral feature tensor into a spectral label sequence. This transforms the two-dimensional spectral feature map into a sequence format that adapts to the transformer architecture, effectively preserving the global structure and local details in the image's frequency domain. This ensures that the feature dimensions of the spectral label sequence match those of the image's visual label sequence, facilitating subsequent joint modeling.
[0059] The formula for the spectral label sequence is:
[0060] Among them, H ss Let B be the spectral label sequence tensor, B be the batch size, and N be the number of spectral labels. s Where is the number of spectral labels, and D is the dimension of the hidden layer aligned with the visual features.
[0061] In some embodiments, converting a spectral feature tensor into a spectral label sequence includes: processing the spectral feature tensor through input shaping, low-frequency and high-frequency partitioning, low-frequency convolution, high-frequency convolution, label construction, and projection normalization to obtain a spectral label sequence with the same dimension as the image visual label sequence.
[0062] Specifically, the two-dimensional spectrogram is flattened into a one-dimensional spectrogram sequence through an input shaping operation, converting the spectral features of the two-dimensional structure into a sequence form to meet the needs of subsequent channel-by-channel processing. The spectrogram sequence is divided into low-frequency and high-frequency parts according to a preset ratio, effectively separating global information and local detail information in the spectrum, facilitating the subsequent targeted extraction of different types of frequency domain features. The global structural information in the spectrum is extracted through low-frequency convolution branches, which better captures the overall distribution and contour features of the image spectrum, enhancing the ability of the spectrogram marker sequence to represent the global content of the image. The texture, edge, and local change information in the spectrum is extracted through high-frequency convolution branches, effectively capturing the detailed changes in the image spectrum, enabling the spectrogram marker sequence to reflect the local texture and edge details of the image. The convolutional features are compressed into a fixed number of spectrogram markers through a marker construction unit, achieving structured compression of frequency domain features and obtaining fixed-length spectrogram markers. The spectrogram markers are projected to the same dimension as the image visual markers through a projection normalization unit, ensuring that the spectrogram markers and image visual markers are consistent in feature dimension, improving the compatibility and stability of subsequent multimodal feature fusion, and finally obtaining a spectrogram marker sequence with the same dimension as the image visual marker sequence.
[0063] In some embodiments, the formula for dividing low frequency into high frequency is:
[0064] in, Given a one-dimensional sequence with a flattened input spectrum, Low frequency ratio, and These are low-frequency sequence and high-frequency sequence tensors, respectively. For batch size, Input the number of channels. and These are the corresponding sequence lengths. It is the set of real numbers. The value range is 0.5-0.8, preferably 0.67.
[0065] Low-frequency sequences mainly represent the overall structure and subject contour information of the image, while high-frequency sequences mainly represent the image edges, textures, and local details. The low-frequency sequences are input into the low-frequency convolution branch, and features are extracted using one-dimensional convolution with a larger receptive field. The high-frequency sequences are input into the high-frequency convolution branch, and features are extracted using one-dimensional convolution with a smaller receptive field. The two convolution features are respectively processed by piecewise average pooling to construct spectral labels, and then projected onto a unified dimension D by linear projection or 1×1 convolution to obtain the spectral label sequence.
[0066] Step 105: Construct a joint global representation based on the image visual marker sequence and the spectral marker sequence, and retrieve the guiding text based on the joint global representation. This method can fuse the spatial semantic information carried by the image visual marker sequence and the frequency domain detail information carried by the spectral marker sequence to construct a more robust multimodal joint global representation that can more comprehensively reflect the content features of the input image.
[0067] The formula for calculating the joint global representation is:
[0068] Where z is the joint global representation, β is the spectral feature weight, preferably 0.1-1, and H v For the extracted image visual label sequence, H ss For spectral label sequence tensors, This is for global pooling.
[0069] The formula for the guiding text is:
[0070] Among them, T g To obtain the optimal guiding text, T i Let z be the i-th sentence in the candidate text library, which consists of manually described texts or reviewed historical descriptions corresponding to each image in the training set. During training or inference, each text in the candidate text library is pre-encoded and stored as a text vector. For the current input image, a joint global representation is first constructed based on the image's visual and spectral features, and then cosine similarity is calculated with the candidate text vectors. The k texts with the highest scores are selected as the guide text set. If k=1, the optimal text is used directly; if k>1, average pooling, attention weighting, or concatenation projection are applied to the encoding results of multiple texts to obtain a unified text guide vector. C is the overall candidate text set, z is the joint global feature representation, and E is the global text vector. For text feature encoding functions, similarity Here is the evaluation function used to measure vector distance (such as cosine similarity), and argmax is the corresponding text that maximizes the similarity.
[0071] Step 106: Encode the guiding text to obtain a text guiding vector. The text guiding vector can be used in the joint modeling process of image visual features and spectral features, providing semantic guidance information for enhancing the relationship between image-spectral joint features, and helping to improve the model's ability to capture semantic association information of images.
[0072] The formula for the text-guided vector is:
[0073] Where G is the guiding vector, Tg The text encoding module is used to retrieve the introductory text. This is a text feature extraction network.
[0074] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the alignment and fusion of image visual features and spectral features in a text-guided spectral image description method according to the present invention.
[0075] Step 107: The image visual label sequence and spectral label sequence are fused to obtain a fused vector. The fused vector and the text guidance vector are then subjected to text guidance spectral-visual relationship enhancement processing to obtain a joint representation with enhanced relationship. This joint representation can be applied to the initial description generation process, which can strengthen the correlation between image visual features, spectral features and text guidance information, and help improve the semantic accuracy and detail richness of the initial description generation.
[0076] In some embodiments, the image visual label sequence and the spectral label sequence are fused to obtain a fused vector. The fused vector and the text guidance vector are then subjected to text guidance spectral-visual relation enhancement processing to obtain a joint representation with enhanced relation, including: The image visual label sequence and the spectral label sequence are aligned by label number. The aligned image visual label sequence and the spectral label sequence are then weighted and fused to obtain the initial joint features. A query matrix, a key matrix, and a value matrix are constructed based on the initial joint features and the text guidance vector. A relation attention weight matrix is constructed based on the query matrix and the key matrix. An attention-enhanced relation feature representation is constructed based on the relation attention weight matrix and the value matrix. The attention-enhanced relation feature representation and the initial joint features are then used to obtain the relation-enhanced joint representation.
[0077] The formula for the aligned spectral characteristics is:
[0078] in, For the aligned spectral features, For the set of real numbers, Where D is the batch size, D is the hidden layer dimension aligned with the visual features, and N is the number of elements in the batch. v The number of visual tags in the image.
[0079] Specifically, the rule for aligning the number of spectral labels between the spectral label sequence and the image visual label sequence is as follows: if the number of spectral labels N... s Number of visual markers N in the image v If they are equal, the spectral label sequence is used directly; if the number of spectral labels is N... s Less than the number of visual markers N in the image vThe spectral marker sequence is repeatedly expanded before the pruning operation is performed; if the number of spectral markers N s Greater than the number of visual markers N in the image v It directly performs a pruning operation on the spectral marker sequence.
[0080] The formula for the initial joint features is:
[0081] in, For the initial joint features, H v The extracted image visual label sequence, The optimal fusion weight is 0.3-0.9. The aligned spectral characteristics.
[0082] The formulas for querying the matrix, key matrix, and value matrix are:
[0083] in, These are the query matrix, key matrix, and value matrix in the attention mechanism, respectively. For initial joint features, This is a global text vector (guiding clue). All of these are linear projection weight matrices that can be learned in the network.
[0084] The formula for the relational attention weight matrix is:
[0085] in, This is the relational attention weight matrix. and These are the query matrix and the key matrix, respectively. This is the transpose of the key matrix. The feature dimension (used for scaling to prevent gradient vanishing). This is the normalized exponential activation function.
[0086] The formula for the text union representation after relation enhancement is:
[0087] in, For the text joint representation after relation enhancement, This is the relational attention weight matrix. It is a value matrix.
[0088] The formula for the joint representation after relation enhancement is:
[0089] in, For the joint representation after the relationship is enhanced, For initial joint features, This represents the joint representation of text after relation enhancement.
[0090] In some implementations, the joint representation after relation enhancement undergoes LayerNorm normalization to improve training stability.
[0091] Step 108: Perform initial description generation processing on the relation-enhanced joint representation to obtain the initial image description. Specifically, the relation-enhanced joint representation... The input is an initial description generator, preferably an autoregressive text decoder based on Transformer. At each time step, the generator predicts the next word based on the historical word sequence and joint features to output an initial image description. This initial image description serves as the base text for subsequent refinement processing, preliminarily presenting the core content and overall semantics of the image. This helps reduce the optimization difficulty of subsequent description refinement stages and improves the fluency of the final description.
[0092] The formula for describing the initial image is:
[0093] in, This is the initial description sequence generated in the first stage of the network. For at any time Predict the specific words in the output. The total length of the generated word sequence.
[0094] Step 109: Concatenate the guiding text with the initial image description to form candidate text. This candidate text can be used in subsequent description refinement processes, providing semantic references for the detailed correction of the initial image description, thereby improving the detail completeness and semantic richness of the final image description result.
[0095] The formula for candidate text is:
[0096] in, The candidate text formed by splicing, To guide the text, For initial image description; This indicates a sequence concatenation operation.
[0097] Step 1010: The candidate text is encoded to obtain a candidate text representation. The joint representation with enhanced relations is then combined with the candidate text representation for descriptive refinement to obtain the image description result. This image description result can comprehensively reflect the spatial domain information, frequency domain details, and textual guidance semantics of the target image, thereby improving the accuracy, detail richness, and fluency of the image content description.
[0098] The formula for calculating the candidate text representation is:
[0099] in, This represents the candidate text. The candidate text is formed by splicing.
[0100] In some embodiments, the joint representation enhanced by relation enhancement and the candidate text representation are subjected to descriptive refinement processing to obtain the image description result, including: performing attention calculation on the joint representation enhanced by relation enhancement based on the current refined decoding hidden state to obtain visual-spectral context features; performing attention calculation on the candidate text representation based on the current refined decoding hidden state to obtain candidate text context features; fusing the visual-spectral context features and the candidate text context features through a gating mechanism to obtain refined context features; using a generator-copy hybrid distribution based on the refined context features to obtain the output word probability distribution; and refining word by word based on the output word probability distribution to obtain the final image description result.
[0101] Visual-spectral context features can be applied to the context modeling stage of description refinement, providing dual visual and frequency domain feature support for the detailed restoration of image content; candidate text context features can be applied to the semantic reference stage of description refinement, providing textual semantic clues for the correction of the initial description; refinement context features can be applied to the word generation decision stage, taking into account both visual spectral information and textual semantic information; this refinement process can be applied to the optimization process of the initial image description, enabling detailed error correction and expression improvement of the description content, which helps to improve the accuracy, fluency and detail completeness of the final image description.
[0102] The formula for calculating visual-spectral context features is:
[0103] in, For visual-spectral context features, This is the current hidden state of the fine-tuned decoding. This represents the joint representation after the relationship has been enhanced.
[0104] The formula for calculating the contextual features of candidate text is:
[0105] in, For candidate text context features, This is the current hidden state of the fine-tuned decoding. This represents the candidate text.
[0106] The formula for calculating refined contextual features is:
[0107] in, To refine contextual features, To integrate weights, Pay attention to the results of candidate texts. For visual-spectral context features, This is an element-wise multiplication. The purpose of this gating structure is to increase the proportion of textual context when there is high-value information in the candidate text, and to increase the proportion of visual-spectral context when the candidate text is noisy, thereby reducing misleading information.
[0108] The formula for calculating the probability distribution of output words is:
[0109]
[0110] in, For the output word probability distribution, The probability of generating a new word from the vocabulary. To select candidate texts The probability of copying an existing word, where σ is the Sigmoid function. To represent the learnable weight matrix of the generation probability gating, This is the current hidden state of the fine-tuned decoding. This is the refined context vector at the current moment. To generate learnable bias terms for probabilistic gating. The generation probability is the probability of generating a new word from the vocabulary at the current moment.
[0111] The final image description result is calculated using the following formula:
[0112] in, For the final image description result, For a moment The output word units are selected based on the final word probability distribution P(w). This represents the total length of the generated sequence.
[0113] Please see Figure 4 , Figure 4This is a schematic diagram of the structure of a text-guided spectral image description system according to the present invention.
[0114] This invention also provides a text-guided spectral image description system, comprising: an image input and preprocessing module for acquiring a target image and preprocessing the target image to obtain an image tensor; an image visual encoding module for performing image visual encoding on the image tensor to obtain an image visual label sequence; a spectral feature construction module for generating a spectral feature tensor based on the image tensor; a spectral feature encoding module for converting the spectral feature tensor into a spectral label sequence; a guided text retrieval module for constructing a joint global representation based on the image visual label sequence and the spectral label sequence, and retrieving guided text based on the joint global representation; and a text encoding module for performing text encoding on the guided text to obtain text guidance. The system comprises the following modules: a vector module; a text-guided spectrum-visual relation encoding module, which fuses the image visual label sequence and the spectrum label sequence to obtain a fused vector; a text-guided spectrum-visual relation enhancement module, which enhances the fused vector and the text-guided vector to obtain a relation-enhanced joint representation; an initial description generation module, which generates an initial image description by performing initial description generation on the relation-enhanced joint representation; a candidate text construction module, which concatenates the guiding text and the initial image description to form candidate text; and a description refinement module, which encodes the candidate text to obtain a candidate text representation; and performs description refinement on the relation-enhanced joint representation and the candidate text representation to obtain the image description result.
[0115] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described text-guided spectral image description method.
[0116] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described text-guided spectral image description method.
[0117] The same or similar parts among the various embodiments in this specification can be referred to mutually, and will not be repeated here.
[0118] This invention introduces a spectral enhancement mechanism into an image description network based on a transformer architecture. This allows the model to extract spectral features from the target image while simultaneously extracting spatial domain visual features from the original image. These spectral features are then constructed as a spectral label sequence and aligned and fused with the image visual label sequence. Compared to existing image description methods that rely solely on RGB image space representation, this invention utilizes frequency domain information to characterize the global structure, edge variations, and detailed textures of the image, thereby enhancing the model's ability to understand fine-grained visual content. Through a spectral construction method based on a four-step phase-shift principle, spectral information reflecting the structural features of the image can be obtained from the input image. Selective modeling of different frequency components enhances key frequency domain information while suppressing interference from redundant noise in feature representation.
[0119] This invention constructs a joint global representation of images and spectrograms based on the obtained visual and spectral label sequences, and introduces guiding text related to the target image, enabling the guiding text to participate in the relationship enhancement process of the joint spectrogram-visual features. Through the text-guided spectrogram-visual relationship encoding module, the model can establish a more comprehensive association between visual, spectral, and textual semantic information, thereby improving the fusion effect between different modal features. Employing a step-by-step processing approach of initial generation and description refinement, a relatively complete initial description with a semantic backbone is first generated, and then candidate text and relationship enhancement features are combined to correct and supplement the detailed information in the description. Therefore, it can alleviate, to some extent, the problems of insufficient image description accuracy, lack of detail expression, insufficient characterization of object relationships, and weak modal fusion effect in existing technologies, ultimately improving the accuracy, detail richness, and semantic consistency of image description results.
[0120] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0121] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A text-guided spectral image description method, characterized in that, include: Acquire the target image, preprocess the target image to obtain an image tensor; The image tensor is subjected to image visual encoding processing to obtain an image visual label sequence; Generate a spectral feature tensor based on the image tensor; Convert the spectral feature tensor into a spectral label sequence; A joint global representation is constructed based on the image visual label sequence and the spectral label sequence, and the guiding text is retrieved based on the joint global representation; The guiding text is processed by text encoding to obtain a text guiding vector; The image visual label sequence and the spectral label sequence are aligned by label number, and the aligned image visual label sequence and the spectral label sequence are weighted and fused to obtain an initial joint feature; Based on the initial joint features and the text guidance vector, a query matrix, a key matrix, and a value matrix are constructed. Based on the query matrix and the key matrix, a relation attention weight matrix is constructed. Based on the relation attention weight matrix and the value matrix, an attention-enhanced relation feature representation is constructed. Based on the attention-enhanced relation feature representation and the initial joint feature, a relation-enhanced joint representation is obtained; The enhanced joint representation of the relationship is subjected to initial description generation processing to obtain an initial image description; The guiding text is concatenated with the initial image description to form candidate text; The candidate text is encoded to obtain a candidate text representation. The enhanced joint representation and the candidate text representation are then combined for descriptive refinement to obtain an image description result.
2. The text-guided spectral image description method according to claim 1, characterized in that, The step of generating a spectral feature tensor based on the image tensor includes: processing the image tensor based on the four-step phase shift principle of single-pixel imaging to obtain a complex spectrum, and sequentially performing amplitude extraction, logarithmic compression, spectrum centering, center cropping, standardization, and channel construction processing on the complex spectrum to obtain a spectral tensor.
3. The text-guided spectral image description method according to claim 2, characterized in that, The formula for the complex spectrum is: in, For the frequency domain coordinates obtained by calculation Complex spectral values at that location, The imaginary unit, These are the projected phase shifts. for , , and When a sinusoidal mask pattern is applied, the total light intensity value is obtained by a single-pixel detector. and The frequency variable in the frequency domain space is set.
4. The text-guided spectral image description method according to claim 1, characterized in that, The step of converting the spectral feature tensor into a spectral label sequence includes: processing the spectral feature tensor through input shaping, low-frequency and high-frequency partitioning, low-frequency convolution, high-frequency convolution, label construction, and projection normalization to obtain a spectral label sequence with the same dimension as the image visual label sequence.
5. The text-guided spectral image description method according to claim 4, characterized in that, The formula for dividing low and high frequencies is as follows: in, Given a one-dimensional sequence with a flattened input spectrum, Low frequency ratio, and These are low-frequency sequence and high-frequency sequence tensors, respectively. For batch size, Input the number of channels. and These are the corresponding sequence lengths. It is the set of real numbers.
6. The text-guided spectral image description method according to claim 1, characterized in that, The step of performing descriptive refinement processing on the joint representation enhanced by the relationship and the candidate text representation to obtain the image description result includes: Attention is calculated on the joint representation after relation enhancement based on the current refined decoding hidden state to obtain visual-spectral context features; Attention is calculated on the candidate text representation based on the current refined decoding hidden state to obtain the candidate text context features; The visual-spectral context features and the candidate text context features are fused through a gating mechanism to obtain refined context features; Based on the refined context features, a generation-copy hybrid distribution is used to obtain the output word probability distribution; The final image description result is obtained by correcting each word based on the output word probability distribution.
7. A text-guided spectral image description system, characterized in that, include: The image input and preprocessing module is used to acquire the target image, preprocess the target image, and obtain an image tensor; An image visual encoding module is used to perform image visual encoding processing on the image tensor to obtain an image visual label sequence; A spectral feature construction module is used to generate a spectral feature tensor based on the image tensor. A spectral feature encoding module is used to convert the spectral feature tensor into a spectral label sequence; The guided text retrieval module is used to construct a joint global representation based on the image visual label sequence and the spectral label sequence, and retrieve guided text based on the joint global representation; The text encoding module is used to encode the guiding text to obtain a text guiding vector. The text-guided spectrum-visual relation encoding module is used to align the image visual label sequence with the spectrum label sequence by label number, and then weight-fuse the aligned image visual label sequence with the spectrum label sequence to obtain initial joint features; Based on the initial joint features and the text guidance vector, a query matrix, a key matrix, and a value matrix are constructed. Based on the query matrix and the key matrix, a relation attention weight matrix is constructed. Based on the relation attention weight matrix and the value matrix, an attention-enhanced relation feature representation is constructed. Based on the attention-enhanced relation feature representation and the initial joint feature, a relation-enhanced joint representation is obtained; An initial description generation module is used to perform initial description generation processing on the joint representation after relationship enhancement to obtain an initial image description; A candidate text construction module is used to concatenate the guiding text with the initial image description to form candidate text; The description refinement module is used to encode the candidate text to obtain a candidate text representation, and then perform description refinement processing on the joint representation after relationship enhancement and the candidate text representation to obtain an image description result.
8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a text-guided spectral image description method according to any one of claims 1 to 6.
9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements a text-guided spectral image description method according to any one of claims 1 to 6.