Vision enhancement based text representation optimization method, device and equipment and medium

Abstract

The application relates to the technical field of intelligent decision-making, and discloses a text representation optimization method and device based on visual enhancement, equipment and a medium, which comprises the following steps: acquiring a picture-text pairing sample, generating an input text sequence and an original image feature set, and obtaining an initial text embedding; performing latent mask processing on the original image feature set to obtain a mask image feature set and a to-be-reconstructed image feature; recovering the to-be-reconstructed image feature based on the initial text embedding to obtain a reconstructed feature set; generating an association score matrix and a visual enhancement text embedding based on the initial text embedding and the reconstructed feature set; and determining a joint loss based on the visual enhancement text embedding, the original image feature set and the reconstructed feature set to obtain a target text encoder. The application can be applied to business scenarios such as financial technology and medical health, and through the introduction of image feature recovery, gated alignment and joint loss constraints, the text representation can absorb image region information and the correspondence between the text representation and a target task can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and in particular to a method, apparatus, device, and medium for optimizing text representation based on visual enhancement. Background Technology

[0002] In recent years, joint text and image modeling techniques have continued to develop in the direction of text-to-image generation, but existing technologies still have significant shortcomings in the text representation generation stage. The text features output by existing text encoders primarily reflect linguistic semantic relationships, lacking a stable correspondence with image region content, which can easily lead to insufficient text representation's ability to point to visual content. Simultaneously, existing cross-modal association methods mostly rely on static attention allocation, making it difficult to continuously correct the correspondence between text and image regions during training. On the other hand, directly fine-tuning the entire joint text and image model often results in a heavy training burden and is not conducive to effectively transferring the cross-modal constraints obtained during training into the text encoding results.

[0003] In the fintech business, text-to-image generation can be used for financial product presentations, investment research visualization, marketing material generation, and risk warning content generation. The input text in these scenarios typically contains financial keywords such as product elements, risk information, return descriptions, and customer segmentation information. This requires the text representation to accurately express financial semantics while also maintaining a stable correspondence with chart, prompt, and display areas in the image. In existing technologies, if the text encoding result lacks visual directional information, it is prone to issues such as inconsistencies between the generated content and the financial semantics, insufficient mapping of key information, and regional expression misalignment. Furthermore, if the correspondence between text and image lacks dynamic adjustment, it struggles to adapt to changes in key semantics within the financial text at different stages of generation.

[0004] In the healthcare field, text-to-image generation can be used to generate health education content, visualize examination instructions, and display treatment processes. The input text in such scenarios typically includes symptom descriptions, examination items, treatment prompts, risk disclosures, and monitoring indicators, requiring a high degree of correspondence between text representation and image regions. In existing technologies, if the text encoding results only remain at the linguistic semantic level and fail to fully absorb information related to image regions, it can easily lead to inaccurate representation of key parts, process nodes, and prompt areas in the generated content. Furthermore, if the training phase lacks continuous constraints on the relationship between text representation and image regions, it is difficult to obtain text representations that simultaneously possess semantic expressiveness and visual directional capabilities. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for optimizing text representation based on visual enhancement, aiming to solve the technical problem that existing technologies are unable to introduce image region information and dynamically establish the correspondence between text and image regions during the text encoding process, resulting in a lack of stable visual pointing ability in the generated text representation.

[0006] To achieve the above objectives, the present invention provides a text representation optimization method based on visual enhancement, comprising: Obtain a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and positional encoding operations to generate an input text sequence, process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set; The input text sequence is input into the text encoding branch, and the input text sequence is processed by a self-attention mechanism to generate a hidden state sequence as the initial text embedding. The original image feature set is processed by latent masking operations to generate a masked image feature set and the image features to be reconstructed. The initial text embedding and the mask image feature set are input into the visual guidance branch to recover the features of the image to be reconstructed and generate a reconstructed feature set. Based on the initial text embedding and the reconstructed feature set, gated alignment is performed to generate an association score matrix; The initial text embeddings are weighted and aggregated based on the association score matrix to generate visually enhanced text embeddings. The joint loss is determined based on the image-text pairing sample set, the visually enhanced text embeddings, the original image feature set, and the reconstructed feature set. The text encoding branch and the visual guidance branch are updated based on the joint loss. When the joint loss satisfies the convergence condition, the updated text encoding branch is retained as the target text encoder. The target task is processed based on the target text encoder to obtain the target task result.

[0007] Furthermore, to achieve the above objectives, the present invention provides a visually enhanced text representation optimization device, comprising: The multimodal data construction module is used to acquire a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and position encoding operations to generate an input text sequence, and process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set. The text encoding module is used to input the input text sequence into the text encoding branch, process the input text sequence through a self-attention mechanism, and generate a hidden state sequence as the initial text embedding; The mask generation module is used to process the original image feature set through latent masking operations to generate a masked image feature set and image features to be reconstructed. The cross-modal reconstruction module is used to input the initial text embedding and the mask image feature set into the visual guidance branch, recover the features of the image to be reconstructed, and generate a reconstructed feature set. The alignment modeling module is used to perform gated alignment based on the initial text embedding and the reconstructed feature set to generate an association score matrix. The joint optimization construction module is used to weight and converge the initial text embedding based on the association score matrix to generate visually enhanced text embedding, and to determine the joint loss based on the image-text pairing sample set, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set; The model update module is used to update the text encoding branch and the visual guidance branch based on the joint loss, and retain the updated text encoding branch as the target text encoder when the joint loss satisfies the convergence condition. The task reasoning module is used to process the target task based on the target text encoder and obtain the target task result.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a visually enhanced text representation optimization program stored in the memory and executable on the processor, wherein when the visually enhanced text representation optimization program is executed by the processor, it implements the steps of the visually enhanced text representation optimization method as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a visually enhanced text representation optimization program, wherein the visually enhanced text representation optimization program, when executed by a processor, implements the steps of the visually enhanced text representation optimization method as described above.

[0010] Beneficial Effects: This invention relates to the field of intelligent decision-making technology and discloses a method, apparatus, device, and medium for optimizing text representation based on visual enhancement. The method includes: acquiring image-text pairing samples to generate an input text sequence and an original image feature set; performing self-attention encoding on the input text sequence to obtain an initial text embedding; performing latent masking on the original image feature set to obtain a masked image feature set and image features to be reconstructed; recovering the image features to be reconstructed based on the initial text embedding and the masked image feature set to obtain a reconstructed feature set; generating an association score matrix based on the initial text embedding and the reconstructed feature set to obtain a visually enhanced text embedding; determining a joint loss based on the image-text pairing samples, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set; updating the text encoding branch and the visual guidance branch; retaining the target text encoder; and outputting the target task result. This invention can be applied to business scenarios such as fintech and healthcare. By introducing image region recovery, gating alignment, and joint loss constraints during the text encoding process, the text representation simultaneously incorporates text semantic information and image region information, thereby enhancing the visual orientation capability of the text representation and improving the consistency between the target task result and the input semantics. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a text representation optimization method based on visual enhancement according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the text representation optimization method based on visual enhancement of the present invention. Figure 3 for Figure 2 A detailed flowchart of step S10; Figure 4 for Figure 2 A detailed flowchart of step S30; Figure 5 for Figure 2 A detailed flowchart of step S40; Figure 6 for Figure 2 A detailed flowchart of step S50; Figure 7 for Figure 2 A detailed flowchart of step S60; Figure 8 for Figure 2 A detailed flowchart of step S70; Figure 9 This is a schematic diagram of the functional modules of a preferred embodiment of the visual enhancement-based text representation optimization device of the present invention; Figure 10 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 11 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The visual enhancement-based text representation optimization method provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain image-text pairing samples from the client, generate an input text sequence and an original image feature set; perform self-attention encoding on the input text sequence to obtain an initial text embedding; perform latent masking on the original image feature set to obtain a masked image feature set and the image features to be reconstructed; recover the image features to be reconstructed based on the initial text embedding and the masked image feature set to obtain a reconstructed feature set; generate an association score matrix based on the initial text embedding and the reconstructed feature set to obtain a visually enhanced text embedding; determine the joint loss based on the image-text pairing samples, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set, update the text encoding branch and the visual guidance branch, retain the target text encoder, and output the target task result. This invention can be applied to business scenarios such as fintech and healthcare. By introducing image region recovery, gating alignment, and joint loss constraints during the text encoding process, the text representation simultaneously inherits text semantic information and image region information, thereby enhancing the visual orientation capability of the text representation and improving the consistency between the target task result and the input semantics. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the visually enhanced text representation optimization method provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the text representation optimization method based on visual enhancement proposed in this invention includes the following steps: S10, obtain the image-text pairing sample set, process the paired text in the image-text pairing sample set through word segmentation and position encoding operations to generate an input text sequence, process the paired images in the image-text pairing sample set through a visual model to generate an original image feature set; In this embodiment, the formation of the image-text pairing sample set relies on the correspondence constraints between text content and image content. These correspondence constraints can be established based on page identifiers, template identifiers, title semantic consistency, tag consistency, page layout relationships, or content unit numbers, ensuring that text content and image content fall within the same representation unit. The resulting set is not a simple juxtaposition of text and images, but rather preserves a calculable and verifiable correspondence between them. To reduce subsequent representational bias, records with inconsistent titles, topic deviations, incorrect image-text binding, or duplicate content can be removed during the set construction stage.

[0016] Word segmentation handles the segmentation of semantic units. Segmentation boundaries are constrained by term integrity, phrase integrity, the binding relationship between numbers and units, and the dependency relationship of qualifiers. Implementation can employ a combination of dictionary matching, rule-based segmentation, statistical segmentation, and phrase integrity correction to ensure that complex expressions retain their complete semantics after segmentation. The resulting semantic units are arranged in the original text order, forming a word sequence. Positional encoding handles the writing of sequence information, including intra-sentence positions, intra-paragraph positions, cross-paragraph positions, and the relative positions of different content blocks. Implementation can employ absolute positional encoding, relative positional encoding, or a combination of paragraph and global positional encoding to write semantic unit representations and positional representations into the same sequence structure, resulting in the input text sequence. Therefore, the input text sequence retains both text content and permutation relationships, rather than just word meaning.

[0017] The visual model is responsible for mapping image content to region-level representations. In implementation, paired images are typically first divided into regions, resulting in multiple image region units. Region division can be based on fixed grids, page block boundaries, salient region boundaries, object boundaries, or chart structure boundaries. Each image region unit after region division is input into the visual model, which extracts region-level features from each unit. The visual model may include image patch mapping units, local visual encoding units, region relation encoding units, and feature aggregation units. The image patch mapping unit converts each image region unit into a vector representation; the local visual encoding unit extracts edge, texture, color distribution, and local contour information; the region relation encoding unit writes the top-bottom, left-right, adjacency, and containment relationships between image region units; and the feature aggregation unit outputs region-level features of a uniform dimension.

[0018] The original image feature set is formed by arranging multiple region-level features according to their positional order in the image. The arrangement order can be layout order, reading order, grid order, or object appearance order. To ensure that subsequent processing stages can continue to utilize region relationships, region numbers, region locations, and region types can be preserved simultaneously during arrangement. The resulting original image feature set retains both the content information of local regions and the distribution relationships and organizational structure among multiple regions.

[0019] In one implementation, paired text is segmented using a domain dictionary constraint, while paired images are processed using a layout block-based visual approach. On a fintech page, product descriptions, return tips, and risk disclosures are organized as paired text, while product cards, return curves, and risk warning boxes are organized as paired images. During segmentation, return comparison benchmarks, risk levels, and redemption restrictions are preserved as complete semantic units. Position encoding writes the title area, return area, and risk warning area into different position segments, forming the input text sequence. Paired images are divided into layout blocks according to the product main image area, chart area, and warning area. The visual model extracts the region-level features of each area and then combines them according to the layout order to form the original image feature set. On a healthcare page, dietary advice, monitoring time, and abnormal notifications are organized as paired text, while dietary illustrations, time icons, and reminder icons are organized as paired images. During word segmentation, post-meal monitoring, nutritional intervention, and exercise frequency are retained as complete semantic units. The location encoding writes the suggestion area, reminder area, and notification area into different location segments. The paired images are divided into page blocks according to the schematic diagram area, icon area, and prompt area. Then, the regional features are extracted and combined according to the reading order.

[0020] In another implementation, paired text employs a combination of statistical segmentation and phrase integrity constraints, while paired images utilize object boundary delineation for visual processing. Line charts, revenue labels, and fee rate prompts on fintech pages can be split into multiple image region units based on object boundaries, allowing chart titles, curve areas, and annotation areas to form independent region-level features. Similarly, process arrows, operation icons, and warning signs on healthcare pages can be split into multiple image region units based on object boundaries, creating finer visual representations of reminder objects, action objects, and warning objects. The text side no longer relies solely on a fixed dictionary but utilizes phrase integrity constraints to retain complex expressions such as revenue ranges, coverage responsibilities, abnormal indicators, and behavioral restrictions. This results in a finer input text sequence and a more refined set of original image features, suitable for page content with numerous chart and graph elements.

[0021] This embodiment, through text and image correspondence organization, complete semantic unit segmentation, positional information writing, and regional visual feature extraction, can simultaneously preserve the business semantics in the text and the regional distribution relationship in the image, so that the input text sequence and the original image feature set have a clearer correspondence basis, thereby alleviating the problem of difficulty in aligning text content with image region content.

[0022] S20, the input text sequence is input into the text encoding branch, and the input text sequence is processed by the self-attention mechanism to generate a hidden state sequence as the initial text embedding; In this embodiment, the input text sequence carries both token representations and positional representations. After receiving the sequence, the text encoding branch constructs a query vector, key vector, and value vector for each token. The self-attention mechanism generates attention weights based on the correlation between the query vector and the key vector, and then uses these attention weights to weighted convergence of the value vectors, ensuring that the representation of a single token simultaneously includes its own semantics and the contextual information provided by other tokens within the sequence. To avoid the dilution of local semantics in long sequences, the text encoding branch can adopt a multi-layer stacked structure. Each layer contains attention units, residual connection units, normalization units, and feedforward transformation units, with the output of the previous layer serving as the input of the next layer. The hidden state sequence is composed of the token representations encoded by each layer in their original order. Each position in the hidden state sequence corresponds to a text representation after context fusion. The initial text embedding is not an additional, independent object, but rather uses the hidden state sequence as the text representation result for subsequent processing. Therefore, the dimension, length, and order of the hidden state sequence need to maintain a one-to-one correspondence with the input text sequence. To balance encoding accuracy and computational complexity, the number of attention heads, encoding layers, feedforward dimension, and upper limit of sequence length can be set to ensure stable output of text encoding branches under different text complexities.

[0023] In one implementation, the text encoding branch employs a multi-layer, multi-head self-attention encoding structure. The input text sequence is first linearly mapped to form a basic sequence representation, which is then sequentially input into multiple encoding layers. Within each encoding layer, multiple attention heads learn different lexical association patterns. The outputs from these heads are concatenated and then linearly transformed to unify the dimensions. Subsequently, residual connections and normalization units stabilize the numerical distribution, followed by a feedforward transformation unit to enhance non-linear expressive power, ultimately outputting the hidden state sequence. This implementation is suitable for content with moderate text length and dispersed semantic dependencies.

[0024] In another implementation, the text encoding branch employs a hierarchical encoding structure combining local window attention and global attention. The first encoding layer calculates attention weights only within adjacent word groups to preserve phrase-level and intra-sentence dependencies, while the second encoding layer calculates attention weights across the entire sequence to supplement cross-sentence and cross-segment semantic relationships, ultimately outputting the hidden state sequence. This implementation is suitable for longer texts and content where local prompts and global explanatory texts coexist. Gating units can also be added between encoding layers to adjust the output ratio of different layers, ensuring that the hidden state sequence balances local semantic clarity and global semantic completeness.

[0025] In the healthcare business, input text can include dietary recommendations, monitoring times, and alerts for abnormalities. After text encoding branching, the semantic relationships between post-meal monitoring and time conditions, and between alerts for abnormalities and behavioral restrictions, are written into the hidden state sequence, ensuring that related content maintains a clearer contextual association in the initial text embedding.

[0026] In the fintech business, input text can include profit descriptions, risk warnings, and redemption rules. After text encoding branching, the semantic relationships between profit ranges and limiting conditions, and between risk warnings and product rules, are written into the hidden state sequence, ensuring that related content maintains a clearer contextual association in the initial text embedding.

[0027] For example, the formula for text self-attention weights is:

[0028] in, This represents the attention weight of the i-th text position to the j-th text position. This represents the input representation corresponding to the i-th input text word. This represents the input representation corresponding to the j-th input text word. This indicates a query for the projection matrix. Represents the key projection matrix. Indicates the embedding dimension. () represents the normalization function.

[0029] The formula for generating the hidden state of text encoding is:

[0030] in, This represents the hidden state corresponding to the i-th text position. This represents the input representation corresponding to the i-th input text word. This represents the attention weight of the i-th text position to the j-th text position. This indicates a query for the projection matrix. This represents the projection matrix. N represents the number of text terms in the input text sequence. LayerNorm() represents the layer normalization operation.

[0031] This embodiment uses a self-attention mechanism to perform contextual modeling on the input text sequence, enabling individual lexical representations to absorb the associated information within the sequence and form a hidden state sequence that maintains the original order after multi-layer encoding. This results in an initial text embedding that simultaneously contains local and global semantics, reducing the situation where the text representation is too isolated or lacks sufficient contextual information.

[0032] S30, the original image feature set is processed by a latent masking operation to generate a masked image feature set and image features to be reconstructed; In this embodiment, the original image feature set consists of multiple image region features arranged in a predetermined order, with each image region feature corresponding to a local region or a structural unit in the paired image. The latent masking operation does not directly delete image region features; instead, it generates a mask selection result on the original image feature set and then divides the original image feature set into a retained portion and a masked portion based on the mask selection result. The purpose of this process is to retain the contextual distribution of some region features while transforming the other portion of region features into objects to be restored. The mask selection result can be controlled by a preset retention probability or by a combination of region location, region density, and region clustering relationships. To avoid excessive dispersion of the masked regions, spatially continuous or positionally adjacent masked regions can be merged after mask selection, ensuring that the image features to be reconstructed include both individual region features and local fragment features. The masked image feature set is not simply a stack of remaining region features; instead, it rearranges the retained image region features in their original order and retains mask markers or empty space information at the masked positions, preserving the original positional relationships within the set. The image features to be reconstructed are composed of the image region features corresponding to the masked regions, used to represent the visual content of the missing parts. The two results obtained in this way retain the visible area information and the missing area information respectively, while maintaining the regional distribution relationship in the original image feature set.

[0033] In one implementation, a binary mask vector is generated on the original image feature set according to a preset retention probability. The retention markers in the mask vector correspond to retained image region features, and the masking markers correspond to masked image region features. Then, masked image region features are extracted from the original image feature set according to the mask vector and combined to form the image features to be reconstructed. Retained image region features are extracted from the original image feature set, and mask markers are written into their original positions to form a masked image feature set. This method is suitable for image content with relatively uniform regional distribution.

[0034] In another implementation, positional adjacency relationships are first established for image region features in the original image feature set. Then, initial masked regions are selected according to a preset retention probability. Adjacent initial masked regions are then merged into continuous segments. The image region features corresponding to these continuous segments are combined to form the image features to be reconstructed. The unselected image region features are arranged in their original order, and segment-level mask markers are written at the corresponding positions of the continuous segments, forming a masked image feature set. This method is suitable for image content with clear local structures and strong correlations between regions.

[0035] Another approach is to use a layered masking method. First, the original image feature set is divided into a global feature subset and a local feature subset. A low-proportion mask is applied to the global feature subset, while a high-proportion mask is applied to the local feature subset. Then, the masked portions are combined into the features of the image to be reconstructed, and the retained portions are combined with the mask markers to form the masked image feature set. This method can increase the intensity of local missing information while preserving overall layout information.

[0036] In the healthcare field, paired images contain dietary indication areas, time icon areas, and reminder marker areas. During processing, the time icon areas and reminder marker areas can be selected as continuous masked regions, while the dietary indication areas are retained in the masked image feature set. The image region features corresponding to the time icon areas and reminder marker areas constitute the image features to be reconstructed.

[0037] In the fintech business, paired images include product card areas, yield curve areas, and risk warning areas. During processing, local curve segments within the yield curve area and the risk warning area can be treated as masked regions, while the product card area and the remaining curve areas are retained in the masked image feature set. The image region features corresponding to the masked curve segments and risk warning areas constitute the image features to be reconstructed.

[0038] This embodiment achieves controlled masking on the original image feature set and generates a masked image feature set and the image features to be reconstructed separately. This allows for the explicit separation of visible and missing content while preserving the regional distribution relationship. This ensures that the image region information maintains a usable contextual structure and forms a clear restoration object, thereby alleviating the problem of overly complete image feature representations making it difficult to establish restoration constraints.

[0039] S40, input the initial text embedding and the mask image feature set into the visual guidance branch, recover the features of the image to be reconstructed, and generate a reconstructed feature set; In this embodiment, the initial text embedding carries the semantic representation of the encoded text sequence, while the mask image feature set carries the visual representation of the preserved image region and the mask position information. The target visual content of the masked region corresponds to the image features to be reconstructed. The visual guidance branch undertakes the cross-modal recovery function and can be composed of a cross-modal attention unit, a feature fusion unit, and a normalization unit. In implementation, the initial text embedding is first mapped to text key features and text value features, and the mask image feature set is mapped to image query features. Then, the correlation between the image query features and text key features is used to generate attention weights, and the text value features are weighted and converged to obtain the text-guided recovery features. The text-guided recovery features and the preserved visual features in the mask image feature set participate in the fusion, so that the recovery result includes both text semantic constraints and preserves the visual context of the visible region.

[0040] The restoration of features in the image to be reconstructed involves more than just filling in missing values ​​at individual locations; it also requires maintaining the continuity between the missing regions and adjacent regions. In practice, mask location information can be used as the basis for restoration and localization. Text-guided restored features are then filled into the corresponding missing locations, and residual fusion and normalization are performed with the preserved visual features to form a complete set of region-level representations. The reconstructed feature set consists of all restored location features arranged in their original order. Each location in the set retains both the original image region distribution relationships and the visual representation of the missing region supplemented by textual semantics. This results in mapping object descriptions, attribute descriptions, and relational descriptions from the text to the visual representation of the masked region.

[0041] In one implementation, the visual guidance branch employs a single-layer cross-modal attention structure. The initial text embedding is processed through two sets of linear mappings to generate text key features and text value features. The masked image feature set is then processed through another set of linear mappings to generate image query features. Relevance weights are calculated between the image query features and the text key features. These relevance weights are then used to weight and converge the text value features, outputting the text-guided recovery features. These text-guided recovery features are written back at the masked locations, concatenated with the preserved visual features, and then linearly fused and normalized to obtain the reconstructed feature set. This approach is suitable for content with a small number of image regions and relatively dispersed missing regions.

[0042] In another implementation, the visual guidance branch employs a fragment-level reconstruction structure. First, the mask image feature set is divided into multiple missing and retained fragments based on the mask position. Then, image query features are constructed for each missing fragment, and text key and text value features are uniformly generated for the initial text embedding. The relevance weight of each missing fragment with the text key feature is calculated to obtain fragment-level text-guided reconstruction features. These features are then locally fused with adjacent retained fragments. Finally, all fragment reconstruction results are merged in their original positional order to obtain the reconstructed feature set. This approach is suitable for image content with continuously distributed missing regions and strong local structural correlations.

[0043] A dual-branch fusion structure can also be used. One branch generates text-guided recovery features using cross-modal attention, while the other branch extracts neighborhood visual statistical features from the masked image feature set. The two branches work together in the fusion unit to form the recovery result, which is then normalized to obtain the reconstructed feature set. This approach can simultaneously utilize the semantic supplementation capabilities of the text and the local visual constraints of the visible region.

[0044] In the healthcare business, input text can include dietary control suggestions, monitoring time prompts, and abnormal alerts. The paired image contains dietary indication areas, time icon areas, and alert label areas. The dietary indication areas are preserved in the masked image feature set, and the missing positions corresponding to the time icon areas and alert label areas are recovered by a visual guidance branch. The recovered area order remains consistent with the original page layout.

[0045] In the fintech business, input text can include profit descriptions, risk warnings, and redemption restrictions. The paired image contains product card areas, profit curve areas, and risk warning areas. The product card area and part of the curve area are preserved in the masked image feature set. Missing curve segments and risk warning areas are recovered by visually guided branches. The recovery result can maintain the relative positional relationship between the chart trend area and the warning area.

[0046] This embodiment feeds the initial text embedding and the mask image feature set into the visual guidance branch, and uses cross-modal attention and feature fusion to complete the recovery of missing regions. This allows the visual representation of the masked region to simultaneously absorb textual semantic information and visible region context information, thus obtaining a reconstructed feature set that is consistent with the original region order and has a stronger content completion capability.

[0047] S50, based on the initial text embedding and the reconstructed feature set, gating alignment is performed to generate an association score matrix; In this embodiment, the initial text embedding contains semantic vectors corresponding to multiple text locations, and the reconstructed feature set contains visual vectors corresponding to multiple image regions. Gated alignment estimates the correspondence strength between the two types of representations, outputting an association score matrix. Typically, the initial text embedding and reconstructed feature set are first mapped to an alignment space of the same dimension, making the text location vectors and image region vectors comparable. The two mapped vector sets are then paired to form candidate pairs between text locations and image regions. The gating unit generates a gating weight for each candidate pair, reflecting the information transfer ratio between the current text location and the current image region. The gating weight can be determined by one or more of the following: concatenation features, difference features, and product features. To avoid alignment shifts caused by relying solely on the gating weight, the similarity score between the text location vector and the image region vector can also be calculated simultaneously, and the gating weight and the similarity score can be combined to form a single association score.

[0048] The row and column relationships in the association score matrix need to be fixed. The row direction can correspond to the text positions in the initial text embedding, and the column direction can correspond to the image regions in the reconstructed feature set. This results in a matrix that not only records the correspondence strength between a single text position and a single image region, but also preserves the local correspondence distribution between the entire text and the entire image. To improve matrix stability, association scores can be normalized, thresholded, or smoothed for segment consistency, ensuring the association results between adjacent image regions remain continuous in their local structure. After the matrix is ​​generated, the score at each position reflects both the degree of matching between the text semantics and the visual content, as well as the cross-modal throughput strength after gating.

[0049] One implementation employs a concatenation gating structure. Each text position vector in the initial text embedding is mapped to a unified dimension with each image region vector in the reconstructed feature set, and then concatenated according to pairwise correspondences. The concatenated result is input into a gating network to obtain gating weights. Simultaneously, the cosine similarity between the two classes of vectors is calculated, and the gating weights and cosine similarity are multiplied to obtain the association score. All association scores are arranged in order of text position and image region position, forming an association score matrix. This approach is suitable for content with significant variations in the number of text positions and image regions.

[0050] Another implementation employs a dual-projection alignment structure. The initial text embedding is passed through a text projection layer to obtain a text alignment vector. The reconstructed feature set is then passed through an image projection layer to obtain an image alignment vector. A bilinear mapping unit is then used to generate candidate correspondence intensities. A gating network generates gating weights based on the candidate correspondence intensities and the local differences between the two types of vectors. These two factors are jointly used to generate the association score. A local smoothing constraint is added to the association score before matrix formation, ensuring greater consistency in the association results between adjacent image regions. This approach is suitable for content where there is strong spatial continuity between image regions.

[0051] Alternatively, a separate gating structure can be used. The alignment space is divided into multiple subspaces, each of which independently generates gating weights and similarity scores. The results from multiple subspaces are then weighted and fused to obtain a single association score. This allows for the separate modeling of different correspondences between attribute words, object words, relation words, and image region textures, contours, and layouts, forming a more granular association score matrix.

[0052] In the healthcare business field, text content includes dietary advice, time reminders, and abnormality notifications, while image content includes dietary illustration areas, time icon areas, and reminder label areas. After gating alignment, the text positions representing time constraints receive higher correlation scores with the time icon areas, and the text positions representing abnormality notifications receive higher correlation scores with the reminder label areas, thus forming a region-discriminatory correlation score matrix.

[0053] In the fintech business, text content includes profit descriptions, risk warnings, and redemption conditions, while image content includes profit curve areas, risk warning areas, and product display areas. After gating alignment, text positions indicating profit changes receive higher correlation scores with the profit curve area, and text positions indicating risk warnings receive higher correlation scores with the risk warning area, thus forming a semantically oriented correlation score matrix.

[0054] This embodiment maps the initial text embedding and reconstructed feature set to a unified alignment space, and combines gating weights and similarity scores to generate an association score matrix. This allows for a more detailed differentiation of the correspondence strength between different text locations and different image regions, reducing the coarse matching caused by a single similarity metric, and making the cross-modal correspondence results more stable and more locally discriminative.

[0055] S60, the initial text embedding is weighted and aggregated based on the association score matrix to generate a visually enhanced text embedding, and the joint loss is determined based on the image-text pairing sample set, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set; In this embodiment, the rows in the association score matrix correspond to text positions, and the columns correspond to image regions. The scores in the matrix reflect the intensity of attention given to different image regions by the text positions. During weighted aggregation, the score of each text position is used as the weight distribution to weight and combine the text position vectors in the initial text embedding to obtain an aggregated result with image region bias. The aggregated vector does not replace the original text representation but is fused with the initial text embedding, so that the output result retains both the original semantic information and the image region pointing information, thus forming a visually enhanced text embedding. The fusion method can be weighted addition, gated fusion, or post-concatenation mapping, as long as the output dimension remains consistent with the subsequent training objective.

[0056] The joint loss consists of cross-modal matching constraints and reconstruction consistency constraints. The image-text pairing sample set provides matching relationships to distinguish correct image-text pairs from non-corresponding ones. A positive constraint is formed between the visually enhanced text embedding and matching image features in the original image feature set, and a negative constraint is formed between it and non-matching image features, thus yielding the contrastive loss. The region correspondence is preserved between the original image feature set and the reconstructed feature set, and the reconstruction loss can be formed by measuring region-level differences. The contrastive loss constrains the visual orientation of the text representation, and the reconstruction loss constrains the consistency between the restored result and the original visual content. These two constraints are combined with weights to form the joint loss, ensuring that text representation enhancement and image feature restoration converge together under the same objective.

[0057] In one implementation, the association score matrix is ​​first row-normalized, and then the initial text embedding is weighted and summed row-wise to obtain aggregated text features. The aggregated text features and the initial text embedding are then gated and fused to generate visually enhanced text embeddings. The correspondence in the image-text pairing sample set is used to construct matching image-text pairs and non-matching image-text pairs. The similarity between the visually enhanced text embedding and the matching image features is increased, while the similarity with the non-matching image features is decreased, forming a contrast loss. The reconstructed feature set and the original image feature set are aligned one-to-one by region, and a reconstruction loss is formed by a region-level distance metric. The two types of losses are then combined into a joint loss according to a preset ratio.

[0058] In another implementation, the association score matrix is ​​first thresholded, with low-weight scores attenuated. Then, the initial text embedding is region-selectively aggregated based on the filtered scores to obtain aggregated text features. These aggregated text features and the initial text embedding are concatenated and mapped to generate visually enhanced text embeddings. A set of image-text paired samples is used to form a hard-negative sample set, allowing the visually enhanced text embeddings to participate in the contrastive loss calculation along with semantically similar but content-discordant image features. The reconstruction loss employs a combination of region difference and structural difference, ensuring that the reconstructed feature set approximates the original image feature set both locally and globally, ultimately forming a joint loss.

[0059] This embodiment uses an association score matrix to perform region-weighted aggregation on the initial text embedding, which can incorporate image region information into the text representation, enabling visually enhanced text embedding to preserve both text semantics and visual orientation. Furthermore, by combining image-text matching constraints and reconstruction consistency constraints to form a joint loss, it can simultaneously constrain cross-modal correspondences and visual recovery quality, thereby mitigating the problems of text representation lacking image region orientation information and the deviation of the recovery result from the original visual content.

[0060] S70, update the text encoding branch and the visual guidance branch based on the joint loss, and retain the updated text encoding branch as the target text encoder when the joint loss meets the convergence condition; In this embodiment, the text encoding branch and the visual guidance branch together constitute a dual-branch network structure during the training phase. The text encoding branch receives the input text sequence and outputs the initial text embedding, while the visual guidance branch receives the mask image feature set and the initial text embedding and outputs the reconstructed feature set. The two are coupled by a joint loss. The text encoding branch can adopt a stacked encoding structure of 6, 8, 12, or 16 layers, with each layer including a multi-head self-attention unit, a feedforward transform unit, a residual connection unit, and a normalization unit. The hidden dimension can be set to 512, 768, or 1024, the number of attention heads can be set to 8, 12, or 16, and the feedforward layer dimension can be set to 2, 4, or 8 times the hidden dimension. The visual guidance branch can include a cross-modal attention unit, a fragment-level recovery unit, a local fusion unit, and a normalization unit. The cross-modal attention unit receives the initial text embedding and the mask image feature set, the fragment-level recovery unit generates recovery features based on the location of missing fragments, the local fusion unit fuses the recovery result with the features of the preserved image region, and the normalization unit constrains the distribution of the recovery features. The output of the text encoding branch is connected to the input of the visual guidance branch, and the output of the visual guidance branch is connected to the gating alignment unit and the joint loss calculation unit, thus forming a training closed loop from text representation generation, image region recovery, cross-modal alignment, loss feedback to bi-branch parameter update.

[0061] After the joint loss is formed, gradient backpropagation is performed on the text encoding branch and the visual guidance branch separately. During gradient backpropagation, the joint loss can be decomposed into the update amounts of the text encoding branch and the visual guidance branch, so that the text encoding branch focuses on absorbing image region pointing information, while the visual guidance branch focuses on absorbing text semantic constraint information. To avoid an imbalance in the update magnitudes of the two branches during training, the gradient norms of the two branches can be calculated separately, and cropping, scaling, or normalization can be performed based on the gradient norms. The upper limit of the gradient norm can be set to 1, 2, 5, or 10. When the gradient norm of the text encoding branch is significantly higher than that of the visual guidance branch, the learning rate of the text encoding branch can be reduced or the update magnitude of high-level parameters can be limited; when the gradient norm of the visual guidance branch remains excessively high, the parameter update ratio of the cross-modal attention unit and the local fusion unit can be reduced. Through this dual-branch gradient coordination method, the text encoding branch gradually absorbs visual constraints while retaining its original semantic expressive ability, while the visual guidance branch maintains its region recovery ability while avoiding excessive traction on the overall training direction.

[0062] During the training phase, either a unified optimizer or a branch-difference optimizer can be used for updates. In the unified optimizer approach, the text encoding branch and the visual guidance branch use the same optimizer, such as Adam, AdamW, or a stochastic gradient descent optimizer with momentum. The learning rate can be set to 1e-5, 3e-5, 1e-4, or 3e-4, and the weight decay coefficient can be set to 1e-2, 5e-3, or 1e-3. In the branch-difference optimizer approach, the text encoding branch uses a smaller learning rate, while the visual guidance branch uses a larger learning rate. For example, the text encoding branch's learning rate can be set to 1e-5 or 3e-5, and the visual guidance branch's learning rate can be set to 1e-4 or 3e-4. This makes the text representation updates smoother and allows the visual recovery branch to adapt to cross-modal constraints more quickly. A phased learning rate strategy can also be used, using a larger learning rate in the early stages of training to drive the visual guidance branch to develop stable recovery capabilities, and then reducing the learning rate in the later stages of training to allow the text encoding branch to gradually and stably converge. The learning rate decay method can be cosine annealing, piecewise decay, or linear decay after warm-up. The number of warm-up rounds can be set to 5%, 10%, or 20% of the total training rounds.

[0063] When inputting training data, image-text pairing sample sets can be organized in batches. Batch size can be set to 16, 32, 64, or 128; maximum text sequence length can be set to 64, 128, 256, or 512; and the upper limit for the number of image region features can be set to 16, 32, 64, or 128. For long texts or high-density image regions, a batch-accumulated gradient method can be used to maintain an effective batch size, with the accumulation step count set to 2, 4, or 8. To reduce memory consumption, a mixed-precision training method can be used, employing a hybrid FP16 and FP32 representation during forward computation and backpropagation, and then restoring to a high-precision parameter copy during parameter updates. For distributed training environments, data parallelism, model parallelism, or tensor parallelism can be used to enable collaborative training of the text encoding branch and the visual guidance branch on different computing devices.

[0064] The convergence condition should not rely solely on the joint loss value of a single epoch, but should be determined by comprehensively considering the trend of joint loss changes, the magnitude of parameter changes, and the stability of the branches. In implementation, after each training epoch, the joint loss value, the first-order difference of the joint loss, the second-order fluctuation of the joint loss, the magnitude of parameter changes in the text encoding branch, and the magnitude of parameter changes in the visual guidance branch can be recorded. If the decrease in joint loss is below a preset threshold for 3, 5, or 10 consecutive epochs, or if the joint loss remains stable within a preset range, and the magnitudes of parameter changes in both branches enter a stable range, then the convergence condition can be considered met. The joint loss decrease threshold can be set to 1e-3, 5e-4, or 1e-4, and the parameter change magnitude threshold can be set to 1e-2, 5e-3, or 1e-3 according to the parameter norm ratio. To avoid misjudgments caused by random fluctuations, the joint loss of the most recent epochs can be processed using a moving average or exponential smoothing method, and then the convergence determination can be made based on the smoothed joint loss sequence.

[0065] To ensure that the final retained text encoding branch corresponds to a relatively optimal parameter state, parameter checkpoints can be continuously saved during training. The saving period can be set by epoch or triggered by the decrease in joint loss. Each checkpoint records at least the text encoding branch parameters, the visual guidance branch parameters, the optimizer state, the joint loss value, gradient norm statistics, and the epoch number. After the convergence condition is met, instead of mechanically retaining the parameters from the last epoch, the updated text encoding branch is selected from the most recent checkpoints based on a comprehensive evaluation of the minimum joint loss, parameter fluctuation range, text encoding branch stability, and visual guidance branch stability. The selection strategy can prioritize the minimum joint loss or use a joint scoring method that minimizes both joint loss and parameter stability. The resulting target text encoder is not just a snapshot of the parameters at the end of training, but a parameter combination that achieves a good balance between text semantics and visual pointing ability in a stable state.

[0066] In terms of parameter update strategies, a phased update approach can also be adopted. In the early stages of training, the low-level parameters of the text encoding branch are frozen, and only the high-level parameters of the text encoding branch and the visual guidance branch are updated, allowing cross-modal constraints to apply to the high-level semantic representation first. In the middle stages of training, the mid-level parameters of the text encoding branch are gradually unfrozen, allowing the text representation to begin absorbing finer-grained visual constraints. In the later stages of training, all parameters of the text encoding branch are released, and the learning rate of the visual guidance branch is reduced, making the text encoding branch the primary convergence target. The number of freeze and unfreeze phases can be set proportionally to the total number of training phases; for example, the low-level parameters are frozen in the first 20% of phases, the mid-level parameters are gradually unfrozen in the middle 40% of phases, and all parameters are released in the last 40% of phases. This phased update approach can mitigate the impact of large-scale cross-modal constraints on the stability of the text representation in the early stages of training, allowing the target text encoder to form a smoother convergence trajectory.

[0067] In fintech business scenarios, the input text sequence can include text content such as profit descriptions, risk disclosures, holding periods, fee descriptions, redemption conditions, and guarantee liability descriptions. Image region features can come from product card areas, profit curve areas, risk warning areas, fee display areas, and description areas. The training output includes initial text embeddings, reconstructed feature sets, association score matrices, visually enhanced text embeddings, and joint loss. The final retained target text encoder is used to generate text representations with image region pointing capabilities, making profit descriptions more easily correspond to chart areas, risk warnings more easily correspond to prompt box areas, and product attributes more easily correspond to product display areas.

[0068] In healthcare scenarios, input text sequences can include monitoring reminders, dietary advice, behavioral instructions, anomaly alerts, screening instructions, and other textual content. Image region features can come from schematic diagrams, icon areas, process areas, and prompt areas. The training output format remains consistent, and the final target text encoder is used to generate text representations with image region pointing capabilities. This makes behavioral prompts more easily correspond to icon areas, process instructions more easily correspond to process node areas, and anomaly alerts more easily correspond to warning areas. This allows the cross-modal constraints absorbed during training to be internalized within the target text encoder, enabling it to output visually oriented text representations even when receiving only text input during the inference phase.

[0069] This embodiment applies joint loss to both the text encoding branch and the visual guidance branch simultaneously, enabling both branches to adjust parameters under the same objective. This allows the text encoding result to absorb visual constraint information without losing its original semantic representation capability. By setting convergence conditions and retaining the updated text encoding branch in a stable state, the instability caused by parameter fluctuations can be reduced, making the parameter state of the target text encoder more stable.

[0070] S80, the target task is processed based on the target text encoder to obtain the target task result.

[0071] In this embodiment, the target text encoder retains the updated parameter state and undertakes the text representation generation function during the inference phase. The target task can receive a target input text sequence, or a word sequence or phrase sequence converted from the target input text sequence. After semantically mapping the input content, the target text encoder outputs the target text embedding. The target text embedding can either maintain the positional sequence or be further compressed into a task-level conditional vector. The target task result is not simply the target text embedding directly as the final output, but rather the result obtained after solving the task based on the target text embedding. The target text embedding can be used as a conditional constraint, retrieval key, generation guide vector, or discrimination criterion in subsequent solving processes, so that the target task result retains the textual semantic information and the visual orientation information formed during the training phase.

[0072] The generation method of the target task result depends on the organization of the task solving unit. If the target task is a generative task, the target text embedding can be input into the result generation unit, which then generates the target output sequence, target image representation, or target content combination result based on the target text embedding. If the target task is a matching or filtering task, the target text embedding can be input into the result determination unit, which calculates the matching score between the candidate result and the target text embedding, and then determines the target task result based on the matching score. To enable the target text encoder output to adapt to different task solving units, a dimension mapping layer, a normalization layer, or a task adaptation layer can be set between the target text encoder and the result generation unit to ensure that the target text embedding meets the input requirements of the target task in terms of dimension, range, and distribution.

[0073] During the inference phase, the target text encoder no longer needs to rely on the visual guidance branch for additional input. Instead, it receives the target input text and outputs the target text embedding. When the target task is text-to-image generation, the target text embedding is fed into the generative model as a conditional input, and the generative model outputs the target task result. When the target task is retrieval, sorting, or discrimination, the target text embedding is fed into the corresponding task processing unit as a query representation or discrimination representation, and the task processing unit outputs the target task result. When the target task has different levels of attention to local and global semantics, the target text embedding can be converged at location, fragment, or globally before the converged representation is fed into the corresponding task processing unit.

[0074] In one implementation, the target task is a generative task. The target input text sequence is encoded by a target text encoder to obtain the target text embedding, which is then input into a result generation unit. The result generation unit can employ a decoding network or a conditional generation network to progressively generate the target output content based on the target text embedding. To enhance the fidelity of local semantics, the positional representations in the target text embedding can be fed into the result generation unit in their original order, allowing the result generation unit to utilize both global and local semantics to generate the target task result. For input content with longer semantics, the target text embedding can be first aggregated at the fragment level, and then the fragment-level representations and global representations can be input together into the result generation unit.

[0075] In another implementation, the target task is a matching task. The target input text sequence is encoded by a target text encoder to obtain the target text embedding. Each candidate in the candidate result set is converted into a candidate representation. Then, the result determination unit calculates the matching score between the target text embedding and each candidate representation. The target task result is determined by sorting the candidates according to their matching scores. To reduce confusion between highly similar candidates, a score correction layer can be added to the result determination unit to suppress mutual interference between candidate representations before outputting the final sorted result.

[0076] Alternatively, a task adaptation approach can be used. After the target text encoder outputs the target text embedding, it first passes through a task adaptation layer to adjust its dimensions and distribution before being fed into different types of task solving units. The task adaptation layer can adopt a linear mapping structure or a gated mapping structure. Generative tasks focus on preserving sequence information, while matching tasks focus on enhancing the global representation. Both share the same target text encoder but use different task adaptation layers, thus obtaining different forms of target task results.

[0077] This embodiment enables the target text encoder to independently generate text representations during the inference phase, thus preserving the cross-modal constraints formed during training within the target text embedding. The target text embedding then participates in result generation or result determination, ensuring that the target task result not only retains textual semantics but also more explicit visual directional information, thereby mitigating the problem of insufficient correspondence between the target task result and the input semantics.

[0078] In one embodiment, such as Figure 3 As shown, step S10 above includes: S101, extract text content and image content from multiple sets of candidate image and text samples, and pair and organize the text content and image content according to the image and text correspondence identifier to generate an image and text pairing sample set; S102, perform word segmentation on the paired text in the image-text pairing sample set, split the paired text into multiple consecutively arranged text words, and generate a word word sequence according to the arrangement order of the multiple text words; S103, perform position encoding operation on the word sequence, write each text word in the word sequence and its corresponding position representation into the same sequence structure to generate the input text sequence; S104, the paired images in the image-text pairing sample set are divided into regions to obtain multiple image region units, and the multiple image region units are input into a visual model to extract region-level features to obtain multiple region features; S105, the multiple regional features are arranged according to their position order in the paired image, and the arranged multiple regional features are combined to obtain the original image feature set.

[0079] In this embodiment, the text and image content in multiple candidate image and text samples need to undergo consistency screening before pairing and processing. The extracted text content can include titles, body text, tag phrases, prompts, and in-image text transcription results. The extracted image content can include full-page images, partial page views, chart areas, icon areas, and prompt box areas. Image-text correspondence identifiers are used to constrain the attribution relationship between text and image content. These identifiers can be page numbers, template numbers, business unit numbers, page block numbers, theme tag combinations, or time window identifiers. During pairing and processing, text and image content with consistent image-text correspondence identifiers are written into the same record unit, and data records with missing identifiers, conflicting themes between text and image content, duplicate attributions, or cross-page mixing are removed, thus forming an image-text pairing sample set. After this stage, the text and image content are no longer independent original records, but rather combined data with clear correspondence constraints.

[0080] The word segmentation operation of paired text performs the function of semantic unit segmentation. The segmentation boundary is jointly constrained by lexical boundaries, phrase integrity, binding relationships between numbers and units, and terminology specificity, ensuring that complex expressions such as profit comparison benchmarks, risk levels, redemption restrictions, post-meal monitoring, nutritional intervention, and abnormal notifications remain complete semantic units. After segmentation, multiple consecutively arranged text words are obtained. The consecutive arrangement does not simply record the order of positions, but preserves the order within sentences, paragraphs, and the connection relationships between adjacent words. After the word word sequence is formed, the positional encoding operation writes each text word word and its corresponding positional representation into the same sequence structure. The positional representation can include absolute position, relative position, paragraph position, or page position, so that the input text sequence retains both semantic content and order information. The resulting input text sequence is not an ordinary text string, but a sequence of data composed of multiple word word representations with positional constraints, which facilitates the subsequent encoding stage in distinguishing the semantic order of main information, supplementary information, and prompt information.

[0081] The region segmentation of paired images serves the function of visual content decomposition. The goal of region segmentation is not simply to cut the image, but to break down different visual functional units into independently representative image region units. Segmentation can be based on fixed grids, page block boundaries, object boundaries, salient region boundaries, or chart structure boundaries. After segmentation, multiple image region units are input into the visual model for region-level feature extraction. The multiple region features output by the visual model should correspond one-to-one with the image region units, ensuring that each region feature simultaneously carries local texture, edge contours, color distribution, icon shape, chart trends, or page block information. To ensure that multiple region features reflect the spatial organization relationships in the paired images, they also need to be arranged according to their positional order within the paired images. This positional order can adopt page reading order, row and column order, or region numbering order, as long as the region sorting rules within the same paired image remain consistent. The arranged multiple region features are further combined into an original image feature set, thereby preserving the local content of individual image regions and the distributional relationships between multiple image regions.

[0082] This embodiment uses image-text correspondence identifiers to constrain the attribution of text and image content, reducing data clutter caused by image-text mismatches and topic shifts. By maintaining the integrity of complex semantic units through word segmentation and incorporating positional representations into the same sequence structure, the input text sequence retains both semantic and sequential information. Region partitioning and region-level feature extraction form the original image feature set, preserving the local content and spatial distribution relationships in the paired images. The resulting input text sequence shares a consistent pairing origin and a clear structural order with the original image feature set, improving the clarity of the correspondence between text content and image region content.

[0083] In one embodiment, such as Figure 4 As shown, step S30 above includes: S301, the position index is marked for each image region feature in the original image feature set to generate an image region position sequence, and the spatial adjacency relationship between each image region feature is established based on the image region position sequence; S302, Randomly mask and screen the features of each image region in the original image feature set according to the preset retention probability to determine the retained image region features and the masked image region features; S303, based on the spatial adjacency relationship, the features of multiple shielded image regions that are continuously distributed in space are combined into segments to identify shielded image region feature groups corresponding to different spatial missing regions; S304, perform fragment merging on each feature group of the masked image region to generate multiple local missing fragments corresponding to different spatial missing regions; S305, Arrange multiple local missing segments sequentially, and combine the sequentially arranged local missing segments to generate the features of the image to be reconstructed; S306, rearrange the preserved image region features according to the image region position sequence to generate a preserved feature sequence; S307, write mask placeholders into the image region positions corresponding to the image features to be reconstructed in the retained feature sequence, and retain the image region position sequence corresponding to the image features to be reconstructed to generate a mask image feature set.

[0084] In this embodiment, each image region feature in the original image feature set corresponds to a local region in the paired image. Location index markers are used to bind each image region feature to a specific spatial location, ensuring that subsequent screening, merging, and rearrangement all revolve around the same location system. Location indexes can use two-dimensional grid coordinates, region numbers, center point coordinates, bounding box sequence identifiers, or page block numbers. After the location indexes are written, an image region location sequence is formed. This sequence records not only the order of image region features but also their relative distribution in the plane. When establishing spatial adjacency relationships based on the image region location sequence, adjacency can be determined by vertical adjacency, horizontal adjacency, diagonal adjacency, shared boundaries, or distance thresholds, forming a computable adjacency graph structure between different image region features. This spatial adjacency relationship allows the subsequent combination of masked regions to no longer depend on discrete random results but can instead form a continuous missing pattern based on the original spatial layout.

[0085] The preset retention probability controls the ratio of retained to masked portions in the original image feature set. Random masking is not simply deleting image region features, but rather creating a location-traceable screening result on the original image feature set. During screening, each image region feature is assigned a retention or masking state according to the preset retention probability, thus dividing the original image feature set into retained and masked image region features. The retained image region features carry visible context information, providing information about the remaining region content and its location distribution; the masked image region features identify missing content, providing information about the region range and target region content to be recovered. If image region features are only masked as discrete points, subsequent recovery is prone to bias towards single-point complementation, making it difficult to reflect local structural constraints. Therefore, after random masking, the masked regions need further organization based on spatial adjacency relationships.

[0086] Multiple spatially continuous masked image region features are fragmented and combined under spatial adjacency constraints. The result of this combination is not a simple stitching together of multiple region features, but rather the identification of image region feature groups belonging to the same spatial missing region. Image region feature groups reflect the continuity, clustering, and boundary relationships of the missing range in the image. Masked image region features sharing adjacent edges, having a center point distance less than a preset value, and located within the same page block can be grouped into the same masked image region feature group. After the masked image region feature group is formed, fragment merging is performed on the image region features within the group. The purpose of fragment merging is to transform the discrete region-level representation into a local fragment-level representation, so that the missing content is no longer just a collection of multiple single-point regions, but a continuous fragment with local structural attributes. The resulting multiple local missing fragments correspond to different spatial missing regions, and each local missing fragment retains the original position, relative order, and local continuity relationships of the regions within the group.

[0087] After multiple locally missing segments are formed, they need to be sequentially arranged. The purpose of sequential arrangement is to preserve the arrangement relationship between different spatially missing regions, so that the image features to be reconstructed include both the local continuity within segments and the global order relationship between segments. Sequential arrangement can be completed according to the original order of appearance in the image region position sequence, the reading order from left to right and then from top to bottom, the order of center point coordinates, or the order of region numbers. The multiple locally missing segments after sequential arrangement are then aggregated to obtain the image features to be reconstructed. The image features to be reconstructed are not a single vector, nor are they a random stack of segments, but rather an ordered set formed with missing segments as the basic unit. The image features to be reconstructed in this way clearly indicate which visual content is missing, which locations need to be restored later, and at the same time, preserve the distribution information of the missing parts in the whole image.

[0088] Preserved image region features remain visible after random masking, but the original arrangement is often disrupted, necessitating rearrangement based on the image region position sequence. This rearrangement results in a preserved feature sequence. The image region features in this sequence are arranged according to their original spatial order, maintaining the contextual relationships of the visible regions. However, the preserved feature sequence is not directly equivalent to the masked image feature set, as the masked image feature set also needs to preserve missing positions. Therefore, placeholder markers are written into the image region positions corresponding to the features to be reconstructed within the preserved feature sequence. These placeholder markers indicate missing positions, enabling accurate location of the areas to be filled during subsequent reconstruction without altering the relative order of the preserved image region features. Simultaneously, the image region position sequence corresponding to the features to be reconstructed is also preserved, ensuring a one-to-one correspondence between missing segments and their original spatial positions. The resulting masked image feature set consists of two parts: preserved image region features arranged in their original positional order, and placeholder markers written to the corresponding missing positions. The masked image feature set and the image features to be reconstructed work together. One provides the visible context and missing location index, while the other provides the target range and local structure description of the missing content. Together, they form the input basis for the subsequent recovery stage.

[0089] For example, the formula for a latent mask for image features is:

[0090] in, This represents the image features of the i-th image region after masking. This represents the mask variable corresponding to the i-th image region. Let represent the feature of the i-th original image region. Bernoulli(0.6) denotes the Bernoulli distribution, with 0.6 as the example retained probability.

[0091] This embodiment fixes the features of each image region into a unified spatial location system by using location index markers and image region location sequences, ensuring that the preserved and masked parts maintain a traceable spatial correspondence after segmentation. By combining random masking with spatial adjacency relationships to form continuous groups of masked image region features, and further merging them into multiple local missing segments, the missing content can be transformed from discrete regions into missing segments with local structural attributes, reducing the likelihood of subsequent restoration focusing only on single-point regions while ignoring regional continuity. By sequentially rearranging the preserved image region features and writing mask placeholders at corresponding positions, both visible context and missing location indices can be preserved in the masked image feature set. The resulting masked image feature set and the image features to be reconstructed retain the spatial distribution relationship of the original image feature set while explicitly separating visible and missing content, facilitating subsequent restoration stages to perform well-defined, clearly defined, and structurally continuous visual content reconstruction of the missing regions.

[0092] In one embodiment, such as Figure 5 As shown, step S40 above includes: S401, input the initial text embedding and the mask image feature set into the visual guidance branch; S402, based on the local missing segments corresponding to the features of the image to be reconstructed, the mask image feature set is grouped into segments to generate multiple missing segment query subsequences; S403, input the initial text embedding and the multiple missing fragment query subsequences into the cross-modal attention layer in the visual guidance branch; S404, In the cross-modal attention layer, the missing fragment query subsequence is subjected to query projection transformation to generate image query features, and the initial text embedding is subjected to key projection transformation and value projection transformation to generate text key features and text value features. S405, Based on the image query features and the text key features, perform scaled dot product attention processing to generate cross-modal attention weights corresponding to each local missing segment; S406, The text value features are weighted and aggregated based on the cross-modal attention weights corresponding to each locally missing segment to generate multiple segment recovery features; S407, based on the features of adjacent preserved image regions of each local missing segment in the mask image feature set, neighborhood constraint fusion is performed on multiple segment recovery features to generate multiple local structure recovery features; S408 performs cross-segment correlation and fusion of multiple local structural recovery features to generate global recovery features; S409, perform residual fusion of the global recovery features and the mask image feature set, and perform layer normalization processing to generate a reconstructed feature set.

[0093] In this embodiment, the initial text embedding carries the semantic representation of the encoded text sequence, the masked image feature set carries the visual representation of the preserved image region and the missing location markers, and the image features to be reconstructed carry the target visual content corresponding to the missing region. After receiving these three types of information, the visual guidance branch does not directly restore the entire region of the whole image uniformly, but organizes the restoration process around the locally missing fragments. The purpose of this setting is to elevate the restoration object from discrete locations to fragment-level objects, enabling the restoration unit to simultaneously perceive the local region content, the missing range, and the distribution relationship of adjacent regions.

[0094] Local missing segments are used to characterize the boundaries, lengths, and location distribution of consecutive missing regions. After grouping the masked image feature set according to local missing segments, multiple missing segment query subsequences are formed. These query subsequences are not simply extracted from several missing locations; instead, they organize the missing locations covered by the same local missing segment into a segment-level sequence according to their original spatial order, ensuring that each segment-level sequence carries spatial structural information of the consecutive regions. These multiple missing segment query subsequences correspond to different local missing regions, facilitating the subsequent cross-modal attention layer's segment-by-segment establishment of the correspondence between textual semantics and missing visual regions.

[0095] The cross-modal attention layer serves to transmit constraint information from textual semantics to visually missing regions. After the missing segment query subsequence enters the cross-modal attention layer, it first undergoes a query projection transformation to form image query features. The query projection transformation maps the segment-level missing region representation to the attention computation space, enabling each missing segment to initiate a retrieval based on textual semantics. The initial text embedding undergoes key projection transformation and value projection transformation within the same attention space, forming text key features and text value features. The text key features serve as a semantic matching index, used to calculate the cross-modal matching strength with the image query features; the text value features serve as a semantic content transfer function, used to inject the necessary textual semantic information into the missing segments after matching is completed.

[0096] Cross-modal attention weights are calculated between image query features and text key features using scaled dot product attention. Scaled dot product attention is not a simple similarity comparison, but rather a quantification of the correlation between each missing segment and all text locations within a unified dimensional space, making the constraint strength of different text locations on different missing segments distinguishable. After the cross-modal attention weights are formed, the text value features are weighted and aggregated according to these weights, outputting multiple segment recovery features. These segment recovery features are not pure text representations, nor are they original visual features, but rather cross-modal recovery representations adapted to each locally missing segment after attention selection. Each segment recovery feature corresponds to a locally missing segment, thus preserving the segment-level target location while incorporating object descriptions, attribute descriptions, and relationship descriptions from the text.

[0097] After the fragment restoration features are formed, it is necessary to introduce features from neighboring preserved image regions for neighborhood constraint fusion. This is because fragment restoration results relying solely on textual semantics are likely to satisfy semantic consistency, but may deviate from the surrounding known visual content in terms of texture continuity, boundary coherence, and regional style consistency. Features from neighboring preserved image regions provide visible contextual information about the missing fragment, including the visual distribution of neighboring regions, the continuation of local structure, and boundary transition relationships. Neighborhood constraint fusion integrates the fragment restoration features with the features from neighboring preserved image regions locally, ensuring that the generated local structural restoration features simultaneously satisfy both textual semantic constraints and local visual continuity constraints. The resulting local structural restoration features are no longer isolated fragments, but fragment-level restoration results that maintain local structural consistency with the surrounding visible regions.

[0098] Cross-segment correlation fusion is also required among multiple local structural restoration features. Although each missing segment is restored independently, there may still be object continuity, style consistency, and layout coordination relationships at the global image scale. If missing regions are restored separately without cross-segment coordination, problems such as inconsistent visual expression, global structural incoordination, or description shift of the same object across regions may occur between different segments. Cross-segment correlation fusion establishes a global relationship model for multiple local structural restoration features, enabling segments to share higher-level structural and semantic constraints. The resulting global restoration features not only reflect the local restoration results of each missing segment but also the overall coordination relationship of all missing segments in the entire image.

[0099] The global restored features and the masked image feature set need to undergo residual fusion and layer normalization to form the reconstructed feature set. Residual fusion allows the global restored features to retain the existing visual information of the visible areas while writing back the missing locations, avoiding the restoration result from overwriting the effective context. Layer normalization is used to control the numerical stability of the feature distribution after restoration, ensuring that the feature scales of different missing segments, different image regions, and different batches remain comparable. After the reconstructed feature set is formed, all image region locations regain a unified representation, with visible locations retaining their original visual context, and missing locations obtaining a restored representation determined by textual semantics, local neighborhood, and cross-segment coordination. The resulting set not only fills in the missing visual content but also maintains the consistency of region order, boundary transitions, and overall distribution.

[0100] For example, the formula for cross-modal attention weights is:

[0101] in, This represents the cross-modal attention weight of the i-th image region or missing segment to the k-th text location. This represents the visual side query projection matrix. This represents the visual side-key projection matrix. This represents the feature of the i-th mask image. This represents the hidden state of the text at the k-th text position. d represents the embedding dimension.

[0102] The formula for cross-modal reconstruction feature generation is:

[0103] in, This represents the visual features of the i-th image region or missing segment after restoration. This represents the feature of the i-th mask image. This represents the cross-modal attention weight of the i-th image region or missing segment to the k-th text location. This represents the visual lateral projection matrix. This represents the hidden state of the text at the k-th text position. N represents the number of text positions. LayerNorm() represents the layer normalization operation.

[0104] This embodiment organizes missing regions by local missing segments and introduces textual semantic constraints within the cross-modal attention layer using a segment-level query approach. This transforms missing region recovery from single-point imputation to segment-level recovery, enhancing the correspondence accuracy between missing regions and text content. By incorporating features from adjacent preserved image regions into neighborhood constraint fusion, the recovery result maintains local continuity with the surrounding visible regions. Cross-segment association fusion of multiple local structural recovery features mitigates the overall inconsistency caused by separate recovery of different missing segments. By forming a reconstructed feature set through residual fusion and layer normalization, the recovery result can be stably written back into the complete image region representation while preserving the visible region context, thereby improving the completeness of missing visual content recovery, local structural consistency, and global distribution coordination.

[0105] In one embodiment, such as Figure 6 As shown, step S50 above includes: S501, map each text embedding unit in the initial text embedding to each image region feature in the reconstructed feature set to a unified alignment space; S502, based on the pairwise combination relationship between each text embedding unit and each image region feature in the unified alignment space, the initial text embedding and the reconstructed feature set are spliced ​​across modally to generate multiple cross-modal combined features; S503, input the multiple cross-modal combined features into the gating unit to generate a gating response value; S504, perform cosine similarity determination processing on each text embedding unit in the initial text embedding and each image region feature in the reconstructed feature set to generate similarity components; S505, The gated response value and the similarity component are jointly calibrated to generate an initial set of association scores; S506, based on the arrangement order of the image region features in the reconstructed feature set, identify multiple image region feature groups that are spatially continuously distributed, and perform intra-segment consistency correction on multiple initial association scores corresponding to the same image region feature group to generate a segment correction association score set. S507, Based on the fragment correction association score set, identify low confidence association scores below a preset threshold, and suppress the low confidence association scores; S508: Based on the arrangement order of text embedding units and image region features corresponding to each association score after suppression processing, the association scores after suppression processing are matrix-arranged to generate an association score matrix.

[0106] In this embodiment, when the text embedding units in the initial text embedding and the image region features in the reconstructed feature set are in different representation spaces, direct comparison will be affected by differences in dimensional structure and distribution scale. A unified alignment space is used to constrain the two types of vectors to the same feature scale and the same semantic resolution. In implementation, text projection matrices and image projection matrices can be set separately to map text embedding units and image region features to aligned vectors of the same dimension. Alternatively, a normalization unit can be added after projection to keep the numerical range of the aligned vectors stable. This unified alignment space allows text position semantics, image region semantics, local attribute information, and cross-modal association strength to be measured together in the same computational space, reducing the interference of original distribution differences on subsequent association generation.

[0107] Pairwise combination relationships are used to transform text embedding units and image region features from individual representations into paired representations. Cross-modal concatenation is not a simple superposition, but rather combines the same text embedding unit with different image region features separately, so that each pair of text positions and image regions forms an independent candidate associated object. Multiple cross-modal combined features are thus formed, each of which simultaneously contains text side vector components and image side vector components. This combination method allows subsequent gating units to no longer rely solely on single-sided features, but to simultaneously see semantic content, visual content, and the differences between the two when they are juxtaposed in the same input. If the number of text embedding units is m and the number of image region features is n, then the combination yields m multiplied by n candidate corresponding objects, and the subsequently generated association results naturally possess the conditions for matrix organization.

[0108] The gating unit is responsible for estimating the pass intensity. After multiple cross-modal combined features are input into the gating unit, it provides a gating response value for each candidate corresponding object. The gating response value is not the final association result, but rather a measure of the degree to which the current text embedding unit and the current image region features can be preserved. The gating unit can adopt a structure of linear mapping plus nonlinear activation, or a multilayer perceptron structure, so that the output falls within a preset numerical range. A high gating response value indicates that the text semantics and image region content in the corresponding object have a high passability under the current task constraints; a low gating response value indicates that although the candidate corresponding object has been combined, its pass intensity is weak and needs to be suppressed in subsequent joint calibration. The gating response values ​​obtained in this way can differentiate and screen all candidate corresponding objects.

[0109] The cosine similarity determination process performs the function of measuring directional similarity. After the text embedding unit and image region features are mapped through a unified alignment space, the similarity of the vector angles can be calculated separately. The cosine similarity component reflects the consistency between the semantic direction of the text and the semantic direction of the image region; the higher the value, the closer the directions of the text embedding unit and the image region features are in the unified alignment space. Relying solely on the gated response value will bias the results towards the parameter learning results of the gated network, while relying solely on the cosine similarity component will bias the results towards the static vector angle relationship. Therefore, the gated response value and the similarity component need to work together. During joint calibration, multiplicative coupling, weighted coupling, or gated weighted multiplicative coupling can be used to combine the two into an initial set of association scores. Multiplicative coupling can amplify the association when both are high and suppress the association when one is low, so that the initial association score simultaneously reflects both the similarity direction and the strength of the connection.

[0110] Once the image region features in the reconstructed feature set are arranged in a predetermined order, they can reflect the continuous distribution of image regions in a plane. Identifying multiple spatially continuous image region feature groups based on their arrangement order can elevate the originally discrete region-level association results to segment-level association results. The identification of image region feature groups can be based on continuous changes in consecutive numbering, adjacency numbering, row and column adjacency relationships, or position indices. Multiple initial association scores within a group are not isolated single-point values, as they correspond to multiple image region features within the same spatially continuous region. Intra-segment consistency correction is used to smooth, balance, or constrain multiple initial association scores within a group, making the association distribution within the continuous region more stable. If a position within a group exhibits abnormally high or low values ​​due to local noise, intra-segment consistency correction can reduce abnormal fluctuations through neighborhood averaging, local weighted smoothing, or segment-level center correction, making the association structure within the segment more consistent with the overall state of the continuous region. The resulting segment-corrected association score set retains fine-grained differences while suppressing discrete jumps within the continuous region.

[0111] After the fragment correction association score set is formed, it is necessary to further identify low-confidence association scores. A preset threshold is used to distinguish between effective and weak associations. Low-confidence association scores below the preset threshold indicate insufficient matching strength between the current text embedding unit and the current image region features. If directly retained, this will introduce regional information with weak semantic relationship to the target into the subsequent convergence stage. Suppression processing is used to compress the influence range of these low-confidence association scores. Suppression methods can include zeroing, proportional attenuation, soft threshold compression, or segmented scaling. After suppression, the high-confidence portion of the association score set is retained, while the low-confidence portion is restricted to a lower numerical range, thereby improving the overall recognizability of the association results.

[0112] Matrix-based arrangement serves as the structured output function. After suppression processing, the association scores already possess numerical validity, but they remain discrete results arranged according to candidate correspondences. After matrix-based arrangement based on the order of text embedding units and image region features, the row direction is fixedly mapped to the text embedding units, and the column direction is fixedly mapped to the image region features, thus transforming the discrete association scores into an association score matrix. Each element in the matrix represents the association strength between a text location and an image region. The matrix not only records local correspondences but also preserves the association distribution pattern between the entire text and the entire image. The resulting association score matrix possesses location traceability, region distinguishability, and numerical computability, providing direct input for subsequent weighted aggregation based on association scores.

[0113] For example, the formula for the gated cross-modal correlation score is:

[0114] in, This represents the correlation score between the i-th text embedding unit and the j-th image region feature. () represents the Sigmoid function. This represents the gating weight matrix. This represents the result of concatenating text features and image features. This represents the i-th text embedding unit. This represents the feature of the j-th image region. This represents the cosine similarity.

[0115] This embodiment maps the initial text embedding and reconstructed feature sets to a unified alignment space and forms cross-modal combination features according to pairwise combinations, enabling fine comparison of text semantics and image region content at the same feature scale. By jointly calibrating the gating response value and similarity component, it can simultaneously reflect the passing strength and directional consistency of candidate corresponding objects, reducing bias caused by a single similarity measure or a single gating estimate. By correcting the intra-segment consistency of spatially continuous image region feature groups, it can reduce discrete jumps within continuous regions, making the local association distribution more stable. By identifying and suppressing low-confidence association scores below a preset threshold, it can reduce the interference of weakly associated regions on the overall association result. The association score matrix formed after matrix arrangement simultaneously preserves the text position order, image region order, and cross-modal correspondence strength, giving the subsequent text enhancement process based on the association result higher region discrimination ability and more stable correspondence constraints.

[0116] In one embodiment, such as Figure 7 As shown, step S60 above includes: S601, based on the association scores between each text embedding unit and each image region feature in the association score matrix, identify low-confidence association scores below a preset threshold; S602, perform attenuation correction on the low-confidence correlation score to generate a corrected correlation score matrix; S603, Based on the modified correlation score matrix, perform region-level weighted aggregation on each text embedding unit in the initial text embedding to generate aggregated text features; S604, The aggregated text features are fused with the initial text embedding to generate a visually enhanced text embedding; S605, based on the image-text correspondence in the image-text pairing sample set, determine a matching image feature subset and a non-matching image feature subset for the visually enhanced text embedding from the original image feature set; S606, Based on the similarity distribution between the visually enhanced text embedding and the mismatched image feature subset, perform hard negative sample screening on the mismatched image feature subset to generate a comparison sample set; S607, Based on the similarity difference between the visually enhanced text embedding and the matching image feature subset and the comparison sample set, determine the contrast loss; S608, Based on the regional correspondence between the original image feature set and the reconstructed feature set, determine the reconstruction consistency score, and determine the reconstruction loss based on the reconstruction consistency score; S609, the contrast loss and the reconstruction loss are weighted and fused to obtain the joint loss.

[0117] In this embodiment, each association score in the association score matrix reflects the strength of the correspondence between text embedding units and image region features. Low-confidence association scores below a preset threshold indicate insufficient reliability of the association between the current text location and the current image region. If directly involved in subsequent aggregation, these scores would introduce information from weakly correlated regions into the text representation, causing visual shifts and semantic disturbances. After low-confidence association scores are identified, they are not directly deleted from the association score matrix but are instead attenuated. The goal of attenuation correction is to preserve the overall matrix structure and positional correspondences while reducing the impact of weakly correlated locations on the aggregation results. Attenuation methods can include proportional reduction, smooth compression, or threshold mapping, as long as the corrected association scores maintain consistency with the original positional relationships between text embedding units and image region features. The resulting corrected association score matrix does not change the matrix's row and column organization but alters the weight distribution of each position, allowing high-confidence association positions to occupy a higher proportion in subsequent aggregation.

[0118] Region-level weighted convergence occurs between the corrected association score matrix and the initial text embedding. The rows in the matrix correspond to text embedding units, and the columns correspond to image region features. Therefore, each text embedding unit can obtain a set of corrected association weights corresponding to all image regions. Applying the corrected association weights corresponding to the same text embedding unit to the initial text embedding yields an aggregation result with region bias. This region-level weighted convergence is not a simple summation of image region features, but rather a reorganization of the contribution ratios of various positions within the text, leveraging the distribution of image region correlations. This allows the text representation to retain its original semantic order while absorbing region-level visual constraints. After the aggregated text features are formed, feature fusion with the initial text embedding is required. This is because the aggregated text features are biased towards cross-modal aligned region-oriented information, while the initial text embedding is biased towards original linguistic semantic information; using either alone will result in the loss of some effective content. Feature fusion can employ weighted superposition, gated fusion, or concatenation mapping to ensure that the output simultaneously retains both the original text semantics and image region correlation information, thus obtaining a visually enhanced text embedding. The visually enhanced text embedding is no longer a simple linguistic representation, but a composite representation that has absorbed image region distribution and cross-modal correspondences.

[0119] The image-text correspondence in the image-text pairing sample set is used to determine the matching and non-matching image feature subsets. The matching image feature subset comes from the original image feature set corresponding to the current visually enhanced text embedding, while the non-matching image feature subset comes from other original image feature sets in the same batch or within a preset candidate range. These two types of image feature subsets represent positive corresponding objects and negative contrast objects, respectively. Treating all non-matching image features as negative objects would result in a large number of easily distinguishable negative samples occupying the contrast learning capacity, preventing truly interfering negative samples from being highlighted. Therefore, it is also necessary to perform hard negative sample screening based on the similarity distribution between the visually enhanced text embedding and the non-matching image feature subsets. The core of hard negative sample screening is to retain candidate objects from the non-matching image feature subsets that have a high similarity to the current visually enhanced text embedding but whose image-text relationship does not hold. This resulting contrast sample set can more effectively constrain the boundary relationship between the visually enhanced text embedding and non-corresponding image features, making subsequent contrast loss focus more on easily confused regions rather than being diluted by a large number of low-relevance objects.

[0120] The contrastive loss is based on the similarity difference between the visually enhanced text embedding, the matching image feature subset, and the contrastive sample set. The visually enhanced text embedding should maintain high similarity with the matching image feature subset and low similarity with the contrastive sample set. A larger similarity difference indicates that the visually enhanced text embedding is better at distinguishing the semantics of correct and non-corresponding image regions; a smaller similarity difference indicates that the visually enhanced text embedding still suffers from cross-modal confusion. The role of the contrastive loss is to transform this difference into a backpropagable optimization objective, enabling the visually enhanced text embedding to continuously converge towards correct image-text pairing during subsequent training.

[0121] The reconstruction consistency score is based on the regional correspondence between the original image feature set and the reconstructed feature set. The original image feature set provides the target visual content before restoration, while the reconstructed feature set provides the restored visual content; there is a one-to-one correspondence between the two in terms of image region location. The reconstruction consistency score measures the degree of consistency between the same image regions before and after restoration. Small regional differences indicate that the restoration result preserves the target visual content well; large regional differences indicate that the restoration result deviates from the original visual distribution. The reconstruction loss is obtained by combining the consistency results of all regions. The reconstruction loss constrains the restoration quality, ensuring that the reconstructed feature set output by the visual guidance branch is not only text-guided but also as close as possible to the corresponding region content in the original image feature set.

[0122] The joint loss consists of contrast loss and reconstruction loss. Contrast loss constrains the cross-modal discriminative ability of the visually enhanced text embedding, while reconstruction loss constrains the region recovery ability of the reconstructed feature set. If only contrast loss is used, the text representation may be more biased towards matching and discrimination, neglecting the quality of image region recovery; if only reconstruction loss is used, the recovery result may be close to the original visual content, but the cross-modal discriminative boundary of the text representation is insufficient. By weightedly fusing the two types of losses, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set are incorporated into the same optimization objective, enabling text representation enhancement and image feature recovery to converge together in the same training direction.

[0123] For example, the relevance-weighted text embedding formula is:

[0124] in, This represents the i-th visually enhanced text embedding. This represents the correlation score between the i-th text embedding unit and the j-th image region feature. Let represent the i-th text embedding unit. M represents the number of image regions.

[0125] The formula for contrast loss is:

[0126] in, This indicates a comparative loss. This represents the i-th visually enhanced text embedding. This represents the image representation that matches the current visually enhanced text embedding. Let represent the k-th image in the batch. s(,) represents the similarity function. This represents the temperature coefficient. Indicates the batch size.

[0127] The reconstruction loss formula is:

[0128] in, This indicates the losses incurred during reconstruction. This represents the target image features in the original image feature set. This represents the recovered features in the reconstructed feature set. () represents a multilayer perceptron mapping. Represents square Norm.

[0129] The formula for joint loss is:

[0130] Where L represents the joint loss. This indicates a comparative loss. This indicates the losses incurred during reconstruction. This indicates the weight of the comparative loss. This indicates the weight of the reconstruction loss.

[0131] This embodiment reduces the interference of weakly correlated image regions on text representation enhancement by identifying and attenuating low-confidence association scores below a preset threshold, making the corrected association score matrix more accurately reflect the effective correspondence between text embedding units and image region features. By performing region-level weighted aggregation on the initial text embedding based on the corrected association score matrix and then fusing features with the initial text embedding, the visually enhanced text embedding can simultaneously retain the original semantic information of the text and the image region pointing information. By dividing the image feature subset into matching and non-matching subsets based on the image-text correspondence, and performing hard negative sample screening within the non-matching subset, the constraint strength of the contrast loss on easily confused objects can be improved. By determining the reconstruction consistency score and forming the reconstruction loss based on the region correspondence between the original image feature set and the reconstructed feature set, the consistency between the restored result and the target visual content can be constrained. After weighted fusion of the contrast loss and the reconstruction loss, the cross-modal discrimination capability of the text representation and the image region restoration quality can be improved simultaneously under the same objective, mitigating the problems of lack of visual pointing information in the text representation and the deviation of the restored result from the original visual content.

[0132] In one embodiment, such as Figure 8 As shown, step S70 above includes: S701, the joint loss is propagated in reverse along the text encoding branch and the visual guidance branch respectively to generate text branch update signal and visual branch update signal; S702, based on the text branch update signal, the update priority of multiple parameter subsets in the text encoding branch is arranged to generate a text parameter update sequence; S703, based on the visual branch update signal, the update priority of multiple parameter subsets in the visual guidance branch is arranged to generate a visual parameter update sequence; S704, according to the text parameter update sequence, multiple parameter subsets in the text encoding branch are written sequentially to generate the text encoding branch after the current round update, and according to the visual parameter update sequence, multiple parameter subsets in the visual guidance branch are written sequentially to generate the visual guidance branch after the current round update. S705, the joint loss change trajectory in multiple consecutive rounds is associated and recorded, and the parameter stability state of the text encoding branch and the visual guidance branch after the current round is updated is recorded, and a joint convergence determination sequence is generated. S706, determine whether the joint loss satisfies the convergence condition based on the joint convergence determination sequence; S707, when the joint loss satisfies the convergence condition, stop updating the parameters of the text encoding branch and the visual guidance branch, extract the updated text encoding branch from the text encoding branch after the current round of updates, and use the updated text encoding branch as the target text encoder.

[0133] In this embodiment, after the joint loss is formed, the text encoding branch and the visual guidance branch share the same optimization objective, but their parameter adjustment responsibilities differ. The text encoding branch is responsible for encoding the input text sequence into a text representation that can carry semantic constraints, while the visual guidance branch is responsible for introducing the text semantic constraints into the recovery process of the missing image region. Therefore, when the joint loss is propagated backward along the two branches, the resulting update signals are different in parameter distribution, amplitude range, and sensitivity level. The text branch update signal mainly acts on the trainable parameters in the word mapping layer, self-attention layer, feedforward transform layer, residual connection layer, and normalization layer, enabling the text representation to absorb image region constraint information while maintaining its original semantic discriminative ability. The visual branch update signal mainly acts on the trainable parameters in the cross-modal attention layer, fragment recovery layer, neighborhood fusion layer, normalization layer, and residual fusion layer, ensuring that the missing region recovery process continuously stays close to the optimization direction corresponding to the joint loss.

[0134] The division of multiple parameter subsets improves the controllability of parameter updates during training. The parameter subsets in the text encoding branch can be divided hierarchically into low-level lexical mapping parameters, mid-level semantic transformation parameters, and high-level context aggregation parameters, or functionally into attention projection parameters, feedforward mapping parameters, and normalization parameters. The parameter subsets in the visual guidance branch can be divided according to their recovery function into text guidance parameters, region fusion parameters, and recovery correction parameters. After the text branch update signal and the visual branch update signal are applied to their respective parameter subsets, they do not directly update all parameters as a whole. Instead, they prioritize the updates of multiple parameter subsets based on gradient norm, gradient variance, parameter sensitivity, hierarchical importance, or historical convergence speed. This results in text and visual parameter update sequences that place parameters requiring higher priority in the current round at the beginning, reducing mutual interference caused by synchronous adjustments of parameters at different levels.

[0135] Sequential writing transforms parameter updates from discrete gradient results into ordered parameter state changes. Once the text parameter update sequence is determined, parameters can be written sequentially to multiple parameter subsets of the text encoding branch. During the writing process, the optimizer state, gradient momentum state, and parameter changes in the current round are preserved, resulting in the updated text encoding branch for the current round. Similarly, once the visual parameter update sequence is determined, parameters can be written sequentially to multiple parameter subsets of the visual guidance branch, maintaining a consistent parameter update time window across the cross-modal attention layer, fusion layer, and normalization layer, ensuring numerical stability of the updated visual branch. Sequential writing is not simply layer-by-layer overwriting; rather, it ensures that the update order of different parameter subsets within the current round is controlled by the same priority rule. This results in the updated text encoding branch and the updated visual guidance branch for the current round reflecting both the synchronization constraints of the joint loss on the two branches and preserving the structural differences between the two branches.

[0136] The joint convergence decision sequence is used to describe the trajectory of the training state as it evolves over multiple epochs. The joint loss value in a single epoch is insufficient to support convergence judgment; it is also necessary to record the magnitude, direction, local fluctuation range, and parameter changes of the text encoding branch and the visual guidance branch over several consecutive epochs. The joint loss trajectory can be recorded using a sliding window method or an exponential smoothing method. Parameter stability can be measured by changes in parameter norm, gradient mean, gradient variance, or consistency of layer update magnitude. When the decrease in the joint loss continuously narrows over several consecutive epochs, and the parameter changes of both the text encoding branch and the visual guidance branch simultaneously fall into a stable range, the joint convergence decision sequence will exhibit a clear convergence pattern. This avoids misjudgments caused by relying solely on a single minimum loss value and reduces the impact of temporary fluctuations on training termination decisions.

[0137] Once the convergence condition is met, the parameters of the text encoding branch and the visual guidance branch are no longer updated. After parameter updates cease, the updated text encoding branch needs to be extracted from the text encoding branch updated in the current round and retained as the target text encoder. This retention fixes the already trained parameter states, the optimizer termination state, and the network configuration required for the inference phase. The target text encoder retains the text representation capability that has absorbed cross-modal constraint information; therefore, during subsequent inference, only the text sequence needs to be input to output a text representation with visual guidance capability, no longer relying on the visual guidance branch for inference. The visual guidance branch performs auxiliary recovery and constraint injection functions during the training phase and is no longer part of the inference path after the convergence condition is met.

[0138] The text encoding branch may include one token mapping layer, 6 to 12 self-attention encoding layers, and one output mapping layer. Each self-attention encoding layer includes a multi-head attention sublayer, a feedforward transform sublayer, a residual connection sublayer, and a normalization sublayer, with layers sequentially connected. The visual guidance branch may include one cross-modal query projection layer, 2 to 6 cross-modal attention layers, one neighborhood fusion layer, and one recovery output layer. The cross-modal attention layer is connected to the output of the text encoding branch, and the neighborhood fusion layer is sequentially connected to the recovery output layer. During the training of the joint loss algorithm, the batch size can be set to 16, 32, or 64; the text sequence length can be set to 64, 128, or 256; the number of image regions can be set to 16, 32, or 64; the learning rate can be set to 1e-5, 3e-5, 1e-4, or 3e-4; the optimizer can be Adam, AdamW, or a gradient descent optimizer with momentum; the gradient clipping threshold can be set to 1, 2, or 5; and the convergence decision window can be set to 3, 5, or 10 consecutive epochs. The priority of parameter subsets can be determined based on the gradient norm of the current epoch or the average gradient change over the most recent epochs.

[0139] This embodiment generates text branch update signals and visual branch update signals separately through joint loss, and prioritizes and sequentially writes updates to multiple parameter subsets. This coordinates the parameter adjustment process of the text encoding branch and the visual guidance branch under the same optimization objective, mitigating numerical disturbances caused by simultaneous updates of parameters at different levels. By continuously recording the joint loss change trajectory and the stable state of the dual-branch parameters to form a joint convergence judgment sequence, the reliability of convergence judgment is improved, and termination misjudgments caused by instantaneous fluctuations are reduced. By stopping the dual-branch parameter update when the convergence condition is met and extracting the updated text encoding branch from the text encoding branch after the current round of updates, the text representation capability that has absorbed cross-modal constraint information can be preserved, enabling the target text encoder to output a representation result with visual guidance capability even when only receiving text input during the inference phase.

[0140] In one embodiment, a visually enhanced text representation optimization apparatus is provided, which corresponds one-to-one with the visually enhanced text representation optimization method described in the above embodiments. (Refer to...) Figure 9 , Figure 9 This is a schematic diagram of the functional modules of a preferred embodiment of the visually enhanced text representation optimization device of the present invention. The modules include a multimodal data construction module 10, a text encoding module 20, a mask generation module 30, a cross-modal reconstruction module 40, an alignment modeling module 50, a joint optimization construction module 60, a model update module 70, and a task inference module 80. Detailed descriptions of each functional module are as follows: The multimodal data construction module 10 is used to acquire a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and position encoding operations to generate an input text sequence, and process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set. The text encoding module 20 is used to input the input text sequence into the text encoding branch, process the input text sequence through a self-attention mechanism, and generate a hidden state sequence as the initial text embedding. The mask generation module 30 is used to process the original image feature set through latent masking operations to generate a masked image feature set and image features to be reconstructed. The cross-modal reconstruction module 40 is used to input the initial text embedding and the mask image feature set into the visual guidance branch, recover the features of the image to be reconstructed, and generate a reconstructed feature set. Alignment modeling module 50 is used to perform gated alignment based on the initial text embedding and the reconstructed feature set to generate an association score matrix; The joint optimization construction module 60 is used to weight and converge the initial text embedding based on the association score matrix to generate visually enhanced text embedding, and to determine the joint loss based on the image-text pairing sample set, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set; Model update module 70 is used to update the text encoding branch and the visual guidance branch based on the joint loss, and retain the updated text encoding branch as the target text encoder when the joint loss satisfies the convergence condition. The task reasoning module 80 is used to process the target task based on the target text encoder and obtain the target task result.

[0141] In one embodiment, the multimodal data construction module 10 is specifically used for: Text content and image content are extracted from multiple sets of candidate image and text samples, and the text content and image content are paired and organized according to the image and text correspondence identifier to generate an image and text pairing sample set; The paired text in the image-text pairing sample set is segmented into multiple consecutive text words, and a word word sequence is generated according to the order of the multiple text words. The word sequence is positionally encoded by writing each text word in the word sequence and its corresponding position representation into the same sequence structure to generate the input text sequence. The paired images in the image-text pairing sample set are divided into regions to obtain multiple image region units, and the multiple image region units are input into a visual model for region-level feature extraction to obtain multiple region features; The multiple regional features are arranged according to their positional order in the paired image, and the arranged regional features are combined to obtain the original image feature set.

[0142] In one embodiment, the mask generation module 30 is specifically used for: The location index is marked for each image region feature in the original image feature set to generate an image region location sequence, and the spatial adjacency relationship between each image region feature is established based on the image region location sequence; Based on a preset retention probability, the features of each image region in the original image feature set are randomly masked and screened to determine the retained image region features and the masked image region features. Based on the spatial adjacency relationship, the features of multiple shielded image regions that are continuously distributed in space are combined into segments to identify the shielded image region feature groups corresponding to different spatial missing regions. For each masked image region feature group, fragment merging is performed to generate multiple local missing fragments corresponding to different spatial missing regions; Multiple locally missing segments are sequentially arranged, and the sequentially arranged local missing segments are aggregated to generate the features of the image to be reconstructed; The preserved image region features are rearranged sequentially according to the image region position sequence to generate a preserved feature sequence; A mask placeholder is written into the image region position corresponding to the image feature to be reconstructed in the preserved feature sequence, and the image region position sequence corresponding to the image feature to be reconstructed is preserved to generate a mask image feature set.

[0143] In one embodiment, the cross-modal reconstruction module 40 is specifically used for: The initial text embedding and the mask image feature set are input into the visual guidance branch; Based on the local missing segments corresponding to the features of the image to be reconstructed, the mask image feature set is grouped into segments to generate multiple missing segment query subsequences; The initial text embedding and the multiple missing fragment query subsequences are input into the cross-modal attention layer in the visual guidance branch; In the cross-modal attention layer, query projection transformation is performed on the missing fragment query subsequence to generate image query features, and key projection transformation and value projection transformation are performed on the initial text embedding to generate text key features and text value features. Based on the image query features and the text key features, scaled dot product attention processing is performed to generate cross-modal attention weights corresponding to each local missing segment; The text value features are weighted and aggregated based on the cross-modal attention weights corresponding to each locally missing segment to generate multiple segment recovery features; Based on the features of adjacent preserved image regions of each local missing segment in the masked image feature set, neighborhood constraint fusion is performed on multiple segment recovery features to generate multiple local structure recovery features. Multiple local structural recovery features are correlated and fused across segments to generate global recovery features; The global recovery features are residually fused with the mask image feature set, and then layer normalization is performed to generate a reconstructed feature set.

[0144] In one embodiment, the alignment modeling module 50 is specifically used for: Map each text embedding unit in the initial text embedding to each image region feature in the reconstructed feature set to a unified alignment space; Based on the pairwise combination relationship between each text embedding unit and each image region feature in the unified alignment space, the initial text embedding and the reconstructed feature set are cross-modal spliced ​​to generate multiple cross-modal combined features; The multiple cross-modal combined features are input into the gating unit to generate a gating response value; Cosine similarity determination is performed on each text embedding unit in the initial text embedding and each image region feature in the reconstructed feature set to generate similarity components; The gated response value and the similarity component are jointly calibrated to generate an initial set of association scores. Based on the arrangement order of the image region features in the reconstructed feature set, multiple image region feature groups that are spatially continuous are identified, and intra-segment consistency correction is performed on multiple initial association scores corresponding to the same image region feature group to generate a segment-corrected association score set. Based on the fragment correction association score set, low-confidence association scores below a preset threshold are identified, and the low-confidence association scores are suppressed. Based on the arrangement order of text embedding units and image region features corresponding to each association score after suppression processing, the association scores after suppression processing are matrix-arranged to generate an association score matrix.

[0145] In one embodiment, the joint optimization construction module 60 is specifically used for: Based on the association scores between each text embedding unit and each image region feature in the association score matrix, low-confidence association scores below a preset threshold are identified. The low-confidence association scores are attenuated and corrected to generate a corrected association score matrix; Based on the modified correlation score matrix, the text embedding units in the initial text embedding are weighted and aggregated at the region level to generate aggregated text features; The aggregated text features are fused with the initial text embedding to generate a visually enhanced text embedding. Based on the image-text correspondence in the image-text pairing sample set, a subset of matching image features and a subset of non-matching image features are determined from the original image feature set for the visually enhanced text embedding; Based on the similarity distribution between the visually enhanced text embedding and the mismatched image feature subset, a hard negative sample screening is performed on the mismatched image feature subset to generate a comparison sample set. The contrast loss is determined based on the similarity difference between the visually enhanced text embedding, the matching image feature subset, and the contrast sample set. Based on the regional correspondence between the original image feature set and the reconstructed feature set, a reconstruction consistency score is determined, and a reconstruction loss is determined based on the reconstruction consistency score. The comparison loss and the reconstruction loss are weighted and fused to obtain the joint loss.

[0146] In one embodiment, the model update module 70 is specifically used for: The joint loss is propagated in reverse along the text encoding branch and the visual guidance branch respectively to generate text branch update signal and visual branch update signal; Based on the text branch update signal, the update priority of multiple parameter subsets in the text encoding branch is arranged to generate a text parameter update sequence; Based on the visual branch update signal, the update priority of multiple parameter subsets in the visual guidance branch is arranged to generate a visual parameter update sequence; Based on the text parameter update sequence, multiple parameter subsets in the text encoding branch are written sequentially to generate the text encoding branch updated in the current round. Similarly, based on the visual parameter update sequence, multiple parameter subsets in the visual guidance branch are written sequentially to generate the visual guidance branch updated in the current round. The joint loss change trajectory in multiple consecutive rounds is correlated and recorded, and the parameter stability state of the text encoding branch and the visual guidance branch after the current round is recorded to generate a joint convergence determination sequence. Determine whether the joint loss satisfies the convergence condition based on the joint convergence determination sequence; When the joint loss meets the convergence condition, stop updating the parameters of the text encoding branch and the visual guidance branch, extract the updated text encoding branch from the text encoding branch after the current round, and use the updated text encoding branch as the target text encoder.

[0147] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements server-side functions or steps of a visually enhanced text representation optimization method.

[0148] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a visually enhanced text representation optimization method.

[0149] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and positional encoding operations to generate an input text sequence, process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set; The input text sequence is input into the text encoding branch, and the input text sequence is processed by a self-attention mechanism to generate a hidden state sequence as the initial text embedding. The original image feature set is processed by latent masking operations to generate a masked image feature set and the image features to be reconstructed. The initial text embedding and the mask image feature set are input into the visual guidance branch to recover the features of the image to be reconstructed and generate a reconstructed feature set. Based on the initial text embedding and the reconstructed feature set, gated alignment is performed to generate an association score matrix; The initial text embeddings are weighted and aggregated based on the association score matrix to generate visually enhanced text embeddings. The joint loss is determined based on the image-text pairing sample set, the visually enhanced text embeddings, the original image feature set, and the reconstructed feature set. The text encoding branch and the visual guidance branch are updated based on the joint loss. When the joint loss satisfies the convergence condition, the updated text encoding branch is retained as the target text encoder. The target task is processed based on the target text encoder to obtain the target task result.

[0150] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Obtain a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and positional encoding operations to generate an input text sequence, process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set; The input text sequence is input into the text encoding branch, and the input text sequence is processed by a self-attention mechanism to generate a hidden state sequence as the initial text embedding. The original image feature set is processed by latent masking operations to generate a masked image feature set and the image features to be reconstructed. The initial text embedding and the mask image feature set are input into the visual guidance branch to recover the features of the image to be reconstructed and generate a reconstructed feature set. Based on the initial text embedding and the reconstructed feature set, gated alignment is performed to generate an association score matrix; The initial text embeddings are weighted and aggregated based on the association score matrix to generate visually enhanced text embeddings. The joint loss is determined based on the image-text pairing sample set, the visually enhanced text embeddings, the original image feature set, and the reconstructed feature set. The text encoding branch and the visual guidance branch are updated based on the joint loss. When the joint loss satisfies the convergence condition, the updated text encoding branch is retained as the target text encoder. The target task is processed based on the target text encoder to obtain the target task result.

[0151] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0153] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0154] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0155] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A text representation optimization method based on visual enhancement, characterized in that, Includes the following steps: Obtain a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and positional encoding operations to generate an input text sequence, process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set; The input text sequence is input into the text encoding branch, and the input text sequence is processed by a self-attention mechanism to generate a hidden state sequence as the initial text embedding. The original image feature set is processed by latent masking operations to generate a masked image feature set and the image features to be reconstructed. The initial text embedding and the mask image feature set are input into the visual guidance branch to recover the features of the image to be reconstructed and generate a reconstructed feature set. Based on the initial text embedding and the reconstructed feature set, gated alignment is performed to generate an association score matrix; The initial text embeddings are weighted and aggregated based on the association score matrix to generate visually enhanced text embeddings. The joint loss is determined based on the image-text pairing sample set, the visually enhanced text embeddings, the original image feature set, and the reconstructed feature set. The text encoding branch and the visual guidance branch are updated based on the joint loss. When the joint loss satisfies the convergence condition, the updated text encoding branch is retained as the target text encoder. The target task is processed based on the target text encoder to obtain the target task result.

2. The text representation optimization method based on visual enhancement as described in claim 1, characterized in that, Obtain a set of image-text pairing samples, process the paired text in the set through word segmentation and positional encoding to generate an input text sequence, and process the paired images in the set through a visual model to generate an original image feature set, including: Text content and image content are extracted from multiple sets of candidate image and text samples, and the text content and image content are paired and organized according to the image and text correspondence identifier to generate an image and text pairing sample set; The paired text in the image-text pairing sample set is segmented into multiple consecutive text words, and a word word sequence is generated according to the order of the multiple text words. The word sequence is positionally encoded by writing each text word in the word sequence and its corresponding position representation into the same sequence structure to generate the input text sequence. The paired images in the image-text pairing sample set are divided into regions to obtain multiple image region units, and the multiple image region units are input into a visual model for region-level feature extraction to obtain multiple region features; The multiple regional features are arranged according to their positional order in the paired image, and the arranged regional features are combined to obtain the original image feature set.

3. The text representation optimization method based on visual enhancement as described in claim 1, characterized in that, The original image feature set is processed through a latent masking operation to generate a masked image feature set and image features to be reconstructed, including: The location index is marked for each image region feature in the original image feature set to generate an image region location sequence, and the spatial adjacency relationship between each image region feature is established based on the image region location sequence; Based on a preset retention probability, the features of each image region in the original image feature set are randomly masked and screened to determine the retained image region features and the masked image region features. Based on the spatial adjacency relationship, the features of multiple shielded image regions that are continuously distributed in space are combined into segments to identify the shielded image region feature groups corresponding to different spatial missing regions. For each masked image region feature group, fragment merging is performed to generate multiple local missing fragments corresponding to different spatial missing regions; Multiple locally missing segments are sequentially arranged, and the sequentially arranged local missing segments are aggregated to generate the features of the image to be reconstructed; The preserved image region features are rearranged sequentially according to the image region position sequence to generate a preserved feature sequence; A mask placeholder is written into the image region position corresponding to the image feature to be reconstructed in the preserved feature sequence, and the image region position sequence corresponding to the image feature to be reconstructed is preserved to generate a mask image feature set.

4. The text representation optimization method based on visual enhancement as described in claim 1, characterized in that, The initial text embedding and the masked image feature set are input into the visual guidance branch to recover the features of the image to be reconstructed, generating a reconstructed feature set, including: The initial text embedding and the mask image feature set are input into the visual guidance branch; Based on the local missing segments corresponding to the features of the image to be reconstructed, the mask image feature set is grouped into segments to generate multiple missing segment query subsequences; The initial text embedding and the multiple missing fragment query subsequences are input into the cross-modal attention layer in the visual guidance branch; In the cross-modal attention layer, query projection transformation is performed on the missing fragment query subsequence to generate image query features, and key projection transformation and value projection transformation are performed on the initial text embedding to generate text key features and text value features. Based on the image query features and the text key features, scaled dot product attention processing is performed to generate cross-modal attention weights corresponding to each local missing segment; The text value features are weighted and aggregated based on the cross-modal attention weights corresponding to each locally missing segment to generate multiple segment recovery features; Based on the features of adjacent preserved image regions of each local missing segment in the masked image feature set, neighborhood constraint fusion is performed on multiple segment recovery features to generate multiple local structure recovery features. Multiple local structural recovery features are correlated and fused across segments to generate global recovery features; The global recovery features are residually fused with the mask image feature set, and then layer normalization is performed to generate a reconstructed feature set.

5. The text representation optimization method based on visual enhancement as described in claim 1, characterized in that, Gated alignment is performed based on the initial text embedding and the reconstructed feature set to generate an association score matrix, including: Map each text embedding unit in the initial text embedding to each image region feature in the reconstructed feature set to a unified alignment space; Based on the pairwise combination relationship between each text embedding unit and each image region feature in the unified alignment space, the initial text embedding and the reconstructed feature set are cross-modal spliced ​​to generate multiple cross-modal combined features; The multiple cross-modal combined features are input into the gating unit to generate a gating response value; Cosine similarity determination is performed on each text embedding unit in the initial text embedding and each image region feature in the reconstructed feature set to generate similarity components; The gated response value and the similarity component are jointly calibrated to generate an initial set of association scores. Based on the arrangement order of the image region features in the reconstructed feature set, multiple image region feature groups that are spatially continuous are identified, and intra-segment consistency correction is performed on multiple initial association scores corresponding to the same image region feature group to generate a segment-corrected association score set. Based on the fragment correction association score set, low-confidence association scores below a preset threshold are identified, and the low-confidence association scores are suppressed. Based on the arrangement order of text embedding units and image region features corresponding to each association score after suppression processing, the association scores after suppression processing are matrix-arranged to generate an association score matrix.

6. The text representation optimization method based on visual enhancement as described in claim 1, characterized in that, The initial text embeddings are weighted and aggregated based on the association score matrix to generate visually enhanced text embeddings. A joint loss is determined based on the image-text pairing sample set, the visually enhanced text embeddings, the original image feature set, and the reconstructed feature set, including: Based on the association scores between each text embedding unit and each image region feature in the association score matrix, low-confidence association scores below a preset threshold are identified. The low-confidence association scores are attenuated and corrected to generate a corrected association score matrix; Based on the modified correlation score matrix, the text embedding units in the initial text embedding are weighted and aggregated at the region level to generate aggregated text features; The aggregated text features are fused with the initial text embedding to generate a visually enhanced text embedding. Based on the image-text correspondence in the image-text pairing sample set, a subset of matching image features and a subset of non-matching image features are determined from the original image feature set for the visually enhanced text embedding; Based on the similarity distribution between the visually enhanced text embedding and the mismatched image feature subset, a hard negative sample screening is performed on the mismatched image feature subset to generate a comparison sample set. The contrast loss is determined based on the similarity difference between the visually enhanced text embedding, the matching image feature subset, and the contrast sample set. Based on the regional correspondence between the original image feature set and the reconstructed feature set, a reconstruction consistency score is determined, and a reconstruction loss is determined based on the reconstruction consistency score. The comparison loss and the reconstruction loss are weighted and fused to obtain the joint loss.

7. The text representation optimization method based on visual enhancement as described in claim 1, characterized in that, The text encoding branch and the visual guidance branch are updated based on the joint loss. When the joint loss satisfies the convergence condition, the updated text encoding branch is retained as the target text encoder, including: The joint loss is propagated in reverse along the text encoding branch and the visual guidance branch respectively to generate text branch update signal and visual branch update signal; Based on the text branch update signal, the update priority of multiple parameter subsets in the text encoding branch is arranged to generate a text parameter update sequence; Based on the visual branch update signal, the update priority of multiple parameter subsets in the visual guidance branch is arranged to generate a visual parameter update sequence; Based on the text parameter update sequence, multiple parameter subsets in the text encoding branch are written sequentially to generate the text encoding branch updated in the current round. Similarly, based on the visual parameter update sequence, multiple parameter subsets in the visual guidance branch are written sequentially to generate the visual guidance branch updated in the current round. The joint loss change trajectory in multiple consecutive rounds is correlated and recorded, and the parameter stability state of the text encoding branch and the visual guidance branch after the current round is recorded to generate a joint convergence determination sequence. Determine whether the joint loss satisfies the convergence condition based on the joint convergence determination sequence; When the joint loss meets the convergence condition, stop updating the parameters of the text encoding branch and the visual guidance branch, extract the updated text encoding branch from the text encoding branch after the current round, and use the updated text encoding branch as the target text encoder.

8. A text representation optimization device based on visual enhancement, characterized in that, The visually enhanced text representation optimization device includes: The multimodal data construction module is used to acquire a set of image-text pairing samples, process the paired text in the set of image-text pairing samples through word segmentation and position encoding operations to generate an input text sequence, and process the paired images in the set of image-text pairing samples through a visual model to generate an original image feature set. The text encoding module is used to input the input text sequence into the text encoding branch, process the input text sequence through a self-attention mechanism, and generate a hidden state sequence as the initial text embedding; The mask generation module is used to process the original image feature set through latent masking operations to generate a masked image feature set and image features to be reconstructed. The cross-modal reconstruction module is used to input the initial text embedding and the mask image feature set into the visual guidance branch, recover the features of the image to be reconstructed, and generate a reconstructed feature set. The alignment modeling module is used to perform gated alignment based on the initial text embedding and the reconstructed feature set to generate an association score matrix. The joint optimization construction module is used to weight and converge the initial text embedding based on the association score matrix to generate visually enhanced text embedding, and to determine the joint loss based on the image-text pairing sample set, the visually enhanced text embedding, the original image feature set, and the reconstructed feature set; The model update module is used to update the text encoding branch and the visual guidance branch based on the joint loss, and retain the updated text encoding branch as the target text encoder when the joint loss satisfies the convergence condition. The task reasoning module is used to process the target task based on the target text encoder and obtain the target task result.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a visually enhanced text representation optimization program stored in the memory and executable on the processor, wherein the visually enhanced text representation optimization program, when executed by the processor, implements the steps of the visually enhanced text representation optimization method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a visually enhanced text representation optimization program, which, when executed by a processor, implements the steps of the visually enhanced text representation optimization method as described in any one of claims 1-7.

Images