Text-guided visual data reconstruction method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, can be applied to business system platforms of medical health, financial science and technology and the like, and discloses a text-guided visual data reconstruction method, device and equipment and a medium. Respectively coding the text description data and the visual data into a text embedding vector and a visual embedding vector; calculating an attention matrix according to the text embedding vector and the visual embedding vector; performing importance evaluation on each visual token in the visual embedding vector according to the attention matrix to obtain a visual token importance score; and screening effective text embedding vectors in the text embedding vectors according to the similarity between a preset threshold value and the text embedding vectors and the visual embedding vectors. Through redundancy trimming, recovery and reconstruction of the visual data, directional deletion of redundant data in the visual data and retention of core visual data, the completeness of the visual data is balanced, and the efficiency of subsequent calculation of the visual data is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a text-guided visual data reconstruction method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, visual language models have become an important research direction in the field of multimodal artificial intelligence. With their ability to deeply integrate and understand text-visual interaction, visual language models have demonstrated powerful performance in core tasks such as image description, visual question answering, and cross-modal retrieval. Moreover, the application scenarios of visual language models are gradually extending to multiple fields.

[0003] For example, in the healthcare field, visual language models can combine medical images and diagnostic text reports to assist doctors in identifying symptoms and interpreting images; in the financial risk control process, visual language models can simultaneously analyze facial images and document text information for user identity authentication to accurately identify risks such as document forgery and mismatch between face and document.

[0004] When processing a high-resolution image or a video, most existing visual language models reduce the number of generated visual tokens at the source by designing smaller visual encoders or more efficient projection modules. However, this usually requires large-scale retraining of the visual language model, which is costly and may result in a loss of the general performance of the visual language model.

[0005] Therefore, how to intelligently and dynamically reduce the number of visual tokens that need to be processed based on the specific problem without increasing additional training costs, thereby improving the running efficiency of visual language models, has become an urgent problem to be solved. Summary of the Invention

[0006] This invention provides a text-guided visual data reconstruction method, apparatus, device, and medium to address the technical problem of how to intelligently and dynamically reduce the number of visual tokens that need to be processed based on specific problems without increasing additional training costs, thereby improving the operating efficiency of visual language models.

[0007] Firstly, a text-guided visual data reconstruction method is provided, including: Obtain text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors respectively; Calculate the attention matrix based on the text embedding vector and the visual embedding vector; The importance of each visual token in the visual embedding vector is evaluated based on the attention matrix to obtain a visual token importance score. Valid text embedding vectors are selected from the text embedding vectors based on the similarity between the text embedding vector and the visual embedding vector according to a preset threshold. The visual data is redundancy-trimmed based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The pruned visual data is recycled and reconstructed based on the visual token importance score to obtain reconstructed visual data.

[0008] Secondly, a text-guided visual data reconstruction device is provided, comprising: The encoding module is used to acquire text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors, respectively. The calculation module is used to calculate the attention matrix based on the text embedding vector and the visual embedding vector; An evaluation module is used to evaluate the importance of each visual token in the visual embedding vector based on the attention matrix, and obtain a visual token importance score. The filtering module is used to filter valid text embedding vectors from the text embedding vectors based on a preset threshold and the similarity between the text embedding vector and the visual embedding vector; A trimming module is used to perform redundancy trimming on the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The reconstruction module is used to reclaim and reconstruct the pruned visual data based on the visual token importance score to obtain reconstructed visual data.

[0009] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described text-guided visual data reconstruction method.

[0010] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described text-guided visual data reconstruction method.

[0011] In the aforementioned text-guided visual data reconstruction method, apparatus, computer equipment, and storage medium, data is acquired and encoded into text / visual embedding vectors. Unstructured text, images, and videos are transformed into computer-computable numerical vectors. Simultaneously, core data features are extracted to provide a standardized data foundation for subsequent cross-modal association analysis. The correlation strength between text and visual tokens is quantified to form an intuitive association matrix, providing data basis for judging the importance of visual tokens and filtering effective text. Visual token importance scores assign a value quantification index to each visual token, distinguishing between high-value and low-value tokens. This provides a precise basis for subsequent redundant pruning, avoiding the blind deletion of key visual information. Visual data is pruned based on effective text and an attention matrix, selectively deleting redundant tokens from the visual data. This reduces the data volume and computational burden without losing core information. Valuable information is recovered from the tokens to be pruned, avoiding information waste from redundant tokens and further compressing the data volume, balancing visual data integrity and computational efficiency. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a text-guided visual data reconstruction method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a text-guided visual data reconstruction method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a text-guided visual data reconstruction device according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to one embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] The present invention provides a text-guided visual data reconstruction method, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain text description data and visual data input by the user, and encode the text description data and visual data into text embedding vectors and visual embedding vectors, respectively; calculate an attention matrix based on the text embedding vectors and visual embedding vectors; evaluate the importance of each visual token in the visual embedding vectors based on the attention matrix to obtain a visual token importance score; filter valid text embedding vectors from the text embedding vectors based on a preset threshold and similarity to the text embedding vectors and visual embedding vectors; perform redundancy trimming on the visual data based on the valid text embedding vectors and the attention matrix to obtain trimmed visual data; reconstruct the trimmed visual data based on the visual token importance score to obtain reconstructed visual data, and feed the reconstructed visual data back to the client. This invention provides a text-guided visual data reconstruction device that, for visual data reconstruction services, selectively deletes redundant data in visual data and retains core visual data by redundancy trimming and reconstructing visual data, balancing the integrity of visual data while improving the efficiency of subsequent visual data calculation. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0016] Please see Figure 2 As shown, Figure 2 A flowchart illustrating a text-guided visual data reconstruction method provided in this embodiment of the invention includes the following steps: S1. Obtain the text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vector and visual embedding vector respectively.

[0017] In this embodiment of the invention, the visual data refers to content containing visual information presented in the form of images, videos, etc., and the encoding refers to the process of converting the original text description data and visual data into numerical vectors that can be recognized and calculated by a computer through a pre-trained algorithm model.

[0018] Specifically, the system acquires text descriptions (text description data) and images / videos (visual data) provided by the user, and then uses a model (text / visual encoding model) to transform the two types of data into numerical vectors with core features (text embedding vector and visual embedding vector), laying the foundation for subsequent cross-modal analysis (such as the correlation calculation between text and vision).

[0019] For example, when processing a user input image containing a red bus and a blue road sign using a visual language model, and asking the question, "What does the blue road sign say?", the image containing a red bus and a blue road sign can be considered visual data, while the question "What does the blue road sign say?" can be considered textual descriptive data.

[0020] In medical settings, textual descriptions of symptoms are transformed into text embedding vectors containing semantic features, and CT images are transformed into visual embedding vectors containing lesion features, in preparation for subsequent "symptom-image correlation diagnosis" (such as matching image lesions with textual symptoms).

[0021] In fintech scenarios, transaction notes are transformed into text embedding vectors containing transaction intent, and facial photos are transformed into visual embedding vectors containing facial features, laying a solid foundation for subsequent "user identity verification (matching facial images with ID card text information) and transaction risk identification (associating transaction descriptions with monitoring scenarios)".

[0022] In this embodiment of the invention, encoding the text description data into a text embedding vector includes: The text description data is cleaned to obtain standard text description data; Semantic features are extracted from the standard text description data to obtain text semantic features; The text semantic features are mapped to a preset dimension numerical space to obtain a text embedding vector.

[0023] In this embodiment of the invention, semantic feature extraction refers to the process of mining meaningful core information behind the text from the normalized text data, and mapping refers to the process of assigning corresponding values ​​to each semantic feature and then combining these values ​​into an ordered vector according to a fixed dimension.

[0024] Specifically, data cleaning of text description data involves steps such as deleting redundant information, correcting format and syntax, and segmenting and standardizing the text. The specific operations include deleting meaningless content in the text, such as special symbols (#, @), repetitive expressions, and irrelevant modifiers; standardizing the text format; correcting typos; supplementing incomplete semantics; splitting long texts into short sentences or vocabulary units according to semantic logic; and standardizing the text length. The final result is a standard text sequence with a uniform format and clear semantics, which is the standard text description data.

[0025] Furthermore, semantic features are extracted from the standard text description data. First, the standard text sequence after being split is input into a pre-trained language model. The Transformer layer inside the language model processes the input sequence and calculates the relationship between each word and all other words in the sequence through a self-attention mechanism, thereby generating a vector representation containing contextual information for each word. The language model outputs a vector sequence corresponding to the input sequence, where each vector represents the rich semantics of the corresponding word in a specific context. This vector sequence represents the extracted text semantic features.

[0026] Furthermore, the text semantic features are mapped to a preset numerical space. First, the preset numerical space dimensions (e.g., 512 dimensions) and the semantic meaning of each dimension are defined. A corresponding numerical value is assigned to each feature in the text semantic feature set. If the text semantic features do not cover all dimensions of the preset dimensions (e.g., only 80 dimensions out of 512 are used), default values ​​(e.g., 0) are filled in the unused dimensions to ensure that the vector dimensions are consistent with the preset dimensions. The values ​​of all dimensions are arranged in order to form an ordered numerical sequence, i.e., the text embedding vector. The steps of encoding visual data into visual embedding vectors are the same as those of encoding text description data into text embedding vectors, and will not be elaborated here.

[0027] S2. Calculate the attention matrix based on the text embedding vector and the visual embedding vector.

[0028] In this embodiment of the invention, calculation refers to analyzing the degree of association between each text embedding vector and all visual embedding vectors one by one through preset rules (such as measuring vector direction similarity and semantic matching degree), and arranging all text-visual vector association degrees according to the rule of text vectors as rows and visual vectors as columns, and transforming them into an ordered numerical matrix.

[0029] Furthermore, by utilizing the generated text embedding vectors (containing text semantic features) and visual embedding vectors (containing visual image features), an attention matrix that reflects the strength of the correspondence between text and visual information is obtained through association analysis and matrix integration.

[0030] In healthcare scenarios, for example, if the text vector of "right lower lobe shadow" in the matrix corresponds to a high value with the visual vector of "specific area of ​​right lower lobe" in the CT image, it can help doctors quickly match medical record descriptions with the location of lesions in the images, thereby improving diagnostic efficiency.

[0031] In fintech scenarios, for example, if the text vector for "out-of-town fraud" and the visual vector for "non-user operation" in surveillance video have high values, the system can help mark the transaction as high-risk.

[0032] In this embodiment of the invention, calculating the attention matrix based on the text embedding vector and the visual embedding vector includes: The text embedding vector and the visual embedding vector are dimensionally aligned to obtain aligned text vectors and aligned visual vectors. The aligned text vector is converted into a text query vector and a text key vector, and the aligned visual vector is converted into a visual query vector and a visual key vector; The correlation between the text query vector and text key vector and the visual query vector and visual key vector is calculated to obtain the attention matrix.

[0033] In this embodiment of the invention, the dimension alignment refers to the process of adjusting the dimensions of the text embedding vector and the visual embedding vector so that they are in the same numerical dimension space. The transformation refers to the process of converting the dimension-aligned text / visual vectors into vectors with "query" and "queried" functions respectively through linear mapping. The relevance calculation refers to the process of measuring the degree of correlation between different query vectors and key vectors.

[0034] Specifically, when aligning the dimensions of the text embedding vector and the visual embedding vector, the dimensions of the text embedding vector (e.g., D1) and the visual embedding vector (e.g., D2) are first calculated. Based on the input requirements of the attention calculation model (e.g., the input dimension D is fixed in the Transformer layer), the target alignment dimension is determined (usually D, or a value that is more suitable for the model between D1 and D2). If D1 ≠ D, the text embedding vector is mapped from D1 dimension to D dimension through a linear mapping layer (the same applies to the visual embedding vector). The dimensions of the adjusted text vector and the visual vector are checked to see if they are both D, and two types of aligned vectors are output.

[0035] Furthermore, when converting aligned text vectors into text query vectors and text key vectors, and aligned visual vectors into visual query vectors and visual key vectors, two independent linear mapping layers are built using the Transformer's Q / K generation module, denoted as the "Q mapping layer" and the "K mapping layer," respectively. The input dimension of both mapping layers is the target dimension D of the aligned vectors, and the output dimension is the same as the input dimension. The aligned text vectors are input into the Q mapping layer, and through matrix multiplication and activation functions, a text query vector with "query" functionality is generated. At the same time, the same aligned text vectors are input into the K mapping layer to generate a text key vector with "queried" functionality. Both have a dimension of D. The conversion of aligned visual vectors is similar.

[0036] Furthermore, when calculating the relevance between the text query vector and text key vector and the visual query vector and visual key vector, firstly, all vectors are paired: text query vector is paired with text key vector, text query vector is paired with visual key vector, visual query vector is paired with text key vector, and visual query vector is paired with visual key vector. Then, dot product similarity is used for quantification. Taking a certain query vector (such as the i-th vector in the text query vector) as an example, it is subjected to dot product operation with all key vectors (such as all text key vectors + all visual key vectors) one by one to obtain the relevance value between the query vector and each key vector.

[0037] Furthermore, all relevant values ​​are integrated according to the rule of "query vectors as rows and key vectors as columns". The row dimension is "the number of text query vectors + the number of visual query vectors" (i.e., the total number of query vectors), and the column dimension is "the number of text key vectors + the number of visual key vectors" (i.e., the total number of key vectors). Each matrix element corresponds to a set of "query vector - key vector" relevant values ​​to generate an attention matrix.

[0038] S3. Evaluate the importance of each visual token in the visual embedding vector based on the attention matrix to obtain a visual token importance score.

[0039] Specifically, based on the generated attention matrix, focusing on "the degree of attention of text tokens in the text embedding vector to visual tokens in the visual embedding vector" or "the degree of mutual attention between visual tokens", a quantitative score is assigned to each visual token through comprehensive calculation (such as averaging the values ​​of the corresponding columns of visual tokens in the matrix).

[0040] In healthcare settings, a submatrix is ​​extracted from the attention matrix, with "medical record text tokens as rows and CT visual tokens as columns." The average value of the column corresponding to each visual token is then calculated. For example, the value of the column corresponding to the "right lower lobe region token" is much higher than that of the column corresponding to the "heart region token." The "right lower lobe region token" has the highest importance score, while the "heart region token" has a lower score. This can help doctors quickly locate key lesion areas in CT images that match the medical record description and reduce attention to irrelevant areas.

[0041] In fintech scenarios, a submatrix with "risk report text tokens as rows and monitoring visual tokens as columns" can be extracted from the attention matrix. High-correlation values ​​in the corresponding columns of each visual token can be filtered out. For example, "ATM operator face tokens" and "ATM operation interface tokens" have the highest importance scores, while "monitoring background area tokens" have the lowest scores. This can help financial risk control systems prioritize the analysis of high-scoring visual tokens and quickly identify risky transactions.

[0042] In this embodiment of the invention, the step of evaluating the importance of each visual token in the visual embedding vector based on the attention matrix to obtain a visual token importance score includes: Extract the submatrix corresponding to the text query vector and the visual key vector in the attention matrix; The average text attention of each visual token in the submatrix is ​​calculated to obtain the preliminary average attention of each visual token. The visual token importance score is obtained by correcting the initial attention mean based on the visual embedding vector.

[0043] In this embodiment of the invention, the extraction refers to selectively filtering the row and column ranges of the corresponding text query vector (i.e., the query vector with the row index being the text token) and the corresponding visual key vector (i.e., the key vector with the column index being the visual token) from the attention matrix to obtain a sub-matrix that only reflects the "attention relationship between the text query vector and the visual key vector". The calculation of the average text attention refers to the process of averaging the "column data corresponding to each visual token" in the extracted sub-matrix.

[0044] In this embodiment of the invention, when extracting the submatrix corresponding to the text query vector and the visual key vector in the attention matrix, a matrix slicing technique is used to extract the row and column regions located by the corresponding text query vector (i.e., the query vector with the row index being the text token) and the visual key vector (i.e., the key vector with the column index being the visual token) in the attention matrix, thus obtaining the submatrix.

[0045] Furthermore, when calculating the average text attention for each visual token in the submatrix, the submatrix is ​​first split by column, with each column serving as an independent "visual token attention dataset". For example, if the submatrix has K columns (corresponding to K visual tokens), it is split into K datasets, each containing M values ​​(the attention intensity of M text query vectors for that visual token). The arithmetic mean is then calculated for each "visual token attention dataset".

[0046] Furthermore, the saliency of each visual token is extracted from the visual embedding vector. For example, through feature variance analysis, the "pixel contrast", "edge complexity", and "key region overlap" of the visual token in the image are calculated to obtain the saliency score of each visual token. The fusion weight of "text attention mean" and "feature saliency score" is set. For example, the weight of text attention mean is α (e.g., 0.6), and the weight of feature saliency score is β (e.g., 0.4), and α+β=1. For each visual token, the importance score is calculated according to the formula "visual token importance score = α × initial attention mean + β × feature saliency score". Finally, the importance scores of all visual tokens are mapped to the 0~1 interval through the Min-Max normalization method to obtain the visual token importance score.

[0047] S4. Filter the effective text embedding vectors from the text embedding vectors based on the similarity between the text embedding vector and the visual embedding vector according to a preset threshold.

[0048] In this embodiment of the invention, the filtering refers to calculating the similarity between each text embedding vector and all visual embedding vectors, and then comparing these similarity results with a preset threshold one by one. If at least one similarity result of a text embedding vector is higher than the threshold, it is retained as a valid text embedding vector. If all similarities are lower than the threshold, it is discarded.

[0049] In this embodiment of the invention, the step of filtering valid text embedding vectors from the text embedding vectors based on a preset threshold and their similarity to the text embedding vectors and visual embedding vectors includes: The text embedding vector and the visual embedding vector are normalized respectively to obtain normalized text vector and normalized visual vector; The normalized text vector and the normalized visual vector are multiplied by a dot product to obtain a similarity score matrix. Calculate an aggregate similarity score for each text vector in the normalized text vector based on the similarity score matrix; Valid text embedding vectors are selected from the text embedding vectors based on a preset threshold and the aggregated similarity score.

[0050] Specifically, when normalizing the text embedding vector and the visual embedding vector respectively, the magnitude of each text embedding vector and the magnitude of the visual embedding vector are first calculated. Then, all elements of each text embedding vector are divided by the magnitude of the vector to obtain a normalized text vector with a magnitude of 1. The normalization process for the visual embedding vector is similar.

[0051] Furthermore, when performing a dot product operation on the normalized text vector and the normalized visual vector, each normalized text vector is used as a row and each normalized visual vector is used as a column to form a combination of vector pairs to be operated on. The dot product operation is performed on each pair of "normalized text vector-normalized visual vector", that is, the elements at corresponding positions of the two vectors are multiplied and then summed. The resulting value is the similarity score of the pair of vectors. All similarity scores obtained from the dot product operation are arranged in the order of "text vector as row and visual vector as column" to form a two-dimensional matrix, namely the similarity score matrix.

[0052] Furthermore, when calculating the aggregated similarity score for each text vector in the normalized text vector based on the similarity score matrix, the row data corresponding to a single normalized text vector is first extracted from the similarity score matrix. This row contains the similarity scores between the text vector and all visual vectors. The extracted row data is aggregated by summing all the scores and dividing by the total number of visual vectors, using a method such as arithmetic mean. Each normalized text vector is aggregated to obtain a scalar value, which is the aggregated similarity score of the text vector.

[0053] Furthermore, based on business scenario requirements (such as the minimum requirement for the strength of text-visual association), a fixed threshold (e.g., 0.5) is set. The aggregated similarity score of each text vector is compared with the preset threshold. All text embedding vectors whose aggregated similarity scores meet the threshold requirements are retained, and vectors that do not meet the requirements are removed, ultimately resulting in a set of effective text embedding vectors.

[0054] S5. Perform redundancy trimming on the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data.

[0055] In this embodiment of the invention, redundancy pruning refers to the process of selectively removing duplicate / low-value visual tokens from visual data based on information redundancy and token importance.

[0056] Specifically, the corresponding attention matrix generated by the effective text embedding vector and visual vector is used. By analyzing the rank of the matrix (the smaller the rank, the more overlapping and redundant the visual token information), the optimal number N of visual tokens to be pruned in the current layer is determined in combination with the hyperparameters. Then, based on the visual token importance score calculated by the attention matrix, the N visual tokens with the lowest importance scores are selected, which is the visual data pruned.

[0057] In this embodiment of the invention, the step of redundancy trimming of the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data includes: Extract sub-attention matrices from the attention matrix based on the effective text embedding vector and the visual embedding vector, and calculate the rank of the sub-attention matrices; The redundancy of the visual data is calculated using the rank of the matrix. The visual data is pruned according to the redundancy and preset hyperparameters to obtain pruned visual data.

[0058] In this embodiment of the invention, the extraction refers to selectively extracting a sub-attention matrix from the attention matrix that contains only "information related to the effective text embedding vector and the visual embedding vector". The redundancy calculation refers to quantifying the degree of information duplication in the visual data based on the rank of the sub-attention matrix. The pruning refers to the process of removing the corresponding number of tokens with the lowest importance from the visual data after determining the number of redundant visual tokens to be removed based on the redundancy and preset hyperparameters.

[0059] Furthermore, when extracting the sub-attention matrix from the attention matrix based on the effective text embedding vector and the visual embedding vector, firstly, in the attention matrix, determine the row index corresponding to the effective text embedding vector (containing only filtered effective text tokens) and the column index corresponding to the visual embedding vector (containing all visual tokens), delineate the area where the two intersect as the range of the sub-matrix to be extracted, and extract the values ​​of the corresponding area from the attention matrix according to the located row and column range to form the sub-attention matrix. Then, through the rank calculation method in linear algebra (such as singular value decomposition), determine the number of linearly independent rows or columns in the sub-attention matrix to obtain the matrix rank.

[0060] Furthermore, the redundancy of the visual data is calculated using the rank of the matrix. The total number of visual tokens corresponding to the visual embedding vector is counted as the baseline value for redundancy calculation. The difference between the total number of visual tokens and the rank of the sub-attention matrix represents the scale of possible redundant information in the visual tokens (the larger the difference, the larger the potential space for redundant information). The above difference is proportionally calculated with the total number of visual tokens (e.g., the difference is divided by the total number) to obtain a normalized redundancy value.

[0061] Furthermore, when pruning visual data based on redundancy and hyperparameters, a preset hyperparameter is introduced (to control the pruning intensity, such as a value of 0.1~1.0). This hyperparameter is then combined with the redundancy and the total number of visual tokens to calculate the specific number N of visual tokens that need to be pruned. Based on the previously calculated visual token importance scores, the tokens are sorted from low to high, and the N visual tokens with the lowest scores are selected as the objects to be pruned. Finally, these N low-importance tokens are removed from the original visual data, and the remaining visual tokens are retained to form the pruned visual data.

[0062] The specific number N of visual tokens that needs to be pruned is calculated by combining it with redundancy and the total number of visual tokens, as expressed by the following formula:

[0063] Where N represents the number of visual tokens to be pruned, and λ is a hyperparameter used to control the intensity of sparsity. The value represents the total number of visual tokens in the current layer, and rank(P) represents the rank of the sub-attention matrix P. The value indicates the redundancy of the visual token. The higher the redundancy, the larger this value, and the more pruning is required.

[0064] S6. Reclaim and reconstruct the pruned visual data based on the visual token importance score to obtain reconstructed visual data.

[0065] In this embodiment of the invention, the recycling and reconstruction refers to the process of selecting some valuable tokens from the visual tokens to be pruned and recombining them.

[0066] Specifically, for the visual tokens to be removed from the pruned visual data, based on their importance scores, a small subset with higher scores is first collected. Then, a density peak-based clustering algorithm is used to divide these collected tokens into several clusters according to semantic similarity. Next, all tokens in each cluster are fused into new tokens by element-wise addition. Finally, these new tokens are combined with the unpruned tokens to form reconstructed visual data that reduces the amount of data while preserving information with low loss.

[0067] In this embodiment of the invention, the step of reclaiming and reconstructing the pruned visual data based on the visual token importance score to obtain reconstructed visual data includes: Based on the visual token importance score and a preset importance score threshold, recycled visual data is filtered out from the pruned visual data; Semantic clustering is performed on the collected visual data to obtain visual data clusters grouped by semantics; The visual data clusters with the same semantic meaning are fused to obtain fused visual data; Based on the visual token importance score and the pruned visual data, select the visual data to retain; Reconstructed visual data is generated by combining the preserved visual data and the fused visual data.

[0068] In this embodiment of the invention, semantic clustering refers to grouping the recovered visual tokens according to the similarity of their implied semantic information.

[0069] Specifically, when filtering the recycled visual data in the pruned visual data based on the visual token importance score and a preset importance score threshold, an importance score threshold (such as the 75th percentile of all tokens to be pruned) is preset according to business needs to define “relatively important tokens to be pruned”. The importance score of the tokens to be pruned is compared with the threshold, and a small number of tokens with scores higher than the threshold (such as the top 7%) are retained.

[0070] Furthermore, when performing semantic clustering on the recovered visual data, semantic features (such as the corresponding image region content and the association features with the effective text) are extracted from each token of the recovered visual data. A density peak-based clustering algorithm (such as k-nearest neighbor density peak aggregation) is used to calculate the semantic similarity between tokens. Based on the similarity, clusters are automatically divided, and tokens that are close to each other and semantically similar are grouped into the same cluster to form multiple visual data clusters. Each cluster represents a type of similar semantic information.

[0071] Furthermore, when fusing visual data clusters with the same semantics, for each visual data cluster, the embedding vectors of all visual tokens within the cluster are collected, and an element-wise addition operation is performed on all embedding vectors within the cluster to generate a new embedding vector. After fusion, each cluster obtains a compact "fusion token", and the fusion tokens of all clusters together constitute "fused visual data".

[0072] Furthermore, when filtering retained visual data from the visual data based on the visual token importance score and the pruned visual data, the pruned visual data is used as a reference to identify visual tokens that have not been marked as to be pruned. The importance scores of these tokens are checked to ensure that their scores are all higher than the preset importance score threshold. These verified tokens are identified as retained visual data. Finally, the retained visual data (high-importance original tokens) are merged with the fused visual data (compact tokens reconstructed from recycled tokens) to generate the final reconstructed visual data.

[0073] As can be seen, in the above scheme, for the reconstruction of visual data, the user-input text description data and visual data are obtained, and the text description data and visual data are encoded into text embedding vectors and visual embedding vectors, respectively; an attention matrix is ​​calculated based on the text embedding vectors and visual embedding vectors; the importance of each visual token in the visual embedding vectors is evaluated based on the attention matrix to obtain a visual token importance score; effective text embedding vectors in the text embedding vectors are filtered based on the similarity to the text embedding vectors and visual embedding vectors according to a preset threshold; redundancy is pruned in the visual data based on the effective text embedding vectors and the attention matrix to obtain pruned visual data; and the pruned visual data is recycled and reconstructed based on the visual token importance score to obtain reconstructed visual data. By redundancy pruning and recycling reconstructed visual data, redundant data in the visual data is deleted in a targeted manner, while core visual data is retained. This balances the integrity of the visual data while improving the efficiency of subsequent visual data calculation.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0075] In one embodiment, a text-guided visual data reconstruction apparatus is provided, which corresponds one-to-one with the text-guided visual data reconstruction method described in the above embodiments. For example... Figure 3 As shown, this text-guided visual data reconstruction device includes an acquisition and encoding module 101, a calculation module 102, an evaluation module 103, a filtering module 104, a trimming module 105, and a reconstruction module 106. Detailed descriptions of each functional module are as follows: The encoding module 101 is used to acquire text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors respectively. The calculation module 102 is used to calculate the attention matrix based on the text embedding vector and the visual embedding vector; Evaluation module 103 is used to evaluate the importance of each visual token in the visual embedding vector according to the attention matrix, and obtain a visual token importance score; The filtering module 104 is used to filter the effective text embedding vectors among the text embedding vectors based on the similarity between the text embedding vector and the visual embedding vector and a preset threshold. The trimming module 105 is used to perform redundancy trimming on the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The reconstruction module 106 is used to reclaim and reconstruct the pruned visual data based on the visual token importance score to obtain reconstructed visual data.

[0076] In one embodiment, the encoding module 101, when encoding the text description data into a text embedding vector, is used to: The text description data is cleaned to obtain standard text description data; Semantic features are extracted from the standard text description data to obtain text semantic features; The text semantic features are mapped to a preset dimension numerical space to obtain a text embedding vector.

[0077] In one embodiment, when calculating the attention matrix based on the text embedding vector and the visual embedding vector, the calculation module 102 is used to: The text embedding vector and the visual embedding vector are dimensionally aligned to obtain aligned text vectors and aligned visual vectors. The aligned text vector is converted into a text query vector and a text key vector, and the aligned visual vector is converted into a visual query vector and a visual key vector; The correlation between the text query vector and text key vector and the visual query vector and visual key vector is calculated to obtain the attention matrix.

[0078] In one embodiment, when the evaluation module 103 evaluates the importance of each visual token in the visual embedding vector according to the attention matrix to obtain a visual token importance score, it is used to: Extract the submatrix corresponding to the text query vector and the visual key vector in the attention matrix; The average text attention of each visual token in the submatrix is ​​calculated to obtain the preliminary average attention of each visual token. The visual token importance score is obtained by correcting the initial attention mean based on the visual embedding vector.

[0079] In one embodiment, the filtering module 104, when filtering valid text embedding vectors from the text embedding vectors based on a preset threshold and their similarity to the text embedding vectors and visual embedding vectors, is configured to: The text embedding vector and the visual embedding vector are normalized respectively to obtain normalized text vector and normalized visual vector; The normalized text vector and the normalized visual vector are multiplied by a dot product to obtain a similarity score matrix. Calculate an aggregate similarity score for each text vector in the normalized text vector based on the similarity score matrix; Valid text embedding vectors are selected from the text embedding vectors based on a preset threshold and the aggregated similarity score.

[0080] In one embodiment, when the trimming module 105 performs redundancy trimming on the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data, it is used to: Extract sub-attention matrices from the attention matrix based on the effective text embedding vector and the visual embedding vector, and calculate the rank of the sub-attention matrices; The redundancy of the visual data is calculated using the rank of the matrix. The visual data is pruned according to the redundancy and preset hyperparameters to obtain pruned visual data.

[0081] In one embodiment, when the reconstruction module 106 reclaims and reconstructs the pruned visual data based on the visual token importance score to obtain reconstructed visual data, it is used to: Based on the visual token importance score and a preset importance score threshold, recycled visual data is filtered out from the pruned visual data; Semantic clustering is performed on the collected visual data to obtain visual data clusters grouped by semantics; The visual data clusters with the same semantic meaning are fused to obtain fused visual data; Based on the visual token importance score and the pruned visual data, select the visual data to retain; Reconstructed visual data is generated by combining the preserved visual data and the fused visual data.

[0082] This invention provides a text-guided visual data reconstruction device. For visual data reconstruction, it acquires user-input text description data and visual data, and encodes the text description data and visual data into text embedding vectors and visual embedding vectors, respectively. An attention matrix is ​​calculated based on the text embedding vectors and visual embedding vectors. The importance of each visual token in the visual embedding vectors is evaluated based on the attention matrix to obtain a visual token importance score. Valid text embedding vectors are selected from the text embedding vectors based on their similarity to the text embedding vectors and visual embedding vectors according to a preset threshold. Redundancy is pruned from the visual data based on the valid text embedding vectors and the attention matrix to obtain pruned visual data. The pruned visual data is then reconstructed based on the visual token importance score to obtain reconstructed visual data. By pruning redundancy and reconstructing visual data, redundant data in the visual data is selectively deleted while retaining core visual data. This balances the integrity of the visual data while improving the efficiency of subsequent visual data calculations.

[0083] For specific limitations regarding a text-guided visual data reconstruction device, please refer to the limitations of a text-guided visual data reconstruction method described above, which will not be repeated here. Each module in the aforementioned text-guided visual data reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0084] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a text-guided visual data reconstruction method on the server side.

[0085] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a text-guided visual data reconstruction method.

[0086] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors respectively; Calculate the attention matrix based on the text embedding vector and the visual embedding vector; The importance of each visual token in the visual embedding vector is evaluated based on the attention matrix to obtain a visual token importance score. Valid text embedding vectors are selected from the text embedding vectors based on the similarity between the text embedding vector and the visual embedding vector according to a preset threshold. The visual data is redundancy-trimmed based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The pruned visual data is recycled and reconstructed based on the visual token importance score to obtain reconstructed visual data.

[0087] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors respectively; Calculate the attention matrix based on the text embedding vector and the visual embedding vector; The importance of each visual token in the visual embedding vector is evaluated based on the attention matrix to obtain a visual token importance score. Valid text embedding vectors are selected from the text embedding vectors based on the similarity between the text embedding vector and the visual embedding vector according to a preset threshold. The visual data is redundancy-trimmed based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The pruned visual data is recycled and reconstructed based on the visual token importance score to obtain reconstructed visual data.

[0088] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0089] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0091] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. If any software tools or components other than those of our company appear in the embodiments, they are merely illustrative examples and do not represent actual use. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A text-guided visual data reconstruction method, characterized in that, include: Obtain text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors respectively; Calculate the attention matrix based on the text embedding vector and the visual embedding vector; The importance of each visual token in the visual embedding vector is evaluated based on the attention matrix to obtain a visual token importance score. Valid text embedding vectors are selected from the text embedding vectors based on the similarity between the text embedding vector and the visual embedding vector according to a preset threshold. The visual data is redundancy-trimmed based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The pruned visual data is recycled and reconstructed based on the visual token importance score to obtain reconstructed visual data.

2. The text-guided visual data reconstruction method as described in claim 1, characterized in that, The step of calculating the attention matrix based on the text embedding vector and the visual embedding vector includes: The text embedding vector and the visual embedding vector are dimensionally aligned to obtain aligned text vectors and aligned visual vectors. The aligned text vector is converted into a text query vector and a text key vector, and the aligned visual vector is converted into a visual query vector and a visual key vector; The correlation between the text query vector and text key vector and the visual query vector and visual key vector is calculated to obtain the attention matrix.

3. The text-guided visual data reconstruction method as described in claim 2, characterized in that, The step of evaluating the importance of each visual token in the visual embedding vector based on the attention matrix to obtain a visual token importance score includes: Extract the submatrix corresponding to the text query vector and the visual key vector in the attention matrix; The average text attention of each visual token in the submatrix is ​​calculated to obtain the preliminary average attention of each visual token. The visual token importance score is obtained by correcting the initial attention mean based on the visual embedding vector.

4. The text-guided visual data reconstruction method as described in claim 1, characterized in that, The step of filtering valid text embedding vectors from the text embedding vectors based on a preset threshold and their similarity to the text embedding vectors and visual embedding vectors includes: The text embedding vector and the visual embedding vector are normalized respectively to obtain normalized text vector and normalized visual vector; The normalized text vector and the normalized visual vector are multiplied by a dot product to obtain a similarity score matrix. Calculate an aggregate similarity score for each text vector in the normalized text vector based on the similarity score matrix; Valid text embedding vectors are selected from the text embedding vectors based on a preset threshold and the aggregated similarity score.

5. The text-guided visual data reconstruction method as described in claim 1, characterized in that, The step of redundancy trimming of the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data includes: Extract sub-attention matrices from the attention matrix based on the effective text embedding vector and the visual embedding vector, and calculate the rank of the sub-attention matrices; The redundancy of the visual data is calculated using the rank of the matrix. The visual data is pruned according to the redundancy and preset hyperparameters to obtain pruned visual data.

6. The text-guided visual data reconstruction method as described in claim 1, characterized in that, The step of reclaiming and reconstructing the pruned visual data based on the visual token importance score to obtain reconstructed visual data includes: Based on the visual token importance score and a preset importance score threshold, recycled visual data is filtered out from the pruned visual data; Semantic clustering is performed on the collected visual data to obtain visual data clusters grouped by semantics; The visual data clusters with the same semantic meaning are fused to obtain fused visual data; Based on the visual token importance score and the pruned visual data, select the visual data to retain; Reconstructed visual data is generated by combining the preserved visual data and the fused visual data.

7. The text-guided visual data reconstruction method as described in claim 1, characterized in that, Encoding the text description data into a text embedding vector includes: The text description data is cleaned to obtain standard text description data; Semantic features are extracted from the standard text description data to obtain text semantic features; The text semantic features are mapped to a preset dimension numerical space to obtain a text embedding vector.

8. A text-guided visual data reconstruction device, characterized in that, include: The encoding module is used to acquire text description data and visual data input by the user, and encode the text description data and the visual data into text embedding vectors and visual embedding vectors, respectively. The calculation module is used to calculate the attention matrix based on the text embedding vector and the visual embedding vector; An evaluation module is used to evaluate the importance of each visual token in the visual embedding vector based on the attention matrix, and obtain a visual token importance score. The filtering module is used to filter valid text embedding vectors from the text embedding vectors based on a preset threshold and the similarity between the text embedding vector and the visual embedding vector; A trimming module is used to perform redundancy trimming on the visual data based on the effective text embedding vector and the attention matrix to obtain trimmed visual data. The reconstruction module is used to reclaim and reconstruct the pruned visual data based on the visual token importance score to obtain reconstructed visual data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the text-guided visual data reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the text-guided visual data reconstruction method as described in any one of claims 1 to 7.

Images