Multimodal sample correction method, system, machine-readable storage medium, and processor

Abstract

Embodiments of the present application provide a multi-modal sample correction method, system, machine-readable storage medium and processor, belonging to the technical field of sample correction. The multi-modal sample correction method comprises: obtaining image sample information and text sample information in a large model; inputting the image sample information and the text sample information into a modal encoder to obtain image sample features and text sample features; inputting the image sample features and the text sample features into a large language model backbone network to generate signal tokens; mapping the signal tokens to a fixed dimension vector space to obtain an image linear function and a text linear function; according to the obtained image linear function and the text linear function; performing deep fusion on the aligned image sample features and text sample features; and inputting the fused features into a knowledge graph guided K-adapter model. The multi-modal correction method can efficiently fuse and correct multi-modal data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sample correction, in particular to a multi-modal sample correction method, system, machine-readable storage medium and processor. BACKGROUND

[0002] There are objective multi-modal data in the domestic power system, and multi-modal data fusion is an effective way to promote the collaborative analysis of multiple heterogeneous networks and realize the controllability of the power grid. With the intelligent development of the power system, the power grid data presents new characteristics in attribute dimension, time granularity, spatial range, etc., which puts forward higher requirements for data storage, calculation and application. Domestic scholars have proposed the concepts of intra-grid fusion and inter-grid fusion, respectively referring to data fusion of different links within the power system and data fusion between different energy networks (electricity, gas, heat, etc.). Integrated fusion analysis of multi-energy flow data has become a research hotspot. In addition, as a visual large-scale semantic network, knowledge graph plays an increasingly important role in the intelligent application of the power system. Researchers have proposed to build a multi-modal knowledge graph of the power system, fuse heterogeneous data such as semantics, physical quantities and images, and support various application scenarios such as intelligent question answering and fault diagnosis. This provides a new idea for knowledge management and intelligent decision-making of the power system. Domestic and foreign scholars have carried out extensive research on multi-modal correction data technology of the power grid, and have made gratifying progress in the fields of distribution network fault diagnosis, state estimation, load forecasting, etc. However, the uncertainty, parameter sparsity and variable diversity of the power grid pose higher requirements for the fusion analysis of multi-modal data. Therefore, it is necessary to explore more efficient and practical data fusion methods and correct multi-modal data. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a multi-modal sample correction method, system, machine-readable storage medium and processor, which can efficiently fuse and correct multi-modal data.

[0004] To achieve the above-mentioned purpose, the embodiments of the present application provide a multi-modal sample correction method, which comprises:

[0005] Obtaining image sample information and text sample information in a large model;

[0006] Inputting the image sample information and the text sample information into a modal encoder to obtain image sample features and text sample features;

[0007] Inputting the image sample features and the text sample features into a large language model backbone network to generate signal tokens;

[0008] Mapping the signal tokens to a fixed-dimensional vector space to obtain an image linear function and a text linear function;

[0009] Based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned;

[0010] The aligned image sample features and text sample features are deeply fused to obtain fused features;

[0011] The fused features are fed into a knowledge graph-guided K-adapter model to repair the image sample information and text sample information corresponding to the fused features.

[0012] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0013] Obtain the image sample features and text sample features;

[0014] Based on the obtained image sample features and text sample features, the cosine similarity between the image sample features and text sample features is calculated using formula (1):

[0015]

[0016] Where I represents image sample features, T represents text sample features, and similarity(I,T) represents cosine similarity;

[0017] The cosine similarity values ​​greater than a preset threshold, along with their corresponding image sample features and text sample features, are filtered to achieve initial alignment of the image and text.

[0018] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0019] Set up an image queue of length k and a text queue of length k to hold the most recently input image sample features and text sample features;

[0020] Calculate the similarity between the current image sample features and all text sample features in the text queue, and the similarity between the current text sample features and all image sample features in the image queue, according to formula (2):

[0021] s(I,T)=f v (v) T f w ′(w)

[0022] s(T,I)=f w (w) T f v′(v), Formula (2)

[0023] Among them, f v (v) represents a linear function that maps image sample features to a fixed-dimensional vector space, f' w (w) represents a linear function mapping text sample features in the text queue to a fixed dimension, s(I,T) represents the similarity between the current image sample features and all text sample features in the text queue, where I represents the image sample features, T represents the text sample features, and f w (w) represents a linear function that maps text sample features to a fixed-dimensional vector space, f' v (v) represents the image sample features in the image queue mapped to a linear function of fixed dimension, and s(T,I) represents the similarity between the current text sample features and the text sample features in the image queue.

[0024] Calculate the similarity distribution between the current image sample features and all texts in the text queue, and the similarity distribution between the current text sample features and all images in the image queue, according to formula (3):

[0025]

[0026] in, This represents the features of the current image sample and the features T of the k-th text sample in the text queue. k similarity, This represents the difference between the current text sample features and the k-th image sample features in the image queue. k The similarity is given by m, where m represents the index parameter in the text queue and the image queue, and M represents the number of parameters in the text queue and the image queue.

[0027] The obtained similarity is normalized by formula (3), and the two normalized similarity distributions are compared to match the image sample features and text sample features with the highest similarity to complete the alignment of the image sample features and text sample features.

[0028] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0029] Obtain the similarity between the current image sample features and the text sample features in the text queue, as well as the similarity between the current text sample features and the image sample features in the image queue;

[0030] The cross-entropy loss function is calculated based on the obtained similarity when matching the image sample features and text sample features;

[0031] The matching process of the image sample features and text sample features is optimized by using the obtained cross-entropy loss function.

[0032] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0033] A first student network and a second student network based on the knowledge distillation method are constructed. After the image sample features and text sample features are input, the corresponding prediction output is output. The prediction output may include the similarity between the image sample features and the text sample features.

[0034] Based on the obtained prediction output, the KL divergence between the two student networks is calculated using formula (4):

[0035]

[0036] Where P1 represents the predicted output of the first student network, P2 represents the predicted output of the second student network, and the predicted output of one student network is used as the soft label of the other student network, D KL Let x represent the KL divergence. i This represents the input labeled i;

[0037] Obtain the KL divergence, and calculate the cross-entropy loss function for matching the image sample features and text sample features using formula (5) based on the obtained KL divergence:

[0038] L1=αL c1 +(1-α)D KL (P2||P1)

[0039] L2=αL c +(1-α)D KL (P1||P2), formula (5)

[0040] Among them, L c1 α represents the cross-entropy loss between the first student network and the true labels of image sample features and text sample features, and α represents the decay factor.

[0041] The first and second student networks are optimized according to the cross-entropy loss function, and the optimal image sample features and text sample features are aligned according to the KL divergence obtained by formula (4).

[0042] Optionally, the aligned image sample features and text sample features are deeply fused to obtain fused features, including:

[0043] The aligned image sample features and text sample features are fed into a multimodal fusion module consisting of a multi-layer Transformer encoder;

[0044] The multimodal fusion module calculates attention weights for the image sample features and text sample features;

[0045] The image sample features and text sample features, after being weighted and summed according to the calculated attention weights, are concatenated to obtain the fused features.

[0046] Optionally, the fused features are fed into a knowledge graph-guided K-adapter model to repair the image sample information and text sample information corresponding to the fused features, including:

[0047] Acquire large-scale data from the power industry and feed it into a Transformer-based pre-trained model to train the pre-trained model;

[0048] Insert the K-adapter into the pre-trained model after training;

[0049] A knowledge graph of the power industry is obtained, and the entities and relations in the knowledge graph are transformed into vector representations and introduced into a pre-trained model with K-adapter inserted, so as to enhance the understanding of power industry knowledge of the pre-trained model.

[0050] The fused features are fed into the pre-trained model after inserting a K-adapter and enhancing the understanding of power industry knowledge. The pre-trained model performs inference based on the given fused features and knowledge graph information, and completes the repair of the corresponding image sample information and text sample information.

[0051] On the other hand, the present invention also provides a multimodal sample correction system, the multimodal correction system comprising:

[0052] The data acquisition module is used to acquire image sample information and text sample information;

[0053] The correction module is used to input the image sample information and text sample information into the modal encoder to obtain image sample features and text sample features;

[0054] The image sample features and text sample features are input into the backbone network of the large language model to generate signal tokens;

[0055] The signal tokens are mapped to a fixed-dimensional vector space to obtain image linear functions and text linear functions;

[0056] Based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned;

[0057] The aligned image sample features and text sample features are deeply fused to obtain fused features;

[0058] The fused features are fed into a knowledge graph-guided K-adapter model to repair the image sample information and text sample information corresponding to the fused features.

[0059] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0060] Obtain the image sample features and text sample features;

[0061] Based on the obtained image sample features and text sample features, the cosine similarity between the image sample features and text sample features is calculated using formula (1):

[0062]

[0063] Where I represents image sample features, T represents text sample features, and similarity(I,T) represents cosine similarity;

[0064] The cosine similarity values ​​greater than a preset threshold, along with their corresponding image sample features and text sample features, are filtered to achieve initial alignment of the image and text.

[0065] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0066] Set up an image queue of length k and a text queue of length k to hold the most recently input image sample features and text sample features;

[0067] Calculate the similarity between the current image sample features and all text sample features in the text queue, and the similarity between the current text sample features and all image sample features in the image queue, according to formula (2):

[0068] s(I,T)=f v (v) T f w ′(w)

[0069] s(T,I)=f w (w) T f v ′(v), Formula (2)

[0070] Among them, f v (v) represents a linear function that maps image sample features to a fixed-dimensional vector space, f' w (w) represents a linear function mapping text sample features in the text queue to a fixed dimension, s(I,T) represents the similarity between the current image sample features and all text sample features in the text queue, where I represents the image sample features, T represents the text sample features, and f w (w) represents a linear function that maps text sample features to a fixed-dimensional vector space, f' v (v) represents the image sample features in the image queue mapped to a linear function of fixed dimension, and s(T,I) represents the similarity between the current text sample features and the text sample features in the image queue.

[0071] Calculate the similarity distribution between the current image sample features and all texts in the text queue, and the similarity distribution between the current text sample features and all images in the image queue, according to formula (3):

[0072]

[0073] in, This represents the features of the current image sample and the features T of the k-th text sample in the text queue. k similarity, This represents the difference between the current text sample features and the k-th image sample features in the image queue. k The similarity is given by m, where m represents the index parameter in the text queue and the image queue, and M represents the number of parameters in the text queue and the image queue.

[0074] The obtained similarity is normalized by formula (3), and the two normalized similarity distributions are compared to match the image sample features and text sample features with the highest similarity to complete the alignment of the image sample features and text sample features.

[0075] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0076] Obtain the similarity between the current image sample features and the text sample features in the text queue, as well as the similarity between the current text sample features and the image sample features in the image queue;

[0077] The cross-entropy loss function is calculated based on the obtained similarity when matching the image sample features and text sample features;

[0078] The matching process of the image sample features and text sample features is optimized by using the obtained cross-entropy loss function.

[0079] Optionally, based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including:

[0080] A first student network and a second student network based on the knowledge distillation method are constructed. After the image sample features and text sample features are input, the corresponding prediction output is output. The prediction output may include the similarity between the image sample features and the text sample features.

[0081] Based on the obtained prediction output, the KL divergence between the two student networks is calculated using formula (4):

[0082]

[0083] Where P1 represents the predicted output of the first student network, P2 represents the predicted output of the second student network, and the predicted output of one student network is used as the soft label of the other student network, D KL Let x represent the KL divergence. i This represents the input labeled i;

[0084] Obtain the KL divergence, and calculate the cross-entropy loss function for matching the image sample features and text sample features using formula (5) based on the obtained KL divergence:

[0085] L1=αL c1 +(1-α)D KL (P2||P1)

[0086] L2=αL c2 +(1-α)D KL (P1||P2), formula (5)

[0087] Among them, L c1 α represents the cross-entropy loss between the first student network and the true labels of image sample features and text sample features, and α represents the decay factor.

[0088] The first and second student networks are optimized according to the cross-entropy loss function, and the optimal image sample features and text sample features are aligned according to the KL divergence obtained by formula (4).

[0089] Optionally, the aligned image sample features and text sample features are deeply fused to obtain fused features, including:

[0090] The aligned image sample features and text sample features are fed into a multimodal fusion module consisting of a multi-layer Transformer encoder;

[0091] The multimodal fusion module calculates attention weights for the image sample features and text sample features;

[0092] The image sample features and text sample features, after being weighted and summed according to the calculated attention weights, are concatenated to obtain the fused features.

[0093] Optionally, the fused features are fed into a knowledge graph-guided K-adapter model to repair the image sample information and text sample information corresponding to the fused features, including:

[0094] Acquire large-scale data from the power industry and feed it into a Transformer-based pre-trained model to train the pre-trained model;

[0095] Insert the K-adapter into the pre-trained model after training;

[0096] A knowledge graph of the power industry is obtained, and the entities and relations in the knowledge graph are transformed into vector representations and introduced into a pre-trained model with K-adapter inserted, so as to enhance the understanding of power industry knowledge of the pre-trained model.

[0097] The fusion features are obtained and fed into the pre-trained model with the K-adapter inserted. The pre-trained model performs inference based on the given fusion features and knowledge graph information, and completes the repair of the corresponding image sample information and text sample information.

[0098] In another aspect, the present invention also provides a machine-readable storage medium storing instructions for causing a machine to execute a multimodal sample correction method as described above.

[0099] In another aspect, the present invention also provides a processor for running a program, wherein the program is run to execute a multimodal sample correction method as described above.

[0100] Through the above technical solution, the multimodal sample correction method, system, machine-readable storage medium, and processor provided by this invention can acquire image sample information and text sample information from a large model. This image sample information and text sample information can then be input into a modal encoder to obtain image sample features and text sample features. The modal encoder's task is to encode inputs from different modalities to obtain corresponding features. After acquiring the image sample features and text sample features, these features can be input into the backbone network of a large language model to generate signal tokens. After obtaining the signal tokens, they can be mapped to a fixed-dimensional vector space to obtain image linear functions and text linear functions. Based on the obtained image linear functions and text linear functions, the image sample features and text sample features can be aligned. After alignment, the aligned image sample features and text sample features can be deeply fused to obtain fused features. After obtaining the fused features, they can be fed into a knowledge graph-guided K-adapter model to repair the input image sample information and text sample information. This multimodal correction method can efficiently fuse and correct multimodal data.

[0101] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0102] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0103] Figure 1 This is a flowchart of a multimodal sample correction method according to an embodiment of the present invention;

[0104] Figure 2 This is a first flowchart of a multimodal sample correction method for text alignment according to an embodiment of the present invention;

[0105] Figure 3 This is a second flowchart of a multimodal sample correction method for text alignment according to an embodiment of the present invention;

[0106] Figure 4 This is a third flowchart of a modal sample correction method for text-image alignment according to an embodiment of the present invention;

[0107] Figure 5 This is a fourth flowchart of a modal sample correction method according to an embodiment of the present invention for image-text alignment;

[0108] Figure 6 This is a flowchart of feature fusion for a multimodal sample correction method according to an embodiment of the present invention;

[0109] Figure 7 This is a flowchart of a multimodal sample correction method according to an embodiment of the present invention. Detailed Implementation

[0110] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0111] In the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, they do not mean that the applicant has used or necessarily used the solution.

[0112] Figure 1 This is a flowchart of a multimodal sample correction method according to an embodiment of the present invention. In this invention, the multimodal sample correction process may include:

[0113] In step S1, image sample information and text sample information from the large model are obtained.

[0114] In step S2, image sample information and text sample information are input into the modal encoder to obtain image sample features and text sample features.

[0115] In step S3, the image sample features and text sample features are input into the backbone network of the large language model to generate signal tokens.

[0116] In step S4, the signal tokens are mapped to a fixed-dimensional vector space to obtain the image linear function and the text linear function.

[0117] In step S5, the image sample features and text sample features are aligned based on the obtained image linear function and text linear function.

[0118] In step S6, the aligned image sample features and text sample features are deeply fused to obtain fused features.

[0119] In step S7, the fused features are fed into the knowledge graph-guided K-adapter model to repair the image sample information and text sample information corresponding to the fused features.

[0120] In this invention, when correcting multimodal data, image and text sample information from a large model can be obtained. This information can then be input into a modal encoder to obtain image and text sample features. The modal encoder's task is to encode inputs from different modalities to obtain corresponding features. Encoders primarily used for images include: NFNet-F6, ViT, CLIP ViT, Eva-CLIPViT, BEiT-3, Open-CLIP, Grounding-DINOT (based on the Swin-T backbone network), DINOv2, SAM-HQ, RAM++ (based on the Swin-B backbone network), InternViT, and Vcoder. After obtaining the image and text sample features, they can be input into a large language model backbone network to generate signal tokens. The large language model backbone network uses a large language model as its core intelligent agent, capable of processing representations from various modalities and participating in semantic understanding, reasoning, and decision-making related to the input. It can generate direct text output as well as signal tokens from other modalities. After obtaining the signal tokens, they can be mapped to a fixed-dimensional vector space, thus obtaining image and text linear functions. Based on the obtained image and text linear functions, the image and text sample features can be aligned. After alignment, the aligned image and text sample features can be deeply fused to obtain fused features. These fused features can then be fed into a knowledge graph-guided K-adapter model to repair the image and text sample information corresponding to the input fused features. This multimodal correction method can efficiently fuse and correct multimodal data.

[0121] In one embodiment of the present invention, such as Figure 2 As shown, the first step in image and text alignment may include:

[0122] In step S8, image sample features and text sample features are obtained.

[0123] In step S9, based on the obtained image sample features and text sample features, the cosine similarity between the image sample features and text sample features is calculated using formula (1):

[0124]

[0125] Where I represents image sample features, T represents text sample features, and similarity(I,T) represents cosine similarity.

[0126] In step S10, cosine similarity values ​​greater than a preset threshold, along with their corresponding image sample features and text sample features, are filtered to achieve preliminary alignment of the image and text.

[0127] In this invention, when obtaining the image linear function and the text linear function, and aligning the image and text according to the image linear function and the text linear function, the image sample information and text sample information can be initially filtered according to the corresponding image sample features and text sample features to initially align the image and text. During the initial alignment, the image sample features and text sample features can be obtained first, and then the cosine similarity between the image sample features and text sample features can be calculated according to the obtained image sample features and text sample features using formula (1). After calculating and obtaining the cosine similarity, since there are multiple images and texts, and the cosine similarity of each pair is different, cosine similarity greater than a preset threshold and the corresponding image sample features and text sample features can be filtered out, thereby removing some image-text pairs with excessively large similarity differences and achieving initial alignment of the image and text.

[0128] In one embodiment of the present invention, such as Figure 3 As shown, the second step in image and text alignment may include:

[0129] In step S11, an image queue of length k and a text queue of length k are set up to accommodate the most recently input image sample features and text sample features.

[0130] In step S12, the similarity between the current image sample features and all text sample features in the text queue, and the similarity between the current text sample features and all image sample features in the image queue are calculated according to formula (2):

[0131] s(I,T)=f v (v) T f w ′(w)

[0132] s(T,I)=f w (w) T f v ′(v), Formula (2)

[0133] Among them, f v (v) represents a linear function that maps image sample features to a fixed-dimensional vector space, f' w (w) represents a linear function mapping text sample features in the text queue to a fixed dimension, s(I,T) represents the similarity between the current image sample features and all text sample features in the text queue, where I represents the image sample features, T represents the text sample features, and fw (w) represents a linear function that maps text sample features to a fixed-dimensional vector space, f' v (v) represents the mapping of image sample features in the image queue to a linear function of fixed dimension, and s(T,I) represents the similarity between the current text sample features and the features of all text samples in the image queue.

[0134] In step S13, the similarity distribution between the current image sample features and all texts in the text queue, and the similarity distribution between the current text sample features and all images in the image queue are calculated according to formula (3):

[0135]

[0136] in, This represents the features of the current image sample and the features T of the k-th text sample in the text queue. k similarity, This represents the difference between the current text sample features and the k-th image sample features in the image queue. k The similarity is given by m, where m represents the index parameter in the text queue and the image queue, and M represents the number of parameters in the text queue and the image queue.

[0137] In step S14, the obtained similarity is normalized by formula (3), and the two normalized similarity distributions are compared to match the image sample features and text sample features with the highest similarity, so as to complete the alignment of image sample features and text sample features.

[0138] In this invention, after obtaining the image sample features and text sample features after preliminary screening, an image queue of length k and a text queue of length k can be set up to accommodate the most recently input image sample features and text sample features. The similarity between the image sample features currently added to the image queue and all text sample features in the text queue can be calculated using formula (2). The similarity between the text sample features currently added to the text queue and all image sample features in the image queue can also be calculated using formula (2). This similarity is calculated using a linear function corresponding to the image sample features and text sample features. After obtaining the similarity, the similarity distribution between the current image sample features and all images in the image queue can be calculated using formula (3). Formula (3) also normalizes the obtained similarity. After obtaining the similarity distribution, the two obtained similarity distributions can be compared, thereby matching the two image sample features and text sample features with the highest similarity to complete the alignment of the image sample features and text sample features.

[0139] In one embodiment of the present invention, such as Figure 4As shown, the third step in image-text alignment may include:

[0140] In step S15, the similarity between the current image sample features and the text sample features in the text queue, as well as the similarity between the current text sample features and the image sample features in the image queue, are obtained.

[0141] In step S16, the cross-entropy loss function is calculated based on the obtained similarity when matching image sample features and text sample features.

[0142] In step S17, the matching process of image sample features and text sample features is optimized by using the obtained cross-entropy loss function.

[0143] In this invention, when aligning images and text using formulas (2) and (3), continuous optimization is required to ensure more accurate alignment. During optimization, the similarity between the current image sample features and the text sample features in the text queue is obtained, and the similarity between the current text sample features and the image sample features in the image queue is also obtained. After obtaining the similarity, the cross-entropy loss function for matching the image sample features and text sample features can be calculated based on the obtained similarity. The matching process between the image sample features and text sample features can be optimized based on the obtained cross-entropy loss function, that is, the alignment process of the image sample features and text sample features is optimized while minimizing the cross-entropy loss function.

[0144] In one embodiment of the present invention, such as Figure 5 As shown, the fourth step in image-text alignment may include:

[0145] In step S18, a first student network and a second student network based on the knowledge distillation method are constructed. After inputting image sample features and text sample features, the corresponding prediction output is output. The prediction output may include the similarity between image sample features and text sample features.

[0146] In step S19, based on the obtained prediction output, the KL divergence between the two student networks is calculated using formula (4):

[0147]

[0148] Where P1 represents the predicted output of the first student network, P2 represents the predicted output of the second student network, and the predicted output of one student network is used as the soft label of the other student network, D KL Let x represent the KL divergence. i This represents the input labeled i.

[0149] In step S20, the KL divergence is obtained, and the cross-entropy loss function for matching image sample features and text sample features is calculated using formula (5) based on the obtained KL divergence:

[0150] L1=αL c1 +(1-α)D KL (P2||P1)

[0151] L2=αL c2 +(1-α)D KL (P1||P2), formula (5)

[0152] Among them, L c1 Let α represent the cross-entropy loss between the first student network and the true labels of image sample features and text sample features, and let α represent the decay factor.

[0153] In step S21, the first student network and the second student network are optimized according to the cross-entropy loss function, and the optimal image sample features and text sample features are aligned according to the KL divergence obtained by formula (5).

[0154] In this invention, when aligning image sample features and text sample features, another method can be used. A first student network and a second student network based on the knowledge distillation method can be constructed. These two student networks can include formulas (2) and (3). After inputting image sample features and text sample features, corresponding prediction outputs can be output. The prediction outputs can include the similarity between image sample features and text sample features. After obtaining the prediction outputs, the KL divergence between the two student networks can be calculated using formula (4). When the KL divergence between the two student networks is sufficient, the output of one student network can be used as the soft label of the other student network. After obtaining the KL divergence, the cross-entropy loss function for matching image sample features and text sample features can be calculated using formula (5) based on the obtained KL divergence. The first student network and the second student network can be optimized based on the obtained cross-entropy loss function. After optimization, the image sample features and text sample features corresponding to the optimal output can be aligned based on the KL divergence obtained by formula (4).

[0155] In one embodiment of the present invention, such as Figure 6 As shown, the feature fusion process may include:

[0156] In step S22, the aligned image sample features and text sample features are fed into a multimodal fusion module composed of a multi-layer Transformer encoder.

[0157] In step S23, the multimodal fusion module calculates attention weights for image sample features and text sample features.

[0158] In step S24, the weighted summation of image sample features and text sample features is concatenated according to the calculated attention weights to obtain fused features.

[0159] In this invention, after image and text alignment, aligned image sample features and text sample features can be obtained. These features are then fed into a multimodal fusion module composed of multiple Transformer encoders. This module performs feature fusion on the image and text sample features. The module calculates attention weights for the image and text sample features, and after calculation, concatenates the weighted summed features based on the attention weights to obtain the fused features.

[0160] In one embodiment of the present invention, such as Figure 7 As shown, the data repair process may include:

[0161] In step S25, large-scale data from the power industry is acquired and fed into a Transformer-based pre-trained model to train the model.

[0162] In step S26, a K-adapter is inserted into the pre-trained model after training.

[0163] In step S27, a knowledge graph of the power industry is obtained, and the entities and relations in the knowledge graph are converted into vector representations and introduced into the pre-trained model with the K-adapter inserted, so as to enhance the pre-trained model's understanding of power industry knowledge.

[0164] In step S28, the fused features are fed into the pre-trained model after inserting the K-adapter and enhancing the understanding of power industry knowledge. The pre-trained model performs inference based on the given fused features and knowledge graph information, and completes the repair of the corresponding image sample information and text sample information.

[0165] The method of using knowledge graphs to guide K-adapters for large-scale model data repair is based on the idea of ​​enhancing the knowledge representation ability of a pre-trained language model by introducing structured knowledge from the knowledge graph into the model, thereby improving the accuracy of data repair. Specifically, given a pre-trained language model... and a knowledge graph in Represents a set of entities. This represents a set of relations. Our goal is to incorporate structured knowledge from the knowledge graph G into the model. In order to enhance its knowledge representation ability. The principle of using the knowledge graph-guided K-Adapter method for large model data repair is based on the extension of BERT or similar large language models, by introducing additional adapter layers (K-Adapters) to integrate domain-specific knowledge. These adapter layers are designed to integrate additional grammar and domain knowledge without interfering with the original model architecture, thereby improving the model's ability to understand and repair text. Therefore, large-scale data from the power industry can be obtained and fed into a Transformer-based pre-trained model for training, and then adapter layers (K-Adapters) can be inserted into the pre-trained model. According to the knowledge graph-guided K-adapter model, after training, the fused features can be fed into the trained model, and then the model can reason according to the given fused features and knowledge graph information, and then repair the corresponding image sample information and text sample information. The K-Adapter structure achieves data repair through the following two key components: (1) Syntax adapter (F-Adapter): enhances the model's understanding of the grammatical structure of the text. (2) Knowledge Adapter (K-Adapter): Introduces facts and relationships from external knowledge graphs to enhance the model’s understanding of domain-specific knowledge.

[0166] In this way, the K-Adapter method not only maintains the capabilities of the original language model but also introduces additional grammatical and domain knowledge through the adapter layer, thereby significantly improving the model's text repair capabilities without altering the original model structure. The key to this method lies in its ability to flexibly integrate rich information from knowledge graphs, making the model more accurate and efficient in handling domain-specific text repair tasks. The training process of the K-Adapter can be implemented by minimizing the loss function to ensure that the model can effectively utilize the introduced knowledge for text repair.

[0167] By incorporating structured knowledge from knowledge graphs into a pre-trained language model, the K-adapter method enhances the model's knowledge representation capabilities, thereby improving the accuracy of data repair. This approach effectively utilizes external knowledge to guide the data repair process while preserving the original capabilities of the language model.

[0168] In one embodiment of the present invention, before inputting the image sample information into the modal encoder, the image sample information can be adjusted to a uniform size, and the pixel values ​​in the image can be scaled to between 0 and 1 to facilitate subsequent feature extraction.

[0169] In one embodiment of the present invention, before inputting the text sample information into the modal encoder, the text in the text sample information can be segmented and encoded, and the text sequence can be padded to a fixed length to facilitate subsequent feature extraction.

[0170] On the other hand, the present invention also provides a multimodal sample correction system, which includes a data acquisition module and a correction module. The data acquisition module can be used to acquire image sample information and text sample information. The correction module can input the acquired image sample information and text sample information into a modal encoder to obtain image sample features and text sample features; input the image sample features and text sample features into a large language model backbone network to generate signal tokens; map the signal tokens to a fixed-dimensional vector space to obtain image linear functions and text linear functions; align the image sample features and text sample features according to the acquired image linear functions and text linear functions; perform deep fusion of the aligned image sample features and text sample features to obtain fused features; and send the fused features into a knowledge graph-guided K-adapter model to repair the image sample information and text sample information corresponding to the fused features.

[0171] When obtaining the image linear function and text linear function, and aligning the image and text based on the image linear function and text linear function, the image sample information and text sample information can be initially filtered according to the corresponding image sample features and text sample features to initially align the image and text. In the initial alignment, the image sample features and text sample features can be obtained first, and then the cosine similarity between the image sample features and text sample features can be calculated according to the obtained image sample features and text sample features using formula (1). After calculating and obtaining the cosine similarity, since there are multiple images and texts, and the cosine similarity of each pair is different, the cosine similarity greater than the preset threshold and the corresponding image sample features and text sample features can be filtered out, thereby removing some image-text pairs with too large a similarity difference and realizing the initial alignment of the image and text.

[0172] After obtaining the image sample features and text sample features after preliminary screening, an image queue of length k and a text queue of length k can be set up to accommodate the most recently input image sample features and text sample features. The similarity between the image sample features currently added to the image queue and all text sample features in the text queue can be calculated using formula (2). The similarity between the text sample features currently added to the text queue and all image sample features in the image queue can also be calculated using formula (2). This similarity is calculated using a linear function corresponding to the image sample features and text sample features. After obtaining the similarity, the similarity distribution between the current image sample features and all images in the image queue can be calculated using formula (3). Formula (3) also normalizes the obtained similarity. After obtaining the similarity distribution, the two obtained similarity distributions can be compared, thus matching the two image sample features and text sample features with the highest similarity to complete the alignment of the image sample features and text sample features.

[0173] When aligning images and text using formulas (2) and (3), continuous optimization is required to ensure more accurate alignment. During optimization, the similarity between the current image sample features and the text sample features in the text queue is obtained, as is the similarity between the current text sample features and the image sample features in the image queue. After obtaining the similarity, the cross-entropy loss function for matching the image sample features and text sample features can be calculated based on the obtained similarity. The matching process between the image sample features and text sample features can be optimized based on the obtained cross-entropy loss function; that is, the alignment process of the image sample features and text sample features is optimized while minimizing the cross-entropy loss function.

[0174] When aligning image sample features and text sample features, another method can be used. A first student network and a second student network based on the knowledge distillation method can be constructed. These two student networks can include formulas (2) and (3). After inputting image sample features and text sample features, corresponding predicted outputs can be output. The predicted outputs can include the similarity between image sample features and text sample features. After obtaining the predicted outputs, the KL divergence between the two student networks can be calculated using formula (4). When the KL divergence between the two student networks is sufficient, the output of one student network can be used as the soft label of the other student network. After obtaining the KL divergence, the cross-entropy loss function for matching image sample features and text sample features can be calculated using formula (5) based on the obtained KL divergence. The first student network and the second student network can be optimized based on the obtained cross-entropy loss function. After optimization, the image sample features and text sample features corresponding to the optimal output can be aligned based on the KL divergence obtained by formula (4).

[0175] After image and text alignment, aligned image and text sample features can be obtained. These features are then fed into a multimodal fusion module composed of multiple Transformer encoders. This module performs feature fusion on the image and text sample features. It calculates attention weights for the image and text sample features, and after calculation, concatenates the weighted summed features based on these attention weights to obtain the fused features.

[0176] The method of using knowledge graphs to guide K-adapters for large-scale model data repair is based on the idea of ​​enhancing the knowledge representation ability of a pre-trained language model by introducing structured knowledge from the knowledge graph into the model, thereby improving the accuracy of data repair. Specifically, given a pre-trained language model... and a knowledge graph in Represents a set of entities. This represents a set of relations. Our goal is to incorporate structured knowledge from the knowledge graph G into the model. In order to enhance its knowledge representation ability. The principle of using the knowledge graph-guided K-Adapter method for large model data repair is based on the extension of BERT or similar large language models, by introducing additional adapter layers (K-Adapters) to integrate domain-specific knowledge. These adapter layers are designed to integrate additional grammar and domain knowledge without interfering with the original model architecture, thereby improving the model's ability to understand and repair text. Therefore, large-scale data from the power industry can be obtained and fed into a Transformer-based pre-trained model for training, and then adapter layers (K-Adapters) can be inserted into the pre-trained model. According to the knowledge graph-guided K-adapter model, after training, the fused features can be fed into the trained model, and then the model can reason according to the given fused features and knowledge graph information, and then repair the corresponding image sample information and text sample information. The K-Adapter structure achieves data repair through the following two key components: (1) Syntax adapter (F-Adapter): enhances the model's understanding of the grammatical structure of the text. (2) Knowledge Adapter (K-Adapter): Introduces facts and relationships from external knowledge graphs to enhance the model’s understanding of domain-specific knowledge.

[0177] In this way, the K-Adapter method not only maintains the capabilities of the original language model but also introduces additional grammatical and domain knowledge through the adapter layer, thereby significantly improving the model's text repair capabilities without altering the original model structure. The key to this method lies in its ability to flexibly integrate rich information from knowledge graphs, making the model more accurate and efficient in handling domain-specific text repair tasks. The training process of the K-Adapter can be implemented by minimizing the loss function to ensure that the model can effectively utilize the introduced knowledge for text repair.

[0178] By incorporating structured knowledge from knowledge graphs into a pre-trained language model, the K-adapter method enhances the model's knowledge representation capabilities, thereby improving the accuracy of data repair. This approach effectively utilizes external knowledge to guide the data repair process while preserving the original capabilities of the language model.

[0179] On the other hand, the present invention also provides a machine-readable storage medium storing instructions for causing a machine to execute a multimodal sample correction method as described above.

[0180] In another aspect, the present invention also provides a processor for running a program, wherein the program is run to execute a multimodal sample correction method as described above.

[0181] Through the above technical solution, the multimodal sample correction method, system, machine-readable storage medium, and processor provided by this invention can acquire image sample information and text sample information from a large model. This image sample information and text sample information can then be input into a modal encoder to obtain image sample features and text sample features. The modal encoder's task is to encode inputs from different modalities to obtain corresponding features. After acquiring the image sample features and text sample features, these features can be input into the backbone network of a large language model to generate signal tokens. After obtaining the signal tokens, they can be mapped to a fixed-dimensional vector space to obtain image linear functions and text linear functions. Based on the obtained image linear functions and text linear functions, the image sample features and text sample features can be aligned. After alignment, the aligned image sample features and text sample features can be deeply fused to obtain fused features. After obtaining the fused features, they can be fed into a knowledge graph-guided K-adapter model to repair the input image sample information and text sample information. This multimodal correction method can efficiently fuse and correct multimodal data.

[0182] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0187] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0188] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0189] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0190] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multimodal sample correction method, characterized in that, The multimodal sample correction method includes: Obtain image and text sample information from a large model; The image sample information and text sample information are input into the modal encoder to obtain image sample features and text sample features; The image sample features and text sample features are input into the backbone network of the large language model to generate signal tokens; The signal tokens are mapped to a fixed-dimensional vector space to obtain image linear functions and text linear functions; Based on the obtained image linear function and text linear function, the image sample features and text sample features are aligned, including: A first student network and a second student network based on the knowledge distillation method are constructed. After the image sample features and text sample features are input, the corresponding prediction output is output, and the prediction output includes the similarity between the image sample features and the text sample features. Based on the obtained prediction output, the relationship between the two student networks is calculated using formula (4). Divergence: , Formula (4) in, This represents the predicted output of the first student network. This represents the predicted output of the second student network, and the predicted output of one student network is used as the soft label in the other student network. express divergence, Indicates that the label is Input; Obtain the Divergence, and based on the obtained The divergence is calculated using formula (5) as the cross-entropy loss function for matching the image sample features and text sample features: , Formula (5) in, The cross-entropy loss represents the relationship between the first student network and the true labels of image sample features and text sample features. Indicates the attenuation factor; The first and second student networks are optimized according to the cross-entropy loss function, and the results are obtained according to formula (4). The divergence aligns the optimal image sample features and text sample features; The aligned image sample features and text sample features are deeply fused to obtain fused features; The fused features are fed into a knowledge graph-guided K-adapter model to repair the image and text sample information corresponding to the fused features.

2. The multimodal sample correction method according to claim 1, characterized in that, Aligning the image sample features and text sample features based on the obtained image linear function and text linear function further includes: Obtain the image sample features and text sample features; Based on the obtained image sample features and text sample features, the cosine similarity between the image sample features and text sample features is calculated using formula (1): , Formula (1) in, Representing image sample features, Representing text sample features, Indicates cosine similarity; The cosine similarity values ​​greater than a preset threshold, along with their corresponding image sample features and text sample features, are filtered to achieve initial alignment of the image and text.

3. The multimodal sample correction method according to claim 2, characterized in that, Aligning the image sample features and text sample features based on the obtained image linear function and text linear function further includes: Set a length of The image queue and its length are A text queue to hold recently input image sample features and text sample features; Calculate the similarity between the current image sample features and all text sample features in the text queue, and the similarity between the current text sample features and all image sample features in the image queue, according to formula (2): , Formula (2) in, It represents a linear function that maps image sample features to a fixed-dimensional vector space. This represents a linear function that maps the features of text samples in a text queue to a fixed-dimensional function. This represents the similarity between the features of the current image sample and the features of all text samples in the text queue. Representing image sample features, Representing text sample features, This represents a linear function that maps text sample features to a fixed-dimensional vector space. This represents a linear function that maps the features of image samples in an image queue to a fixed-dimensional function. This represents the similarity between the features of the current text sample and the features of all text samples in the image queue. Calculate the similarity distribution between the current image sample features and all texts in the text queue, and the similarity distribution between the current text sample features and all images in the image queue, according to formula (3): , Formula (3) in, This indicates that the features of the current image sample are compared with the first one in the text queue. Features of a text sample similarity, This indicates that the features of the current text sample are compared with the first image in the image queue. Image sample features similarity, This represents the index parameter in the text queue and image queue. Indicates the number of parameters in the text queue and the image queue; The obtained similarity is normalized using formula (3).

4. The multimodal sample correction method according to claim 3, characterized in that, Aligning the image sample features and text sample features based on the obtained image linear function and text linear function further includes: Obtain the similarity between the current image sample features and the text sample features in the text queue, as well as the similarity between the current text sample features and the image sample features in the image queue; The cross-entropy loss function is calculated based on the obtained similarity when matching the image sample features and text sample features; The matching process of the image sample features and text sample features is optimized by using the obtained cross-entropy loss function.

5. The multimodal sample correction method according to claim 1, characterized in that, The aligned image sample features and text sample features are deeply fused to obtain fused features, including: The aligned image sample features and text sample features are fed into a multimodal fusion module consisting of a multi-layer Transformer encoder; The multimodal fusion module calculates attention weights for the image sample features and text sample features; The image sample features and text sample features are concatenated based on the calculated attention weights and weighted summation to obtain fused features.

6. The multimodal sample correction method according to claim 1, characterized in that, The fused features are fed into a knowledge graph-guided K-adapter model to repair the image and text sample information corresponding to the fused features, including: Acquire large-scale data from the power industry and feed it into a Transformer-based pre-trained model to train the pre-trained model; Insert the K-adapter into the pre-trained model after training; A knowledge graph of the power industry is obtained, and the entities and relations in the knowledge graph are transformed into vector representations and introduced into a pre-trained model with K-adapter inserted, so as to enhance the understanding of power industry knowledge of the pre-trained model. The fused features are fed into the pre-trained model after inserting a K-adapter and enhancing the understanding of power industry knowledge. The pre-trained model performs inference based on the given fused features and knowledge graph information, and completes the repair of the corresponding image sample information and text sample information.

7. A multimodal sample correction system based on a large model, characterized in that, The multimodal sample correction system includes: The data acquisition module is used to acquire image sample information and text sample information; The correction module executes a multimodal sample correction method based on a large model as described in any one of claims 1-6, according to the obtained image sample information and text sample information.

8. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores instructions for causing the machine to perform a multimodal sample correction method as described in any one of claims 1-6.

9. A processor, characterized in that, Used to run a program, wherein the program is run to execute: a multimodal sample correction method as described in any one of claims 1-6.

Images