Multimodal large model mode missing data completion method

Abstract

The application provides a multimodal large model modal missing data completion method, relates to the technical field of data processing, and comprises the following steps: step 1, missing value detection and encoding processing are performed on multimodal input data, the correlation between internal features of the data is analyzed, structured feature representation is generated through feature reorganization, feature calibration parameters are generated in combination with the correlation between local features, and numerical feature vectors and mask matrices of each mode are obtained; step 2, the numerical feature vectors and the mask matrices are input into a feature fusion model, and the context dependence relationship between modes is analyzed, the major axis parameters of the feature distribution ellipse are calculated, the ellipse features are corrected by rotation, and preliminary corrected fusion feature representation is generated. Through feature distribution ellipse correction and global context retrieval, the accuracy and intelligence of multimodal missing data completion are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method for completing missing modal data in a multimodal large model. Background Technology

[0002] In multimodal data processing, modality loss is a common problem. Most existing methods complete the loss by weighting features or simple imputation. For example, some models use attention mechanisms to fuse multimodal features, but they do not fully consider the structural relationships of feature distributions, resulting in limited ability to model complex contextual dependencies. Taking image-text data as an example, when the text modality is missing, existing methods may only generate the completion content based on image features, but ignore the rotation characteristics of the feature distribution ellipse and the consistency of the global pattern, causing the completion result to deviate from the real data distribution. In addition, existing technologies do not adequately calibrate the co-correlation of intramodal and intermodal relationships, which may affect the semantic coherence of multimodal data. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method for completing missing modal data in a multimodal large model. By using feature distribution ellipse correction and global context retrieval, the accuracy and intelligence of completing missing multimodal data are improved.

[0004] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0005] Firstly, a method for completing missing modal data in a multimodal large model, the method comprising:

[0006] Step 1: Perform missing value detection and encoding on the multimodal input data, analyze the correlation between features within the data, generate structured feature representations through feature recombination, and generate feature calibration parameters by combining the correlation between local features to obtain the numerical feature vectors and mask matrices of each modality.

[0007] Step 2: Input the numerical feature vector and mask matrix into the feature fusion model, analyze the contextual dependencies within and between modes, and perform rotation correction on the elliptical features by calculating the principal axis parameters of the feature distribution ellipse to generate a preliminary corrected fusion feature representation.

[0008] Step 3: Based on the preliminarily corrected fusion feature representation, retrieve relevant context patterns from the global multimodal pattern library, and determine the positional relationship between the two circles by comparing the distance between their centers, the sum of their radii, and the absolute value of the difference between their radii, thereby obtaining the enhanced global context vector.

[0009] Step 4: Use the enhanced global context vector to perform a second correction on the initially corrected fusion feature representation to obtain the second-corrected fusion feature representation;

[0010] Step 5: Input the fused feature representation after secondary correction and the global context vector into the prediction model to generate inference results for completing the missing modality data;

[0011] Step 6: Generate the final multimodal completion data based on the inference results, and use the completion data to periodically update the parameters of the global multimodal pattern library and the prediction model to achieve adaptive completion of missing data.

[0012] In a second aspect, a computing device includes:

[0013] One or more processors;

[0014] A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the method.

[0015] Thirdly, a computer-readable storage medium storing a program that, when executed by a processor, implements the method.

[0016] The above-described solution of the present invention has at least the following beneficial effects:

[0017] By detecting missing values, reconstructing features, and calibrating them, incomplete multimodal data is processed to generate high-quality, structured feature representations. By analyzing the contextual dependencies within and between modes and combining elliptical feature rotation correction, the accuracy of feature fusion is improved, and interference caused by missing data is reduced. By retrieving relevant contextual patterns from the global multimodal pattern library and judging positional relationships based on geometric relationships, the understanding of global semantics is enhanced, and the rationality and coherence of the completion results are improved. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the multimodal large model modality missing data completion method provided in an embodiment of the present invention.

[0019] Figure 2 This is a schematic diagram of the process of performing secondary correction on the pre-corrected fusion feature representation using an enhanced global context vector, as provided in an embodiment of the present invention, to obtain a secondary corrected fusion feature representation. Detailed Implementation

[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0021] like Figure 1 As shown, embodiments of the present invention propose a method for completing missing modal data in a multimodal large model, the method comprising the following steps:

[0022] Step 1: Perform missing value detection and encoding on the multimodal input data, analyze the correlation between features within the data, generate structured feature representations through feature recombination, and generate feature calibration parameters by combining the correlation between local features to obtain the numerical feature vectors and mask matrices of each modality.

[0023] Step 2: Input the numerical feature vector and mask matrix into the feature fusion model, analyze the contextual dependencies within and between modes, and perform rotation correction on the elliptical features by calculating the principal axis parameters of the feature distribution ellipse to generate a preliminary corrected fusion feature representation.

[0024] Step 3: Based on the preliminarily corrected fusion feature representation, retrieve relevant context patterns from the global multimodal pattern library, and determine the positional relationship between the two circles by comparing the distance between their centers, the sum of their radii, and the absolute value of the difference between their radii, thereby obtaining the enhanced global context vector.

[0025] Step 4: Use the enhanced global context vector to perform a second correction on the initially corrected fusion feature representation to obtain the second-corrected fusion feature representation;

[0026] Step 5: Input the fused feature representation after secondary correction and the global context vector into the prediction model to generate inference results for completing the missing modality data;

[0027] Step 6: Generate the final multimodal completion data based on the inference results, and use the completion data to periodically update the parameters of the global multimodal pattern library and the prediction model to achieve adaptive completion of missing data.

[0028] In this embodiment of the invention, incomplete multimodal data is processed through missing value detection, feature recombination and calibration to generate high-quality, structured feature representations. By analyzing the contextual dependencies within and between modes and combining elliptical feature rotation correction, the accuracy of feature fusion is improved and interference caused by missing data is reduced. By retrieving relevant contextual patterns from the global multimodal pattern library and judging positional relationships based on geometric relationships, the understanding of global semantics is enhanced, and the rationality and coherence of the completion results are improved.

[0029] In a preferred embodiment of the present invention, step 1 above, which involves detecting and encoding missing values ​​in the multimodal input data, analyzing the correlation between features within the data, generating structured feature representations through feature recombination, and generating feature calibration parameters by combining the correlation between local features, to obtain the numerical feature vectors and mask matrices for each modality, may include:

[0030] In this embodiment of the invention, step 110 involves detecting missing values ​​in the multimodal input data, identifying the locations of missing data in each modality, and generating an initial mask matrix. Specifically, this includes: first, determining the composition of the multimodal input data, covering different types of modal data such as text, images, and audio. Each modal data is stored in a preset structured format, such as text stored at the paragraph-sentence-word level, images stored at the frame-pixel region level, and audio stored at the segment-frequency point level. Next, for each modal data, integrity checks are performed layer by layer and at each data location according to its storage level. For the text modality, each word position is checked for valid characters. Valid characters refer to recognizable characters that are not spaces or empty values. If a position contains consecutive valid characters, it is considered valid data; if the position contains no valid characters or only spaces, it is considered completely missing; if the position contains only some valid characters, it is considered partially missing. For the image modality, each pixel region is checked for recognizable images. For prime values, if a pixel region has a complete pixel value, it is considered valid data; if no pixel value is found, it is considered completely missing; if a pixel value is partially damaged, it is considered partially missing. For audio modalities, each frequency point is checked for spectral amplitude information. If complete amplitude information is found, it is considered valid data; if no amplitude information is found, it is considered completely missing; if amplitude information is partially lost, it is considered partially missing. After determining the missing state of all data locations, an initial mask matrix is ​​generated. The dimensions of the mask matrix correspond one-to-one with the total number of data locations in the multimodal input data. Each matrix element corresponds to the missing state of a data location. Valid data is marked as 1, completely missing data is marked as 0, and partially missing data is marked as a specific value between 0 and 1. This value is determined by the proportion of valid information in the partially missing data. For example, if only the first 50% of valid characters are retained in a text word, it is marked as 0.5; if 60% of the pixel values ​​in an image pixel region are valid, it is marked as 0.6, ensuring that the mask matrix accurately reflects the degree of missing information at each data location.

[0031] Step 111: Based on the initial mask matrix, encode the multimodal input data to convert the original data into numerical feature representations. Specifically, this includes: first, distinguishing the valid states of each data position according to the initial mask matrix; encoding only data positions marked as 1 (fully valid) and between 0 and 1 (partially valid); positions marked as 0 (completely missing) are not encoded, but their missing markers are retained; corresponding encoding methods are used for different types of modal data. For text modalities, a unified keyword dictionary is pre-defined, containing common core keywords in the application scenario. Each keyword corresponds to a unique identification identifier. The text content of each valid data position is read sequentially, and the frequency of each keyword in the text content is counted. If there are words outside the dictionary in the text content, they are marked as 0. The frequency of all keywords is arranged in dictionary order to form a numerical array for the text modal. For images… For each modality, the grayscale values ​​of the effective pixel regions are read (range 0-255) and directly retained as numerical values. These values ​​are arranged sequentially according to the row and column order of the pixel regions. Simultaneously, edge features of each effective pixel region are calculated, such as line length based on the actual number of pixels and the number of corners based on the number of inflection points of pixel changes. The grayscale value sequence is combined with the edge feature values ​​to form a numerical array for the image modality. For the audio modality, the spectral amplitude of the effective frequency points is read. The amplitude values ​​are retained as the original values ​​acquired by the device and arranged in ascending order of frequency. The total duration of the effective audio segment is also calculated. The spectral amplitude sequence is combined with the duration values ​​to form a numerical array for the audio modality. Finally, the numerical arrays of each modality are uniformly adjusted into ordered vectors of fixed length. If the numerical array length is insufficient, zeros are added to the end; if the length exceeds the limit, the first half is truncated, ensuring that the numerical feature representation format and dimensions are consistent across all modalities.

[0032] Step 112 involves analyzing the correlations between features within the numerical feature representation and performing structured reorganization of the numerical features through feature recombination to generate a unified structured feature representation. Specifically, this includes: first, determining each feature dimension in the numerical feature representation, i.e., the feature information corresponding to each element in the numerical vector. For example, the frequency of occurrence of artificial intelligence in the text modality is one feature dimension, and the grayscale value of a pixel in a certain row and column in the image modality is another feature dimension. For each modality's numerical feature representation, two arbitrary feature dimensions are selected, denoted as feature A1 and feature B1, for correlation analysis. The specific process of correlation analysis is as follows: extracting the values ​​of all valid data positions under these two feature dimensions, retaining only the values ​​marked as 1 in the initial mask matrix, and temporarily excluding some valid values ​​from the correlation calculation; statistically analyzing the changes in the value of feature B1 as feature A1 increases. If the number of times feature B1 also increases when feature A1 increases exceeds 50% of the total valid data counts, then a positive correlation is determined; if the number of times feature B1 decreases when feature A1 increases exceeds 50%, then a negative correlation is determined; if there is no obvious pattern in the changes between the two, then a positive correlation is determined. If the correlation is not found, it is considered unrelated. Based on this, the correlation strength is calculated: positive correlation strength = number of positive changes ÷ total number of valid data points; negative correlation strength = number of negative changes ÷ total number of valid data points. The larger of these two values ​​is taken as the final correlation strength for the two feature dimensions. The correlation strength ranges from 0 to 1, with a larger value indicating a stronger correlation. After completing the correlation analysis for all feature dimensions, a correlation strength threshold is set. The threshold is adjusted according to the modality type: 0.6 for text modality, 0.5 for image modality, and 0.55 for audio modality. The core principle is to retain closely related features. Feature dimensions with correlation strengths higher than the threshold are grouped into the same feature group, with each feature group containing multiple closely related feature dimensions. Then, the features within each feature group are sorted, and the range of numerical changes for each feature dimension within the group is calculated. The range of changes = maximum value - minimum value for that feature dimension. A larger range of changes indicates higher feature discrimination. Features within the group are sorted in descending order of range of changes. Finally, all feature groups of all modalities are integrated using a fixed format: feature group number - feature sequence number within the group - feature value, to generate a unified structured feature representation.

[0033] Step 113: Based on the correlation between local features in the structured feature representation, calculate the feature calibration parameters used to adjust the feature representation. Specifically, local features in the structured feature representation refer to all features within each feature group. Features within the same feature group constitute a local feature set. Because features within a group are closely related, they form local feature units. First, for each local feature set, calculate the overall strength of the local features of that set, extract the effective values ​​of all feature dimensions within the group, and include values ​​between 1 and 0-1 in the initial mask matrix. Calculate the magnitude of change for each feature dimension, where the magnitude of change = the value of that feature dimension - the average value of that feature dimension. Sum the absolute values ​​of the magnitudes of change for all feature dimensions and divide by the number of feature dimensions in that feature group to obtain the overall strength of that local feature set. Next, set a target strength, statistically analyze the overall strength of the local features of all feature groups, and divide the sum of all overall strengths by the number of feature groups. The process begins by obtaining the average overall strength, which is the target strength. The target strength aims to balance the strengths of all local feature sets. Next, the relevance weight of each local feature set is calculated. For each feature dimension within a feature set, its correlation strength with all other feature dimensions within the set is calculated. These correlation strengths are summed and divided by the number of feature dimensions within the set to obtain the average correlation strength of that feature dimension. The average correlation strengths of all feature dimensions within the set are then summed and divided by the number of feature dimensions to obtain the relevance weight of that local feature set. The relevance weight reflects the overall correlation strength of features within the set and ranges from 0 to 1. Finally, feature calibration parameters are calculated. The calibration parameter for each local feature set is calculated as: Target Strength ÷ Overall Strength of the Local Feature Set × Relevance Weight of the Local Feature Set. The calibration parameter adjusts the strength of local feature sets to maintain a balance between the strengths of different feature sets while also considering the correlation strength within the set.

[0034] Step 114: Based on the feature calibration parameters, adjust the structured feature representation and obtain the numerical feature vectors and mask matrices for each modality. Specifically, this includes: first, determining the feature calibration parameters corresponding to each local feature set; multiplying the value of each feature in the structured feature representation with the calibration parameters of the feature group to which that feature belongs, i.e., the adjusted feature value = original feature value × corresponding calibration parameter. Each feature corresponds to the calibration parameters of its group, and the multiplication operation is performed one-to-one. This operation adjusts the feature intensity of different feature groups, making the intensity of all feature groups approach the target intensity level. After completing the feature value adjustment, update the mask matrix. According to the feature group order and feature index within the group after feature recombination in Step 112, rearrange the elements in the initial mask matrix according to the new... The feature dimensions are rearranged in order. The missing status markers at each data position in the initial mask matrix are mapped to each feature dimension of the adjusted structured features, ensuring that the dimensions of the updated mask matrix are completely consistent with the dimensions of the adjusted feature representation. Each matrix element retains its original missing status marker: 1 for valid, 0 for completely missing, and 0-1 for partially missing. Finally, the adjusted structured feature representation is processed in the order of feature group number - feature sequence number within the group. The adjusted values ​​of each feature dimension are extracted and arranged in order to form a one-dimensional ordered vector, which is the numerical feature vector of each modality. At the same time, the updated mask matrix is ​​reorganized into a one-dimensional matrix corresponding to the dimensions of the numerical feature vectors. Finally, the numerical feature vectors of each modality and the corresponding mask matrix are output.

[0035] By verifying the integrity of multimodal data position by position, valid data, completely missing data, and partially missing data are identified. The generated initial mask matrix can clearly reflect the missing status of each data position, avoiding processing errors caused by confusion between valid and missing data.

[0036] In a preferred embodiment of the present invention, step 2 above, which involves inputting the numerical feature vector and mask matrix into the feature fusion model, analyzing the intra-modal and inter-modal context dependencies, and performing rotation correction on the elliptical features by calculating the principal axis parameters of the feature distribution ellipse to generate a preliminary corrected fused feature representation, may include:

[0037] In this embodiment of the invention, step 220 involves inputting numerical feature vectors and mask matrices into a feature fusion model. The input data is processed through self-attention computation and cross-modal interaction computation within the feature fusion model to obtain complex intra-modal and inter-modal contextual dependencies, resulting in context-enhanced feature representations. Specifically, this includes: firstly, establishing a complete process for the feature fusion model, which sequentially includes an input receiving stage, an intra-modal association mining stage, an inter-modal association mining stage, and a feature integration output stage. The input receiving stage specifically receives numerical feature vectors and mask matrices corresponding to each modality. The numerical feature vectors are structured feature data obtained in step 1 after feature recombination and calibration parameter adjustment; each vector contains all valid feature information for the corresponding modality. The mask matrix has the same dimension as the numerical feature vectors, with 1 indicating missing data and 0 indicating valid data, used to filter invalid information in subsequent calculations. The intra-modal association mining stage handles feature associations for a single modality, the inter-modal association mining stage establishes feature associations for different modalities, and the feature integration output stage summarizes all processed features and forms a unified output.

[0038] The numerical feature vector of a single modality and the mask matrix of that modality are fed together into the intra-modal association mining stage. For each feature element to be processed in that modality, the original association value between the target feature element and all other feature elements in the same modality is calculated. Other feature elements in the same modality are selected one by one, and the value of the target feature element is multiplied by the value of the currently selected other feature elements to obtain a pairwise product. All these product results are the original association values ​​between the target feature element and all other feature elements. Referring to the mask matrix, the mask position corresponding to each other feature element is checked. If the mask position is 1, the original association value between that feature element and the target feature element is removed, and only the original association values ​​corresponding to feature elements with a mask position of 0 are retained. All retained valid values ​​are counted. The sum of the original association values ​​is used to divide each valid original association value by this sum to obtain the attention weight of each valid feature element relative to the target feature element. The intra-modal enhancement value of the target feature element is calculated by multiplying the value of each valid feature element by its corresponding attention weight to obtain multiple weighted values, and then summing all these weighted values ​​to obtain the intra-modal enhancement value of the target feature element. Following the above process, each feature element of the modality is processed in turn to obtain the intra-modal enhancement value of each feature element. All enhancement values ​​are arranged in the order of the original feature elements to form the intra-modal enhancement feature vector of the modality. The above self-attention calculation process is performed on all modalities such as text, image, and audio to obtain the intra-modal enhancement feature vector of each modality.

[0039] The intra-modal enhancement feature vectors of all modalities and their corresponding mask matrices are fed into the inter-modal association mining stage. For each different modal combination, one modality is designated as the target modality and the other as the associated modality. The cross-modal raw association values ​​between the target feature elements and the associated feature elements are calculated. Each feature element in the target modality is selected one by one, and then each valid feature element with a mask marker of 0 in the associated modality is selected one by one. The value of the target cross-modal element is multiplied by the value of each associated element to obtain multiple cross-modal raw association values. The cross-modal elements of the target modality are statistically analyzed. The sum of all corresponding original cross-modal association values ​​is used to divide each original association value by this sum, yielding the cross-modal attention weight of each association element relative to the target cross-modal element. The cross-modal enhancement value of the target cross-modal element is calculated by multiplying the value of each association element by its corresponding cross-modal attention weight, resulting in multiple weighted cross-modal values. These values ​​are then summed to obtain the cross-modal enhancement value of the target cross-modal element. This process is repeated for each feature element of the target modality to obtain the cross-modal enhancement value for each feature element. The values ​​are arranged in the original feature element order to form the cross-modal augmented feature vector of the target modality for the current associated modality; the cross-modal interaction results of the target modality with all other modalities are integrated, and the cross-modal augmented feature vectors of the target modality for different associated modalities are concatenated in the order of text association, image association, and audio association to obtain the comprehensive cross-modal augmented feature vector of the target modality; for all modalities as target modalities, the above cross-modal interaction calculation process is performed to obtain the comprehensive cross-modal augmented feature vector of each modality; for each modality The intra-modal augmented feature vector and the comprehensive cross-modal augmented feature vector are merged one by one according to the order of the feature elements. That is, the intra-modal augmentation value of the first feature element of the modality is immediately followed by its comprehensive cross-modal augmentation value, the intra-modal augmentation value of the second feature element is immediately followed by its comprehensive cross-modal augmentation value, and so on, to form the complete context augmented feature vector of the modality. Then, the complete context augmented feature vectors of all modalities are concatenated in a fixed order of text modality, image modality, and audio modality to obtain a global context augmented feature representation that covers all modal association information.

[0040] Step 221: Calculate the distribution characteristics of the context-enhanced feature representation in the feature space, and construct the corresponding covariance matrix using the distribution characteristics; perform eigenvalue decomposition on the covariance matrix, and determine the principal axis parameters of the feature distribution ellipse based on the decomposition results, including the direction vector and length value of the major axis in the original feature space, and the direction vector and length value of the minor axis in the original feature space; specifically, this includes: first determining the total number of dimensions of the context-enhanced feature representation, then for each dimension, statistically analyzing the feature values ​​of all samples in that dimension, summing all the feature values ​​of all samples in that dimension, and then dividing by the total number of samples to obtain the feature mean of that dimension, i.e., the average level of the feature values ​​of that dimension; calculate the feature mean of each of the N dimensions in sequence according to the above method; calculate the correlation between dimensions, for any two Different dimensions, denoted as dimension X1 and dimension Y1, are used to calculate the product of biases for each sample. For each sample, the bias in dimension X1 is obtained by subtracting the mean of the features in dimension X1 from the sample's feature value in dimension X1. Similarly, the bias in dimension Y1 is obtained by subtracting the mean of the features in dimension Y1 from the sample's feature value in dimension Y1. These two biases are then multiplied to obtain the product of biases between dimensions X1 and Y1. The degree of correlation between dimensions is calculated by summing all the products of biases between dimensions X1 and Y1 and dividing by the total number of samples. This yields the degree of correlation between dimensions X1 and Y1, i.e., the synchronicity of changes in the features of the two dimensions. This calculation is performed on all possible combinations of dimensions to obtain the degree of correlation between all combinations of dimensions.

[0041] Construct an N x N matrix. The element in the i-th row and j-th column corresponds exactly to the correlation between dimensions i and j. For example, the element in the first row and first column represents the correlation between dimensions 1 and j, i.e., the variance of dimension 1, calculated as the sum of the squares of the biases of all samples in that dimension divided by the number of samples. The element in the first row and second column represents the correlation between dimensions 1 and j. The element in the second row and first column has the same value as the element in the first row and second column, and so on, filling all positions in the matrix to form a complete covariance matrix. Perform eigenvalue decomposition on the constructed covariance matrix. This process uses mathematical operations to find multiple eigenvalues ​​that represent the core features of the matrix, as well as the eigenvector corresponding to each eigenvalue. Arrange all the obtained eigenvalues ​​in descending order, and select the two largest eigenvalues. These are called the first principal eigenvalue and the second principal eigenvalue. The corresponding eigenvectors for these two eigenvalues ​​are also selected and called the first principal eigenvector and the second principal eigenvector, respectively. The first principal eigenvector is defined as the direction vector of the major axis of the feature distribution ellipse. Each element of this vector represents the direction of the major axis in the corresponding feature dimension. The length of the major axis is calculated by first finding the square root of the first principal eigenvalue and then multiplying that square root by 2, resulting in a major axis that is twice the length of the square root. The second principal eigenvector is defined as the direction vector of the minor axis of the feature distribution ellipse. Each element of this vector represents the direction of the minor axis in the corresponding feature dimension. The length of the minor axis is calculated by first finding the square root of the second principal eigenvalue and then multiplying that square root by 2. Finally, the direction vectors and lengths of the major and minor axes of the feature distribution ellipse are obtained, thus completing the acquisition of the principal axis parameters.

[0042] Step 222: Based on the major axis direction vector in the principal axis parameters, calculate the rotation transformation matrix required to rotate the major axis of the feature distribution ellipse to the horizontal direction. Then, use this rotation transformation matrix to perform an overall rotation transformation on the context-enhanced feature representation, aligning the major axis of the feature distribution ellipse with the horizontal direction and the minor axis with the vertical direction, thus obtaining the rotated and corrected elliptical feature. Specifically, this includes: first, extracting the horizontal component of the major axis direction vector (corresponding to the element values ​​along the x-axis of the feature space) and the vertical component (corresponding to the element values ​​along the y-axis of the feature space); then, determining the deflection angle of the major axis relative to the horizontal direction (x-axis) based on the proportional relationship between the horizontal and vertical components. If both the horizontal and vertical components are positive, the deflection angle is the angle of counterclockwise rotation from the positive x-axis direction to the major axis direction; if both are negative, the deflection angle is the angle of clockwise rotation from the positive x-axis direction to the major axis direction. The angle along the major axis is used to ensure that it accurately reflects the actual deflection direction of the major axis. The rotation transformation matrix is ​​a 2x2 matrix. Each element of the matrix is ​​determined by the cosine and sine values ​​of the deflection angle. The first element in the first row of the matrix is ​​the cosine value of the deflection angle, calculated by dividing the horizontal component of the major axis vector by the total length of the major axis vector. The total length of the major axis vector is obtained by taking the square root of the sum of the squares of the horizontal and vertical components. The second element in the first row of the matrix is ​​the negative sine value of the deflection angle, calculated by dividing the vertical component of the major axis vector by the total length of the major axis vector and then adding a negative sign before the sine value. The first element in the second row of the matrix is ​​the sine value of the deflection angle, and the second element in the second row of the matrix is ​​the cosine value of the deflection angle. By determining the four elements of the matrix in the above manner, the complete rotation transformation matrix is ​​constructed.

[0043] Since context-enhanced feature representations are high-dimensional vectors, they need to be grouped according to their dimensions, with each group consisting of two dimensions. Each group corresponds to a two-dimensional plane in the feature space, ensuring that the feature distribution of each group can be corrected through rotation transformation. For each group of two-dimensional feature vectors, one feature vector within the group is selected as the vector to be rotated, containing two elements. The first element of the rotated vector is calculated by multiplying the first element of the vector to be rotated by the first element of the first row of the rotation transformation matrix, and then multiplying the second element of the vector to be rotated by the second element of the first row of the rotation transformation matrix. These two products are then added together to obtain the first element of the rotated vector. The second element of the rotated vector is calculated by multiplying the first element of the vector to be rotated by... The first element of the second row of the rotation transformation matrix is ​​multiplied by the second element of the vector to be rotated, and the two products are added together to obtain the second element of the rotated vector. This rotation calculation is then performed on all feature vectors within the group to obtain the rotated two-dimensional feature vectors for that group. All rotated two-dimensional feature vectors are then concatenated in their original dimensional order to restore the complete high-dimensional feature vector. For each sample in the context-enhanced feature representation, the above grouping rotation and concatenation operations are performed on its corresponding high-dimensional feature vector to complete the overall rotation transformation, ensuring that the major axis of the feature distribution ellipse for all samples is aligned with the horizontal direction and the minor axis with the vertical direction, resulting in the rotated and corrected elliptical feature vector.

[0044] Step 223: Input the rotation-corrected elliptical features into a fully connected layer. The fully connected layer performs nonlinear transformations on the rotation-corrected features, adjusting dimensions and compressing features, to generate a preliminary corrected fused feature representation. Specifically, this includes: constructing a fully connected processing flow containing two linear transformation layers and one nonlinear activation layer. In the first linear transformation layer, a weight matrix and a bias vector are set. The number of rows in the weight matrix is ​​exactly the same as the total dimension of the rotation-corrected elliptical features, and the number of columns is set to the target intermediate dimension. The initial values ​​of the weight matrix are randomly generated through a uniform distribution, with each value falling between -0.1 and 0.1, and each element's value is independent of the others. To avoid excessive initial weight bias, the bias vector has the same dimension as the weight matrix, with each element initialized to 0.05 to compensate for insufficient representation due to small feature values. A Sigmoid activation method is used specifically for range compression and nonlinear mapping of the linearly transformed feature values, enhancing feature discriminative power. In the second linear transformation stage, a weight matrix and a bias vector are set. The number of rows in the weight matrix matches the target intermediate dimension of the first linear transformation, while the number of columns is set to the target dimension of the final preliminary correction and fusion features. This target dimension is half the target intermediate dimension of the first layer, further simplifying the dimensionality. To reduce subsequent computational pressure, the initial values ​​of the weight matrix are also randomly generated through a uniform distribution, ranging from -0.1 to 0.1, with each element's value being independent. The dimension of the bias vector is the same as the number of columns in the weight matrix, and both are initially set to 0.05. In the first linear transformation, the rotated and corrected elliptical features are input into the first linear transformation stage. For each sample's feature vector, the following operations are performed: each element in the feature vector is selected one by one and denoted as the current element; all column elements in the weight matrix corresponding to the row containing the current element are found, i.e., the current row of the weight matrix; the current element is multiplied by the first column element of the current row of the weight matrix to obtain the first product result. The result is as follows: Multiply the current element by the second column element of the current row of the weight matrix to obtain the second product result; In this way, multiply the current element by all column elements of the current row of the weight matrix to obtain a set of product results; Add all of these product results to obtain the weighted sum corresponding to the current element; Find the corresponding element in the bias vector, add the weighted sum to the corresponding element of the bias vector to obtain the value of the current element after the first linear transformation; Traverse all elements of the feature vector and perform linear transformation on all elements to obtain the intermediate dimension feature vector of the sample; Perform the above operation on the elliptical feature vector of all samples to obtain the overall intermediate dimension feature representation.

[0045] The intermediate dimension feature representation is input into the nonlinear activation stage. For each sample's intermediate dimension feature vector, the following operations are performed: each element in the vector is selected, and its value is transformed using the Sigmoid activation method, mapping the value to a range between 0 and 1. The larger the value, the closer it is to 1 after mapping; the smaller the value, the closer it is to 0. This process not only compresses the range of feature values, avoiding overflow in subsequent calculations due to excessively large values, but also enhances the differences between different features through nonlinear transformation, improving the expressive power of the features. The intermediate dimension feature vectors of all samples are traversed, and the nonlinear transformation is completed to obtain the nonlinear enhanced feature representation. A second linear transformation and preliminary correction are then fused to generate the feature representation, transforming the nonlinear... The enhanced feature representation is input into the second-layer linear transformation stage. Following the operation method of the first-layer linear transformation, the following operations are performed on the feature vector of each sample: each element after nonlinear enhancement is selected and multiplied by all column elements of the corresponding row of the second-layer weight matrix to obtain a set of product results; all these product results are added together to obtain a weighted sum, which is then added to the corresponding element of the second-layer bias vector to obtain the value of the element after the second linear transformation; all elements of the feature vector are traversed to complete the second linear transformation, resulting in the final target dimension feature vector of the sample; the above operations are performed on the feature vectors of all samples, and all final target dimension feature vectors are arranged in sample order to form a preliminary corrected fusion feature representation.

[0046] By employing self-attention computation and cross-modal interactive computation, we first delve into the local dependencies between each feature and other features within a single modality, and then comprehensively establish global semantic associations between features across different modalities. Furthermore, during the computation process, we precisely filter out interference from missing data using a mask matrix to ensure the accuracy of the association computation. This enables context-enhanced feature representations to fully integrate the local details and global association information of multimodal data, avoiding the problem of insufficient feature representation caused by the one-sidedness of information from a single modality.

[0047] In a preferred embodiment of the present invention, step 3 above, based on the preliminarily corrected fusion feature representation, retrieves relevant context patterns from the global multimodal pattern library, determines the positional relationship between the two circles by comparing the distance between their centers, the sum of their radii, and the absolute value of the difference in their radii, and obtains the enhanced global context vector, may include:

[0048] In this embodiment of the invention, step 330 involves using the preliminarily corrected fusion feature representation as a query feature, and retrieving multiple similar context patterns from the global multimodal pattern library based on the query feature to form a candidate context pattern set. Specifically, this includes: firstly, building a complete system for the global multimodal pattern library, which includes data collection and filtering, feature standardization processing, pattern classification and storage, and index construction. In the data collection and filtering stage, a large amount of complete, non-missing multimodal data from different application scenarios is collected, with each data point covering at least two or more core modalities. The collected data undergoes quality screening, removing samples with data distortion, semantic contradictions, or format abnormalities, while retaining complete and semantically coherent valid samples as the original data foundation for the pattern library. In the feature standardization processing stage, the filtered original multimodal data is processed step by step according to the complete process from step 1 to step 2. First, missing value detection, encoding processing, feature recombination, and calibration parameter calculation are performed on each data point to obtain numerical feature vectors for each modality. Then, the numerical feature vectors and mask matrix are input into the feature fusion process, and self-attention calculation is performed. The process involves cross-modal interactive computation, calculation of the principal axis parameters of the feature distribution ellipse, rotation transformation, and fully connected layer processing to ultimately generate a pattern feature vector with dimensions completely consistent with the initially corrected fused feature representation. Simultaneously, each piece of original data is labeled with a scene tag and bound to its corresponding pattern feature vector. In the pattern classification and storage stage, all bound original data + pattern feature vector + scene tag combinations are classified and stored according to scene tags. An independent storage directory is established under each scene category, with data arranged in chronological order for easy subsequent scene-based filtering. Each pattern feature vector is assigned a unique identifier number and associated with its corresponding original data storage path to ensure traceability. In the index construction stage, a multi-dimensional hierarchical indexing approach is used to build a fast retrieval system. First, each dimension of the pattern feature vector is divided into multiple intervals based on its numerical range. Then, an index code is assigned to each interval of each dimension. Each pattern feature vector generates an index sequence composed of multiple dimension index codes based on the interval to which its numerical values ​​belong in each dimension. Finally, a mapping table between the index sequence and the pattern feature vector identifier number is established and stored in the index database.

[0049] The pre-corrected fusion feature representation is used as the query feature. Preprocessing operations consistent with those in the pattern feature vector library are performed, extracting the value of each dimension of the query feature vector. The value of that dimension is compared to the maximum value of all pattern feature vectors in the pattern library. The value of the query feature for that dimension is divided by this maximum value to obtain the standardized value of that dimension. All dimensions are processed in this manner to ensure that the value of each dimension of the query feature vector is consistent with the value range of the feature vectors in the pattern library, eliminating similarity calculation biases caused by differences in value ranges. Based on the application scenario of the original multimodal data corresponding to the query feature, all pattern feature vectors under the same scene label are first selected from the global multimodal pattern library. If the number of pattern feature vectors under the same scene is less than 50, pattern feature vectors under adjacent scene labels are further selected to ensure that the initial selection range includes a sufficient number of candidate patterns. For each selected pattern feature vector, similarity is calculated with the pre-processed query feature vector. The specific process is as follows: Dimensional difference calculation is performed, extracting the first dimension value of the query feature vector, then extracting the first dimension value of the current pattern feature vector, and using the query feature... Subtracting the value of the pattern feature vector in the first dimension from the value of the first dimension yields the difference. This process is repeated for all corresponding dimensions of the two vectors, ensuring each dimension yields an independent difference. The difference is then squared and summed by multiplying the difference in each dimension by itself. The sum of the squared differences across all dimensions is then obtained, representing the total sum of the squared differences for each dimension. For similarity assessment, a smaller total sum of squares indicates a smaller difference in the dimensional values ​​of the two vectors, indicating higher similarity; a larger total sum of squares indicates a lower similarity. The lower the similarity, the more accurate the similarity determination. All pattern feature vectors with calculated total sum of squares are sorted in ascending order of total sum of squares, i.e., sorted in descending order of similarity. The top 20 pattern feature vectors are selected to correspond to the context patterns, which include the original data fragments, pattern feature vectors, and scene labels, and integrated to form a candidate context pattern set. If the number of patterns that meet the conditions after sorting is less than 20, they are supplemented in ascending order of total sum of squares until the candidate set contains 20 context patterns, ensuring that the number of candidate sets is sufficient.

[0050] Step 331: For each context pattern in the candidate context pattern set, calculate the geometric relationship parameters between the feature representation circle corresponding to the context pattern and the query feature representation circle, including the distance between the centers of the two circles, the sum of the radii of the two circles, and the absolute value of the difference between the radii of the two circles. Specifically, this includes: determining the center coordinates, extracting the preprocessed query feature vector, extracting the values ​​of all its dimensions one by one, summing all these values ​​to obtain the sum of all dimension values; dividing this sum by the total number of dimensions of the query feature vector, i.e., the number of dimensions, to obtain the average value of all dimension values; this average value serves as the core value of the center coordinates of the query feature representation circle. In high-dimensional space, each dimension corresponds to a coordinate component of the center, and the average values ​​of all dimensions together constitute the complete coordinates of the center, such as the three-dimensional feature vector. The center coordinates are a combination of the average values ​​of the three dimensions. To determine the radius, first calculate the variance of each dimension of the query feature vector, extract the value for that dimension, find the single value corresponding to that dimension of the query feature vector, combine it with all historical valid values ​​for that dimension in the original data sample set, subtract the average value of that dimension from each value (i.e., the component of the dimension corresponding to the center coordinates), and obtain the deviation of each value. Multiply each deviation by itself to obtain the square of each deviation. Sum all the squared deviations and divide by the total number of values ​​for that dimension (i.e., the number of historical valid samples) to obtain the variance for that dimension. Sum all the variances of all dimensions to obtain the total variance. Divide the total variance by the total number of dimensions of the query feature vector to obtain the average variance. Take the square root of this average to obtain the radius of the circle representing the query feature.

[0051] For each context pattern in the candidate context pattern set, perform the same operation as calculating the parameters of the query feature representation circle. Once the center coordinates are determined, extract the pattern feature vector of that context pattern, sum all dimension values, divide the sum by the total number of dimensions to obtain the average value of each dimension, and use this average value to form the center coordinates of the pattern feature representation circle. Once the radius is determined, calculate the variance of each dimension of the pattern feature vector, combine it with all historical valid values ​​of that dimension in the sample set corresponding to the original data of that pattern, sum all dimension variances and divide by the total number of dimensions to obtain the average variance. Take the square root of the average to obtain the radius of the pattern feature representation circle. Calculate the geometric relationship parameters item by item, using the query feature representation circle as the base circle, denoted as circle O, and the feature representation circle of each context pattern as the comparison circle, denoted as circle P. Calculate the three core geometric relationship parameters sequentially, including the distance between the centers of the two circles. Extract the coordinate components of each dimension of the center of circle O and the corresponding coordinate components of the center of circle P. Subtracting the coordinate component of circle P from the coordinate component of this dimension yields the coordinate difference for each dimension. Multiplying each coordinate difference by itself yields the square of the coordinate difference. Summing all the squared coordinate differences yields the total sum of squares. Taking the square root of the total sum of squares yields the distance between the centers of the two circles. The sum of the radii is obtained by taking the radius values ​​of circle O and circle P, adding them together. The absolute value of the difference between the radii is obtained by comparing the radius values ​​of circle O and circle P, subtracting the smaller radius from the larger one, and ignoring the sign of the difference, keeping only the value itself. A parameter record table is created for each context pattern, recording the center coordinates and radius of its feature representation circle, as well as three geometric relationship parameters with the queried feature representation circle. The parameter record table is then bound to the identifier number of the context pattern.

[0052] Step 332: Based on geometric relationship parameters, by comparing the relationship between the distance between the centers of two circles and the sum of their radii, and the relationship between the distance between the centers of two circles and the difference between their radii, determine the positional relationship category between each context pattern and the query feature, including disjoint, tangent, or intersecting. Specifically, this includes: retrieving the parameter record table for each context pattern, extracting three geometric relationship parameters: the distance between the centers of the two circles (D), the sum of the radii of the two circles (S), and the absolute value of the difference between the radii of the two circles (T). Ensure the accuracy of the three parameters, and execute the judgment process in a fixed order to ensure consistent judgment logic. The first step is to determine the disjoint relationship by comparing the center distance D with the sum of the radii S. If the value of D is greater than the value of S, it means that the two circles have no intersection point, and one circle is completely outside the other circle. The positional relationship between the context pattern and the query feature is determined to be disjoint, indicating that the feature correlation between the two is extremely low, with almost no effective correlation information. The second step is to determine the tangent relationship. If the first step did not determine disjointness, the center distance D is then compared with the sum of the radii S and the absolute value of the difference between the radii T. The first step is to compare the values ​​of D and S. If the values ​​of D and S are equal, then the two circles have exactly one common point, and one circle is outside the other. Alternatively, if the values ​​of D and T are equal, then the two circles have exactly one common point, and one circle is inside the other. In this case, the positional relationship between the context pattern and the query feature is determined to be tangent, indicating a moderate degree of correlation. The second step is to determine the intersection relationship. If neither the first nor the second step yields a result, i.e., the value of D is neither greater than S nor equal to S or T, then the relationship between D and T, and between D and S is further compared. If the value of D is greater than the value of T and the value of D is less than the value of S, it means that the two circles have two common points and some areas overlap. In this case, the positional relationship between the context pattern and the query feature is determined to be intersecting, indicating a very high degree of correlation. The overlapping area corresponds to the core correlation feature. After determining the positional relationship for each context pattern, a corresponding positional relationship category label is added to its parameter record table, and the patterns in the candidate context pattern set are temporarily grouped according to the category label.

[0053] Step 333: Based on the positional relationship category, filter out contextual patterns from the candidate contextual pattern set that are tangent or intersecting with the positional relationship of the query feature, forming a relevant contextual pattern subset. Specifically, this includes: traversing all patterns with positional relationship category labels in the candidate contextual pattern set, examining the label of each pattern one by one; retaining only patterns labeled as tangent or intersecting, directly excluding patterns labeled as disjoint; integrating the retained tangent and intersecting patterns together to form a temporary subset, counting the number of patterns contained in the temporary subset, and determining whether it meets the minimum requirements for subsequent feature fusion. If the number of patterns in the temporary subset reaches or exceeds 3, it indicates that the subset contains enough valid associated patterns, and the temporary subset is directly determined as the relevant contextual pattern subset. If the number of patterns in the temporary subset is less than 3, the current subset is determined to be invalid, and a supplementary search process needs to be executed; supplementary search and filtering expand the search scope, returning to the global multi-mode search. The modal pattern library removes scene label restrictions. From the pattern feature vectors under all scene categories, the top 50 context patterns with the highest similarity are selected according to the similarity calculation method in step 330. The remaining 30 patterns are added as supplementary candidate patterns. Supplementary pattern parameter calculation: For each of the 30 supplementary candidate patterns, following the complete process in step 331, the center coordinates and radius of its feature representation circle, as well as the three geometric relationship parameters with the query feature representation circle, are calculated one by one. Supplementary pattern positional relationship judgment: According to the judgment rules in step 332, the positional relationship category of each supplementary candidate pattern is judged, and supplementary patterns labeled as tangent or intersecting are selected. Subset expansion and determination: The selected effective supplementary patterns are added to the previous temporary subset, and the number of subsets is counted again. If the number reaches 3 or more, the subset is determined as a relevant context pattern subset; if it is still less than 3, the supplementary retrieval process is repeated until the number of subsets meets the requirement.

[0054] Step 334: Based on the relevant context pattern subset, extract the feature vector representation of each context pattern in the relevant context pattern subset, and combine the closeness of the positional relationship between the context pattern and the query feature to generate an enhanced global context vector through feature fusion processing. Specifically, this includes: traversing the relevant context pattern subset, extracting the corresponding pattern feature vector for each pattern. This vector is generated during the pattern library construction process in steps 1-2 and has the same dimension as the initially corrected fused feature representation. During the extraction process, the integrity of the vector needs to be verified to ensure that the value of each dimension is valid. If an invalid vector is found, the pattern is immediately removed from the subset, and a valid pattern is selected from the supplementary candidate patterns to replace it, ensuring that all extracted feature vectors have complete association information. Based on the positional relationship category between each pattern and the query feature, combined with the similarity value, differentiated association weights are assigned. The specific process is as follows: setting a basic weight ratio for different positional relationship categories. For example, the base weight of intersecting patterns is higher than that of tangent patterns to reflect the higher degree of correlation between intersecting patterns. If there are multiple intersecting patterns in the subset, first calculate the similarity between each intersecting pattern and the query feature; take the reciprocal of the sum of squares of the similarities of all intersecting patterns, and then add all the reciprocal results to get the sum of reciprocals; the subdivision weight of each intersecting pattern is the reciprocal result of that pattern divided by the sum of the reciprocals; finally, multiply the subdivision weight by the base weight of the intersecting pattern to get the final weight of that intersecting pattern. If there are multiple tangent patterns in the subset, calculate the similarity between each tangent pattern and the query feature in the same way, calculate the sum of the reciprocals to get the subdivision weight of each tangent pattern, and then multiply it by the base weight of the tangent pattern to get the final weight of that tangent pattern. Add the final weights of all patterns to ensure that the total weight is 1. If it is not 1, readjust it according to the weight ratio of each pattern to make the total meet the requirements and ensure the rationality of the weight allocation.

[0055] Dimension traversal and numerical extraction: First, determine the total number of dimensions of the feature vectors. Then, starting from the first dimension, perform fusion processing one by one. For the current dimension, extract the feature vector values ​​of all patterns in the relevant context pattern subset for that dimension, and simultaneously extract the final weight corresponding to each pattern. Dimensional numerical weighting calculation: Multiply the value of the first pattern in that dimension by its final weight to obtain the first weighted value; then multiply the value of the second pattern in that dimension by its final weight to obtain the second weighted value; repeat this process for all patterns in that dimension, multiplying their values ​​by their weights to obtain a set of weighted values; sum all these weighted values ​​to obtain the fused value for that dimension; full-dimensional fusion is complete, and the process is repeated dimension-by-dimensional... The process involves sequentially extracting, weighting, and summing numerical values ​​across all dimensions of the feature vector to obtain the fused value for each dimension. These fused values ​​are then arranged in the original feature vector's dimensional order to form a complete high-dimensional fused vector. The global context vector generation is enhanced by normalizing the high-dimensional fused vector to ensure its stability and comparability. The sum of the fused values ​​across all dimensions is calculated. Each dimension's fused value is then divided by this sum until the sum of all dimensions equals 1. The normalized high-dimensional vector is the enhanced global context vector, which comprehensively integrates the core association information of all relevant context patterns and highlights the dominant role of highly associated patterns.

[0056] By constructing a global multimodal pattern library containing complete data from multiple scenarios, and through standardized preprocessing and hierarchical index design, combined with precise similarity calculation and scenario filtering, we ensure that the candidate context pattern set has potential correlation with the query features. By refining the calculation of the center, radius and geometric relationship parameters of the feature representation circle in each dimension, we realize the quantification of the degree of feature correlation, avoid the ambiguity and subjectivity of correlation calculation, and improve the semantic coherence and overall accuracy of missing modality data completion.

[0057] like Figure 2 As shown, in a preferred embodiment of the present invention, step 4 above, which uses the enhanced global context vector to perform a secondary correction on the initially corrected fusion feature representation to obtain a secondary corrected fusion feature representation, may include:

[0058] In this embodiment of the invention, step 440 involves determining a reference direction for feature correction based on the enhanced global context vector, establishing a correction direction benchmark, and mapping the initially corrected fused feature representation to the feature space according to the correction direction benchmark to generate feature representation line segments. Specifically, this includes: extracting the correction reference direction; retrieving the enhanced global context vector, which is a high-dimensional vector integrating core information of relevant context patterns from a global multimodal pattern library, where the value of each dimension represents the standard level of that dimension in a globally reasonable distribution; for each feature dimension, performing the following operations: directly extracting the value of the enhanced global context vector in that dimension and using it as the initial correction reference direction value for that dimension; and determining the initial value. For positive and negative attributes, if the value is positive, the correction direction for that dimension is to adjust in the direction of increasing value; if the value is negative, the correction direction is to adjust in the direction of decreasing value; if the value is 0, the correction direction is to keep the current value unchanged. Determine the correction priority for that dimension, using the absolute value of the initial correction reference direction value as the priority criterion. The larger the absolute value, the greater the deviation of that dimension from the global reasonable distribution, and the higher the correction priority; the smaller the absolute value, the lower the correction priority. Traverse all feature dimensions to complete the determination of the initial correction reference direction value, correction direction, and correction priority for each dimension, forming a four-dimensional reference direction list containing dimension number, initial reference value, correction direction, and correction priority.

[0059] The calibration direction benchmark is established by standardizing the initial reference values ​​in the four-dimensional reference direction list to construct a unified and quantifiable calibration direction benchmark vector. The specific process is as follows: 1. Calculate the total absolute value sum: Extract the initial calibration reference direction values ​​for all dimensions from the list, calculate the absolute value of each value individually, and then sum all the absolute values ​​to obtain the total absolute value sum of the initial reference values ​​for all dimensions. 2. Assign dimension benchmark weights: For each dimension, divide the absolute value of the initial reference value for that dimension by the total absolute value sum to obtain the benchmark weight for that dimension. For example, if the absolute value of the initial reference value for a certain dimension is 0.6 and the total absolute value sum is 3, then the benchmark weight for that dimension is 0.6 divided by 3, resulting in 0.2. 3. The sum of the benchmark weights for all dimensions is 1 to ensure that the weights are equal. Reasonable allocation; Calculate the baseline components of each dimension by multiplying the initial correction reference direction value of each dimension by the baseline weight of that dimension to obtain the baseline component of that dimension. For example, if the initial reference value of a dimension is 0.8 and the baseline weight is 0.2, then the baseline component of that dimension is 0.8 multiplied by 0.2, resulting in 0.16; if the initial reference value is -0.5 and the baseline weight is 0.1, then the baseline component is -0.5 multiplied by 0.1, resulting in -0.05; Generate the correction direction baseline vector by arranging the baseline components of all dimensions in the order of the original feature dimensions to form a complete correction direction baseline vector. This vector retains the correction direction of each dimension and balances the correction priority of different dimensions through the baseline weight, avoiding over- or under-correction of a single dimension.

[0060] Feature space mapping preparation involves unifying spatial dimensions. First, it's confirmed that the fusion feature representation denoted as vector X and the calibration direction reference vector denoted as vector Y have the same number of dimensions. If there's a dimension mismatch, the average value of that dimension in the global multimodal pattern library is used to fill the missing dimensions of vector X, ensuring dimensional consistency. Mapping rules are then established, creating a one-to-one dimension mapping rule: the first dimension of vector X maps only to the first dimension of vector Y, the second dimension of vector X maps only to the second dimension of vector Y, and so on. Each dimension corresponds to only one mapping object, avoiding spatial chaos caused by cross-dimensional mapping. Spatial consistency is verified by extracting the values ​​of the three key dimensions from vectors X and Y. It's confirmed that both values ​​fall within a reasonable range for that dimension in the global multimodal pattern library, i.e., between the minimum and maximum values ​​of all pattern feature vectors in that dimension. If a key dimension value of vector X exceeds this range, it's adjusted to the reasonable range boundary; if it's below the minimum, the minimum value is used; if it's above the maximum, the maximum value is used, ensuring both are in the same feature space and have a mapping basis.

[0061] The feature representation line segment generation process determines the coordinates of the line segment endpoints. Vector X is used as the starting point A of the line segment. The values ​​of all dimensions of vector X represent the coordinates of starting point A in each dimension of the feature space. For example, if the value of vector X in the first dimension is 0.3 and the value in the second dimension is 0.5, then the coordinates of starting point A in the first dimension are 0.3 and the coordinates in the second dimension are 0.5. Vector Y is used as the ending point B of the line segment. The values ​​of all dimensions of vector Y represent the coordinates of ending point B in each dimension. A complete line segment is constructed by connecting starting point A and ending point B in a unified feature space according to the correspondence of coordinates in each dimension, forming the feature representation line segment AB. Each point of this line segment corresponds to a set of mixed feature states of local features and global references. The length and direction of the line segment reflect the degree of deviation between vector X and the globally reasonable distribution, providing a clear range boundary for the selection of subsequent correction points.

[0062] Step 441: Calculate the intersection point of the feature representation line segment and the correction direction reference in the feature space, determine the feature projection point, and locate the final correction point on the feature representation line segment based on the feature projection point. Specifically, this includes: calculating the feature projection point, which is the perpendicular projection of vector X onto the line containing the correction direction reference vector Y, and this point must fall on the feature representation line segment AB. The calculation process is as follows: calculate the basic parameters dimension by dimension, and for each feature dimension, extract the value of vector X in that dimension (denoted as X). i The value of vector Y in that dimension (denoted as Y). i ); calculate the sum of squares of vector Y, and sum the squares of Y in each dimension. i Multiply by itself (to obtain Y) iThe square of vector X is calculated by multiplying the square of X by the square of Y in each dimension. Then, the squares of all dimensions are summed to obtain the total sum of squares for vector Y. For example, if Y has three dimensions with values ​​of 0.2, 0.4, and 0.6, the squares of each dimension are 0.04, 0.16, and 0.36, respectively. The total sum of squares is 0.04 + 0.16 + 0.36, resulting in 0.56. The total product of vectors X and Y is calculated by multiplying Xᵢ in each dimension by the corresponding value of Y. i First, obtain the XY product value for that dimension. Then, sum the XY product values ​​for all dimensions to get the total XY product. For example, if the dimensions corresponding to X are 0.1, 0.3, and 0.5, the product values ​​for each dimension are 0.02, 0.12, and 0.3, respectively. The total product is 0.02 plus 0.12 plus 0.3, resulting in 0.44. Next, determine the projection coefficient by dividing the total XY product by the sum of squares of vector Y. For example, if the total product is 0.44 and the sum of squares is 0.56, the projection coefficient is 0.44 divided by 0.56, resulting in approximately 0.786. This coefficient reflects the projection ratio of vector X onto vector Y. When the coefficient is between 0 and 1, the projection point falls on line segment AB. If the coefficient is less than 0, the projection point coincides with the starting point A; if it is greater than 1, the projection point coincides with the ending point B. Finally, calculate the coordinates of the projection point in each dimension. For each dimension, calculate the Y coordinates of that dimension. i Multiply by the projection factor to obtain the projected coordinate value of that dimension, such as Y. i If the projection coefficient is 0.4 and the projection coefficient is 0.786, then the projected coordinate value of this dimension is 0.4 multiplied by 0.786, which is approximately 0.314. By traversing all dimensions and arranging the projected coordinate values ​​of each dimension in order, the complete coordinates of the feature projection point P can be obtained.

[0063] To calculate the distance between the projection point and the endpoints of the line segment, and to determine the position of the projection point on line segment AB, it is necessary to calculate the distance from the projection point P to the starting point A and the ending point B. The total sum of squares indirectly reflects the distance. The calculation involves subtracting the coordinate value of A in that dimension from the coordinate value of P in that dimension (X) for each dimension. i ), obtain the dimensional differences; multiply each dimensional difference by itself to obtain the squared difference; then sum the squared differences of all dimensions one by one to obtain the total sum of squares of PA. The larger the total sum of squares, the farther P is from A; calculate the total sum of squares related to the distance from P to B, and follow the same logic, subtract the coordinate value of B in that dimension (Y) from the coordinate value of P in that dimension. i The dimensional difference is obtained, the square of the difference is calculated and summed to obtain the total sum of squares of PB. The larger the total sum of squares, the farther away P is from B.

[0064] The final correction point location and refinement, combined with the projection point position and the priority of the global reference, determine the final correction point Q. The specific operations are as follows: Determine the offset direction. If the total square sum of PB is less than the total square sum of PA, it means the projection point P is closer to the endpoint B. In this case, the correction direction should be further offset towards B to strengthen the constraints of the global model. If the total square sum of PA is less than or equal to the total square sum of PB, it means P is closer to the starting point A. In this case, the correction direction should be offset towards A to preserve the core information of the local features. Calculate the offset in each dimension. If the offset is towards B, for each dimension, use the coordinate value of B in that dimension (Y). i Subtract the coordinate value of P in that dimension from the coordinate value of A to obtain the basic offset for that dimension; if offsetting towards A, use the coordinate value of A in that dimension (X). i Subtract the coordinate value of dimension P from the actual offset to obtain the base offset; determine the actual offset value by multiplying the base offset by the offset ratio of 0.3 to obtain the actual offset value for each dimension. For example, if the base offset is 0.2 and the offset ratio is 0.3, then the actual offset value is 0.2 multiplied by 0.3, resulting in 0.06; calculate the coordinates of the correction point. If the offset is towards B, add the actual offset value to the coordinate value of dimension P to obtain the correction coordinates for that dimension; if the offset is towards A, add the actual offset value to the coordinate value of dimension P to obtain the correction coordinates for that dimension; traverse all dimensions and arrange the correction coordinates of each dimension in order to obtain the complete coordinates of the final correction point Q.

[0065] Step 442: Based on the final correction point, adjust the position of the initially corrected fusion feature representation to generate the adjusted feature representation, and verify the feature integrity of the adjusted feature representation to obtain the fusion feature representation after secondary correction. Specifically, this includes: dimensional feature position adjustment, using the coordinates of the final correction point Q as the standard, finely adjusting the initially corrected fusion feature representation (vector X); dimensional value replacement, for each feature dimension, directly replacing the original value of vector X in that dimension with the corrected coordinate value of Q in that dimension; reasonable range verification and correction, extracting from the global multimodal pattern library... For all feature vectors of a given dimension, find the minimum and maximum values ​​for that dimension to form a reasonable range of values. If the corrected coordinate value after replacement is lower than the minimum value of that dimension, adjust the value of that dimension to the minimum. If it is higher than the maximum value, adjust it to the maximum value. If it is within the range, retain the corrected coordinate value. Generate the adjusted feature vector. Arrange the values ​​of all dimensions after replacement and verification one by one in the original feature dimension order to form the adjusted feature vector, denoted as vector Z. This vector is the result of the collaborative optimization of local features and global reference, which corrects the deviation of the original features without deviating from the reasonable distribution range.

[0066] The detailed construction and execution of the feature integrity verification system includes a complete process encompassing standard setting, multi-dimensional correlation calculation, and iterative verification. This ensures the effectiveness of the adjusted feature vectors. The verification standard is clearly defined, using the enhanced global context vector Y as the core reference standard. Achieving a global correlation threshold is set as the verification pass condition. Global correlation measures the consistency between vectors Z and Y in semantic expression and feature distribution; higher correlation indicates better feature integrity. A correlation threshold is also set. The global correlation is calculated step-by-step, calculating core parameters dimension by dimension. For each dimension, three values ​​are calculated: the value of vector Z in that dimension is multiplied by the value of vector Y in that dimension to obtain the ZY dimension product; the value of vector Z in that dimension is multiplied by itself to obtain the square of the Z dimension; and the value of vector Y in that dimension is multiplied by itself to obtain the square of the Y dimension. The total parameters are calculated by summing the ZY dimension products of all dimensions; summing the squares of the Z dimension of all dimensions to obtain the total ZY product; summing the squares of the Y dimension of all dimensions to obtain the total Z square; and summing the squares of the Y dimension of all dimensions to obtain the total Y square. The correlation determination is based on the relationship between the total product of Z and Y, the total square of Z, and the total square of Y to determine whether the correlation meets the standard. The core logic is that the larger the total product of Z and Y, the stronger the positive correlation between Z and Y in each dimension; the closer the values ​​of the total square of Z and the total square of Y, the more consistent the feature distribution strength of Z and Y; when both conditions are met, the correlation meets the standard. The process involves iterative verification and adjustment. In the first verification, if the global correlation of vector Z reaches the set threshold, it indicates good integrity and consistency, and Z is directly determined as the fused feature representation after secondary correction. If the correlation does not meet the standard, it indicates insufficient or excessive correction. The process returns to step 441, adjusting the offset ratio from 0.3 to 0.5, recalculating the feature projection point P and the final correction point Q, and then returning to this step to perform feature adjustment and correlation calculation. This process is repeated multiple times. If adjusting the offset ratio to 0.5 still does not meet the standard, the process returns to step 441, adjusting the offset ratio to 0.7, and repeating the above process until the global correlation of vector Z meets the standard, ensuring that the feature representation after secondary correction has no integrity defects.

[0067] By refining the correction reference direction, priority, and standardized benchmark construction for each dimension, it is ensured that the feature correction direction fully conforms to the reasonable distribution of global multimodal patterns, avoiding the imbalance problem of a single dimension dominating correction; the rigorous verification of feature space mapping and line segment generation provide clear and consistent correction points.

[0068] A more realistic range of choices improves the semantic coherence and reliability of the completion results.

[0069] In a preferred embodiment of the present invention, step 5 above, which involves inputting the fused feature representation after secondary correction and the global context vector into the prediction model to generate inference results for completing missing modality data, may include:

[0070] In this embodiment of the invention, step 550 involves generating a comprehensive feature representation based on the fused feature representation after secondary correction and the enhanced global context vector through feature dimension concatenation. Specifically, this includes: first, determining the essential attributes of the two core input vectors; the fused feature representation after secondary correction is a set of core features after rotation correction and global context calibration, where each dimension corresponds to key association information of multimodal data, and the dimensions are unified to a preset 256 dimensions to ensure direct fusion of features from different modalities; the enhanced global context vector is a global information carrier obtained by retrieving similar patterns from a global multimodal pattern library, filtering by geometric relationships, and fusing with weights. The sample has 256 dimensions, with each dimension corresponding to the common features of the same type of multimodal scene. This dimension is consistent with the dimensions of the fusion feature representation after secondary correction, providing a compatibility basis for the splicing operation. Before splicing, a strict preprocessing operation is required. For the fusion feature representation after secondary correction, the values ​​of its 256 dimensions are traversed one by one to determine whether each value is within a reasonable range of 0-1. If a value of a certain dimension is greater than 1, it is adjusted to 1; if the value is less than 0, it is adjusted to 0; if the value is between 0 and 1, it remains unchanged. The same preprocessing process is performed on the enhanced global context vector to ensure that the numerical range of the two vectors is completely consistent and there is no difference in numerical scale between dimensions.

[0071] After preprocessing, feature dimension concatenation is performed according to fixed logic, determining the concatenation order: the fused feature representation after secondary correction comes first, followed by the enhanced global context vector. Specifically, the first dimension value of the fused feature representation after secondary correction is extracted and used as the first dimension value of the comprehensive feature representation; then, the second dimension value of the vector is extracted and used as the second dimension value of the comprehensive feature representation; and so on, until the 256th dimension value of the vector is extracted, at which point the comprehensive feature representation has formed the first 256 dimensions; subsequently, the first dimension value of the enhanced global context vector is extracted and used as the 257th dimension value of the comprehensive feature representation; the second dimension value of the vector is then extracted and used as the 258th dimension value of the comprehensive feature representation; and so on. The process continues until the 256th dimension value of the vector is extracted, resulting in a complete vector with 512 dimensions, which is the comprehensive feature representation. After concatenation, a double verification is required. The first verification is for dimensional completeness, counting the number of values ​​in the comprehensive feature representation one by one to confirm that it is exactly 512 (256 dimensions + 256 dimensions). If there are fewer than 512, the process goes back to the concatenation step to check if any dimension value is missing. If there are more than 512, the process checks for duplicate concatenation. The second verification is for the rationality of the values, iterating through the 512 dimensions again to confirm that all values ​​are in the 0-1 range. If there are values ​​outside this range, the process goes back to the preprocessing step to readjust the outlier values ​​in the corresponding vector until the comprehensive feature representation simultaneously meets the requirements of dimensional completeness and rationality of the values.

[0072] Step 551: Input the comprehensive feature representation into the input layer of the neural network prediction model. The input layer performs initial feature mapping on the comprehensive feature representation to obtain the initial feature mapping result. Specifically, this includes: first, completing the full construction of the input layer of the neural network prediction model. The core function of the input layer is to convert the comprehensive feature representation into a feature form suitable for subsequent calculations by the neural network prediction model. Therefore, the number of neurons strictly corresponds to the dimensions of the comprehensive feature representation, i.e., constructing 512 independent neurons. Each neuron is only responsible for processing the value of a specific dimension in the comprehensive feature representation, such as the first neuron corresponding to the first dimension of the comprehensive feature, the second neuron corresponding to the second dimension of the comprehensive feature, and so on. To ensure that information in each dimension is accurately captured without cross-interference, two core trainable parameters are configured for each neuron: initial weights and a bias term. The initial weights are randomly selected from the range of 0.01 to 0.05, and the initial weight values ​​of each neuron are independent and do not affect each other. The bias term is uniformly set to 0.1. The validated 512-dimensional comprehensive feature representation is input into the input layer in dimensional order. The value of the first dimension of the comprehensive feature representation is accurately passed to the first neuron of the input layer, the value of the second dimension is passed to the second neuron, and so on, until the value of the 512th dimension is passed to the 512th neuron. This ensures that each neuron receives only a single value of the corresponding dimension without any extraneous information interference.

[0073] Each neuron performs initial feature mapping calculations according to fixed logic. First, the received feature value is multiplied by the initial weight of the neuron to obtain a product. Second, the product is added to the bias term of the neuron to obtain the output value of the neuron. After all 512 neurons complete the calculation according to the logic of value × initial weight + bias term, the output values ​​of each neuron are arranged in order of their corresponding dimensions. The output of the first neuron is the first dimension of the initial feature mapping result, the output of the second neuron is the second dimension, and so on, until the output of the 512th neuron is the 512th dimension, forming a 512-dimensional vector, which is the initial feature mapping result.

[0074] Step 552: Based on the initial feature mapping results, perform multi-layer nonlinear transformations using the hidden layers of the neural network prediction model to extract and generate deep feature representations layer by layer. Specifically, this includes constructing three fully connected hidden layers. Fully connected means that each neuron in each layer is connected to all neurons in the previous layer, ensuring comprehensive information transmission. The number of neurons in each layer decreases progressively in the order of 512, 256, 128, and 64. The specific construction details are as follows: The first hidden layer has 256 neurons, and each neuron is connected to the 512 neurons in the input layer. Each connection corresponds to an independent trainable connection weight, with an initial value range of 0.01 to 0. The first hidden layer has 128 neurons, each fully connected to the 256 neurons in the first hidden layer, with independent connection weights, random values ​​between 0.01 and 0.05, and a bias term. The second hidden layer has 64 neurons, each fully connected to the 128 neurons in the second hidden layer, with the initial settings of connection weights and bias terms the same as the first two layers. Additionally, all hidden layers are configured with the ReLU activation function, which filters out invalid negative features, retaining only non-negative features and enhancing effective information. The nonlinear transformation process of the first hidden layer is as follows. Taking the 512-dimensional initial feature mapping result output from the input layer as input, each neuron in the first hidden layer receives the output values ​​of all dimensions in this 512-dimensional vector. Taking the 10th neuron in the first hidden layer as an example, its calculation process is as follows: First, the 512 received values ​​are multiplied by the 512 connection weights corresponding to the neuron. For example, the first dimension value of the initial feature mapping result is 0.1126 × connection weight 0.02 = 0.002252, the second dimension value is 0.098 × connection weight 0.04 = 0.00392, and so on, resulting in a total of 512 product results. Second, all 512 product results are added together. The process involves several steps: 1) obtaining a sum, such as 2.15 after adding all products; 2) adding the sum to the neuron's bias term 0.1 to obtain an intermediate result of 2.25; 3) inputting the intermediate result 2.25 into the ReLU activation function, which outputs 2.25 directly since it is non-negative. This is the output value of the neuron. After all 256 neurons in the first hidden layer have completed the above process of multiplication, summation, bias term addition, and ReLU activation, the output values ​​of all neurons are arranged in order to form the 256-dimensional output result of the first hidden layer. This result has initially filtered out redundant features and preserved the basic correlation information of the multimodal data.

[0075] The nonlinear transformation process of the second hidden layer takes the 256-dimensional result output from the first hidden layer as input. Each neuron in the second hidden layer receives the values ​​of all dimensions in this 256-dimensional vector. Taking the 5th neuron in the second hidden layer as an example, the 256 input values ​​are first multiplied by the corresponding 256 connection weights. For example, the first dimension value of the first layer output is 2.25 × connection weight 0.03 = 0.0675, the second dimension value is 1.98 × connection weight 0.05 = 0.099, and so on, resulting in 256 product results. These product results are then summed, and a bias term of 0.1 is added to obtain 1.96. Finally, after processing by the ReLU activation function, after all 128 neurons have completed the calculation, the 128-dimensional result of the second hidden layer is output. This result further compresses the dimensions and strengthens the core associated features; the third layer... The nonlinear transformation process of the hidden layer takes the 128-dimensional result output from the second hidden layer as input. Each neuron in the third hidden layer receives the values ​​of all dimensions in the 128-dimensional vector. Each neuron performs a logical calculation by multiplying the 128 values ​​with the corresponding connection weights, summing them, adding a bias term, and then performing ReLU activation. For example, if the sum of the 128 values ​​received by a neuron after multiplying them with the connection weights is 1.52, adding a bias term of 0.1 gives 1.62. After ReLU activation, the output is 1.62. After the 64 neurons complete the calculation, all the output values ​​are arranged in order to form a 64-dimensional deep feature representation. This feature representation has fully eliminated noise and redundant information, retaining only the core intrinsic correlation rules of multimodal data, such as the precise matching relationship between text semantics and image vision, and the synchronous correlation between audio temporal sequence and video frame.

[0076] Step 553: Based on deep feature representation, the final mapping calculation is performed through the output layer of the neural network prediction model to generate the inference result of the missing modality data. Specifically, this includes: first, completing the customized construction of the output layer. The core objective of the output layer is to convert the deep feature representation into a result consistent with the format of the missing modality data. Therefore, the number of its neurons must strictly match the feature dimension of the missing modality. For example, if the missing modality is text, and the feature dimension of the text modality after encoding is 150, that is, each text sample needs to be encoded with 150 values ​​to represent the complete semantics, then the output layer is set with 150 neurons; if the missing modality is image, and the pixel matrix of the image is 512×512, with a total of 262,144 pixels, and each pixel needs a value to represent color information, then the output layer is set with 262,144 neurons; if the missing modality is audio, and the audio spectrum sequence is 300 frames × 512 frequency points, with a total of 153,600 spectral energy values, then the output layer... The layer has 153,600 neurons. Each neuron in the output layer is fully connected to 64 neurons in the third hidden layer. Each connection is configured with an independent trainable connection weight, with an initial value ranging from 0.01 to 0.05. Each neuron contains a bias term. The appropriate activation function is selected based on the type of missing modality. If the missing modality is text, the Sigmoid activation function is selected, which maps the output value to the 0-1 range, perfectly matching the numerical range of text word encoding, facilitating subsequent decoding into specific words. If the missing modality is image, the Tanh activation function is selected, which maps the output value to the -1-1 range, consistent with the normalized range of RGB pixel values ​​in an image, accurately restoring pixel color information. If the missing modality is audio, a linear activation function is selected, which does not change the numerical range and completely preserves the continuous energy values ​​of the audio spectrum, ensuring the integrity of the audio information.

[0077] The final mapping calculation is performed. Each neuron in the output layer receives the values ​​of all dimensions in the 64-dimensional deep feature representation output by the third hidden layer. The calculation follows a fixed procedure: First, each of these 64 values ​​is multiplied by the corresponding 64 connection weights of the neuron. For example, when the text modality is missing, the 25th neuron in the output layer receives the following deep feature values: 1.62 × 0.04 = 0.0648 for the first dimension, 1.35 × 0.06 = 0.081 for the second dimension, and so on, resulting in 64 product results. Second, all 64 product results are summed. Third, this sum is added to the neuron's bias term of 0.1 to obtain an intermediate result of 3.55. Fourth, the intermediate result is input into a preset... The activation function is used for processing. After all neurons have completed their calculations, the output values ​​are arranged according to the original data format of the missing modality. If it is a text modality, the output values ​​of 150 neurons are arranged in the order of word encoding to form a 150-dimensional word encoding sequence. If it is an image modality, the output values ​​of 262,144 neurons are arranged in row-major order of the pixel matrix, first all pixels in the first row, then the second row, and so on until the last row, forming a 512×512 pixel value matrix. If it is an audio modality, the output values ​​of 153,600 neurons are arranged in the order of spectrum frame × frequency point, first all frequency points in the first frame, then the second frame, and so on until the 300th frame, forming a 300×512 spectrum value sequence. The result after this arrangement is the inference result of the missing modality data.

[0078] By using dimensional concatenation to deeply integrate the local core features after secondary correction with the enhanced global contextual features, we can retain the intramodal and intermodal collaborative correlation information of the current multimodal data, and also incorporate the common regularity information of the global multimodal pattern library. The two types of information are complementary and non-overlapping, avoiding the one-sidedness of inference caused by single features, and providing comprehensive information basis for inference of missing modalities.

[0079] In a preferred embodiment of the present invention, step 6 above, which generates final multimodal completion data based on the inference results and uses the completion data to periodically update the parameters of the global multimodal pattern library and the prediction model to achieve adaptive completion of missing data, may include:

[0080] In this embodiment of the invention, step 660 integrates the inference result with the valid modal data in the original multimodal input data to generate complete multimodal completion data. Specifically, this includes: firstly, decoding the inference result of the missing modality, converting the numerical representation into the original modal data format; if the inference result is a word encoding sequence of the text modality, such as a numerical sequence in the 0-1 interval, then matching each encoded value with a preset vocabulary, where the encoded value corresponds to a unique word in the vocabulary, and concatenating them into complete text content, such as sentences or paragraphs, according to the encoding order; if the inference result is a pixel value matrix of the image modality, such as a numerical matrix in the -1-1 interval, then converting each pixel value into an RGB color value, and through the numerical mapping relationship, mapping the -1-1 interval values ​​to the RGB range of 0-255, and restoring them into image areas according to the matrix row and column order. If the inferred result is a spectral value sequence of the audio modality, the spectral values ​​are converted into audio electrical signals. Through inverse spectral transformation logic, the frequency domain values ​​are restored to the time domain audio signals to form a complete audio segment. Then, the missing positions in the original multimodal input data are located. Based on the mask matrix generated in step 1, where 0 indicates the missing position and 1 indicates the valid position, the filling area of ​​each missing modality is precisely matched. For example, if the third field of the text modality in the original data is missing, and the corresponding position in the mask matrix is ​​0, then the decoded text content is filled into this field. If the upper right corner 100×100 pixel area of ​​the image modality is missing, and the corresponding area in the mask matrix is ​​0, then the restored image pixel data is filled into this area. If the 0.5-1.0 second time period of the audio modality is missing, then the generated audio segment is inserted into this time period.

[0081] Next, the valid data and the completed data are integrated. All valid modal data marked as 1 in the mask matrix of the original multimodal input data are retained, such as images without missing parts, complete audio segments, and text fields without missing parts. The decoded inference results are then accurately filled into the missing positions, ensuring that the format of different modal data is consistent and logically coherent. For example, after text modality completion, the semantic connection between the completed content and the original text needs to be checked, such as whether the grammar is fluent and the referents are clear; after image modality completion, the visual consistency between the completed area and the original image needs to be verified, such as whether the color, brightness, and outline match; after audio modality completion... First, it is necessary to confirm whether the sound quality and volume of the completed segment are consistent with the original audio. Finally, completeness and rationality are checked, and the data is checked modally to ensure that all 0 positions in the mask matrix are filled with valid data. At the same time, the coordination of cross-modal data is checked, such as whether the objects described in the text are consistent with the objects in the image, and whether the audio timing is synchronized with the video frame sequence. If there are semantic conflicts or visual / auditory mismatches, the process is backtracked to the decoding or filling steps of the inference results, and the completed content is adjusted until all modal data are complete, logically coherent, and cross-modal coordinated, and finally, complete multimodal completion data is generated.

[0082] Step 661: Based on the complete multimodal completion data, initiate the pattern library update process according to a preset time period, and obtain new pattern features through feature extraction. Specifically, this includes: first, preset the pattern library update period; the period type can be a time period, such as starting at 2 AM daily, or a data volume period, such as starting after accumulating 1000 complete multimodal completion data entries. The period length can be adjusted according to the actual application scenario to ensure the pattern library can absorb new data patterns in a timely manner without frequently consuming computing resources. When the preset update period is reached, the pattern library update process is automatically initiated, performing batch preprocessing on all accumulated complete multimodal completion data. For text modalities, invalid characters are removed, words are segmented, and stop words are removed; for image modalities, the resolution is adjusted to a uniform size, and keyframes are extracted; for audio modalities, the sampling rate is unified, and the data is converted to a spectrogram; for video modalities, keyframes are extracted and the size is standardized. The preprocessing standards are completely consistent with the multimodal data preprocessing in Step 1, ensuring data format uniformity. The feature extraction process is then executed on the preprocessed completion data, completely reusing the feature processing logic of Step 1. First, each modal data is encoded... The code processing involves converting text to vocabulary encoding, images to pixel values, and audio to spectral values ​​to generate numerical feature representations. Next, the internal relationships between features are analyzed, and the features are structurally reorganized into core feature groups, secondary feature groups, and auxiliary feature groups. Then, local feature correlations are calculated to generate feature calibration parameters and adjust the structured features. Finally, the adjusted structured features are converted into fixed-dimensional core feature vectors, which become the new pattern features. To facilitate subsequent pattern retrieval and classification, scene labels are added to each new pattern feature. The label content includes the missing modality type, effective modality combination, and application scenario. For example, for a completion dataset containing missing text but complete images and audio in an autonomous driving scenario, the pattern feature label would be: missing text - complete image + audio - autonomous driving. Finally, all newly extracted pattern features are validated to check if the feature vector dimension is the preset fixed dimension, if the values ​​are within a reasonable range of 0-1, and if the scene labels are accurate and complete. If there are any abnormal dimensions, values ​​exceeding the range, or incorrect labels, the preprocessing or feature extraction steps are re-processed to ensure the standardization of the new pattern features.

[0083] Step 662: Based on the new pattern features, expand and improve the global multimodal pattern library by adding new pattern features and adjusting the distribution of existing pattern features. Specifically, this includes: first, performing a deduplication operation on the new pattern features; traversing all existing pattern features in the global multimodal pattern library; for each new pattern feature, calculating its similarity to existing features in the library; the similarity calculation logic is to compare the numerical differences between the new pattern feature and existing features dimension by dimension, summing the absolute values ​​of the numerical differences across all dimensions to obtain the total difference value; the smaller the total difference value, the higher the similarity; a preset similarity threshold is set, such as a total difference value less than 0.1 being considered highly similar. If the total difference value of a new pattern feature is less than a threshold, it is determined to be a repeating pattern, and the new pattern feature is discarded and not included in the database. If the total difference value is greater than or equal to the threshold, it is determined to be a non-repeating pattern and proceeds to the subsequent expansion process. Next, existing pattern features are improved and adjusted, and existing features in the database that match the scene label of the new pattern feature are selected. If the total difference value of such existing features is greater than the threshold but less than twice the threshold, it indicates that the feature distribution in that scene needs optimization. The adjustment logic is to calculate the average value of each dimension of the existing feature and the new pattern feature, add the value of the corresponding dimension of the new pattern feature to the existing feature dimension value, divide by 2, and use this average value. The average value is used to update the corresponding dimension values ​​of existing features, making the distribution of existing features converge towards the new pattern features and increasing the concentration of features within the same scene. If the total difference between existing features and new pattern features is greater than or equal to twice the threshold, the existing features are retained without adjustment, and only the new pattern features are added as a new branch. Then, the new pattern features are expanded and added. The deduplicated new pattern features are classified according to scene labels, and corresponding label classification directories are created or assigned to them in the global multimodal pattern library. When storing, the complete information of the new pattern features must be recorded, including feature vectors, scene labels, and generation time, to facilitate subsequent filtering by label or time during retrieval. At the same time, the data is updated. The feature index of the pattern library is sorted by the core dimension values ​​of the feature vectors to improve retrieval efficiency. Finally, the storage structure of the pattern library is optimized. When the number of features under a certain scene label in the pattern library exceeds a preset limit, the average feature vector of all features under that label is calculated. The values ​​of each dimension are summed and divided by the number of features. The average feature vector and the top 500 features with the highest similarity are retained, and the remaining features with low similarity are deleted to avoid excessive expansion of the pattern library. At the same time, the storage format of feature distribution is adjusted to cluster feature vectors under the same scene according to the core dimension values, so that similar feature clusters can be quickly located during retrieval, improving retrieval speed.

[0084] Step 663: Based on the complete multimodal completion data and the updated global multimodal pattern library, initiate the retraining process of the prediction model and adjust the prediction model parameters. Specifically, this includes: first, preparing the retraining dataset by dividing the complete multimodal completion data generated in step 660 and all historical feature data in the updated global multimodal pattern library into training and validation sets in a 7:3 ratio. During the division, ensure that the dataset proportions are consistent for each scene label to avoid excessive concentration of data in a particular scene category, which could affect the model's generalization ability. Then, initialize the prediction model by loading the currently used prediction model. In step 5, all existing parameters of the neural network prediction model used to generate the inference results, including the connection weights and neuron biases of the input, hidden, and output layers, are adjusted. The network structure of the prediction model remains unchanged, with 512 neurons in the input layer, 3 hidden layers, and the number of neurons in the output layer adapted to the modal dimension. Only the parameter values ​​are adjusted through retraining to avoid adaptation problems caused by the reconstruction of the prediction model structure. The prediction model retraining process is then initiated, converting the multimodal data of the training set into a comprehensive feature representation according to the feature processing logic of steps 1 to 5, and inputting it into the prediction model. The prediction model generates the inference results according to the forward propagation logic. The prediction results are generated, and then the error between the prediction results and the actual imputed data in the training set is calculated. The error calculation logic is to compare the predicted values ​​with the actual data values ​​dimensionally, calculate the absolute value of the difference in each dimension, and sum the absolute values ​​of the differences in all dimensions to obtain the total error for a single data point. The total error of all training data is then summed and divided by the number of training data points to obtain the average error. The prediction model parameters are adjusted based on the average error. The parameter adjustment uses backpropagation logic, starting from the output layer and adjusting the connection weights and bias terms layer by layer. The adjustment rule is: the new value of the connection weight = the original value of the connection weight - the average error. × Weight adjustment coefficient (the weight adjustment coefficient is a preset fixed value used to control the adjustment range and avoid excessive parameter fluctuations); The new value of the neuron bias term = original bias term value - average error × bias adjustment coefficient. The bias adjustment coefficient is consistent with the weight adjustment coefficient. For example, if the original connection weight of a hidden layer neuron is 0.03, the average error is 0.05, and the weight adjustment coefficient is 0.1, then the new connection weight = 0.03 - 0.05 × 0.1 = 0.025; the original bias term is 0.1, and the bias adjustment coefficient is 0.1, then the new bias term = 0.1 - 0.05 × 0.1 = 0.095; After each training round, the performance of the prediction model is evaluated using validation set data. The validation set data is input into the adjusted prediction model to generate validation prediction results. The average error between the validation prediction results and the real data in the validation set is calculated, and the completion accuracy is also calculated. If the average error of the validation set continues to decrease and the completion accuracy continues to improve, the next round of training continues. If, after three consecutive rounds of training, the average error of the validation set no longer decreases or the completion accuracy no longer improves, it indicates that the prediction model has reached a stable state, and the retraining process is stopped. Finally, the updated prediction model parameters are saved, overwriting the original parameters with the retrained connection weights, bias terms, etc., and stored in the model parameter file. Information such as parameter update time, training dataset size, and validation set accuracy is recorded. Simultaneously, the updated prediction model is set as the currently active model for missing modality completion in new data.

[0085] Step 664: Based on the retrained prediction model and the updated global multimodal pattern library, an adaptive completion system is formed to achieve continuous adaptive completion of input missing data. Specifically, this includes: first, clarifying the core components of the adaptive completion system, using the retrained prediction model as the core processing unit and the updated global multimodal pattern library as contextual retrieval support. The two are linked in real time through a data transmission channel, forming a closed-loop architecture of retrieval-processing-inference-updating. The system also includes a data input interface, a preprocessing mode, and a result output interface to ensure a smooth end-to-end completion process. When new multimodal missing data is input into the system, adaptive completion is achieved according to the following process: First, new data is received through the data input interface, and the preprocessing module performs missing data completion. The system performs several steps: 1) Value detection, encoding, and feature recombination to generate numerical feature vectors and a mask matrix; 2) Based on the generated feature vectors, it retrieves the updated global multimodal pattern library, filters by scene label and sorts by feature similarity to obtain relevant contextual patterns, and generates an enhanced global context vector; 3) Inputs the feature vectors, mask matrix, and global context vectors into the retrained prediction model, performs feature fusion, secondary correction, and multi-layer nonlinear transformation to generate inference results for missing modes; 4) Integrates the inference results with valid modal data from new data and outputs complete supplementary data through the result output interface; After the supplementary data is generated, the system's iterative update mechanism is automatically triggered, storing the newly generated complete supplementary data in a temporary data pool. When the amount of data reaches the preset update cycle threshold in step 661 or reaches the preset time node, the system automatically starts the process of steps 661-663, extracts the supplementary data from the temporary data pool as new mode features, and updates the global multimodal mode library; then, it uses the new supplementary data and the updated mode library to train the prediction model and adjust the parameters; the updated mode library and prediction model are re-integrated into the system, replacing the original components, completing one iteration. For newly emerging missing scenarios, such as a video + text double missing - audio complete scenario that has never been processed before, the system adapts through the following logic: after the new data is preprocessed, if the mode library is searched and no corresponding scene label is found, the feature vector of the data is marked as a new scene feature and directly added to the mode library according to the rules of step 662, creating... A new scene label directory is created; during retraining, the completion data of this new scene serves as the core training sample, and the prediction model parameters are adjusted accordingly. This allows the model to quickly retrieve the corresponding pattern and generate accurate completion results when encountering similar scene data. The accuracy of the completion results is monitored in real time. If the completion accuracy of a certain scene is lower than a preset threshold, the update cycle of the pattern library for that scene is automatically shortened, the retraining frequency is increased, and the prediction model's adaptability to that scene is strengthened. At the same time, the feature distribution of that scene in the pattern library is optimized, and invalid features with low similarity are deleted to improve retrieval accuracy. Through continuous iteration and dynamic optimization, the system can continuously adapt to new multimodal data types and new missing scenes, achieving long-term, stable, and accurate continuous adaptive completion of input missing data.

[0086] By decoding the inference results, matching missing positions, and integrating effective data, the generated multimodal completion data has a unified format and logical coherence. This ensures both the integrity of each modality's data and the collaborative consistency of cross-modal data. New pattern features are periodically incorporated, and duplicates are removed and the distribution of existing features is optimized. This allows the pattern library to continuously cover new scenarios and update old patterns, avoiding retrieval bias caused by static libraries and providing contextual support for completion inference that is more in line with current data patterns.

[0087] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0088] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0089] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for multi-modal large model modality missing data completion, characterized in that, The method comprises: Step 1, missing value detection and encoding processing is performed on the multi-modal input data, the correlation between the internal features of the data is analyzed, the structured feature representation is generated through feature reorganization, the feature calibration parameters are generated combined with the correlation between local features, the numerical feature vectors and the mask matrix of each modality are obtained; Step 2, the numerical feature vector and the mask matrix are input into the feature fusion model, and the context dependence relationship within and between the modes is analyzed, the principal axis parameters of the feature distribution ellipse are calculated to correct the rotation of the ellipse feature, and the preliminary corrected fusion feature representation is generated; including: input the numerical feature vector and the mask matrix into the feature fusion model, process the input data through self-attention calculation and cross-modal interaction calculation inside the feature fusion model, obtain the complex context dependence relationship within and between the modes, and obtain the context enhanced feature representation; calculate the distribution characteristics of the context enhanced feature representation in the feature space, and use the distribution characteristics to construct the corresponding covariance matrix; perform eigenvalue decomposition on the covariance matrix, determine the principal axis parameters of the feature distribution ellipse according to the decomposition result, including the direction vector and length value of the long axis in the original feature space, and the direction vector and length value of the short axis in the original feature space; based on the long axis direction vector in the principal axis parameters, calculate the rotation transformation matrix required to rotate the long axis of the feature distribution ellipse to the horizontal direction, and use the rotation transformation matrix to perform overall rotation transformation on the context enhanced feature representation, so that the long axis of the feature distribution ellipse is aligned with the horizontal direction and the short axis is aligned with the vertical direction, and the rotation corrected ellipse feature is obtained; the rotation corrected ellipse feature is input into the full connection layer, the rotation corrected feature is dimensionally adjusted and compressed through the nonlinear transformation of the full connection layer, and the preliminary corrected fusion feature representation is generated; Step 3, based on the preliminary corrected fusion feature representation, retrieve relevant context patterns from the global multi-modal pattern library, determine the positional relationship between the two circles by comparing the distance between the centers of the two circles, the sum of the radii, and the absolute value of the radius difference, and obtain an enhanced global context vector; including: taking the preliminary corrected fusion feature representation as the query feature, and retrieving multiple similar context patterns from the global multi-modal pattern library based on the query feature to form a candidate context pattern set; calculating the geometric relationship parameters between the feature representation circle corresponding to each context pattern in the candidate context pattern set and the query feature representation circle, including the distance between the centers of the two circles, the sum of the radii of the two circles, and the absolute value of the radius difference; based on the geometric relationship parameters, by comparing the relationship between the distance between the centers of the two circles and the sum of the radii, and the relationship between the distance between the centers of the two circles and the radius difference, determine the positional relationship category between each context pattern and the query feature, including apart, tangent or intersection; based on the positional relationship category, filter out the context patterns with positional relationship as tangent or intersection from the candidate context pattern set to form a relevant context pattern subset; based on the relevant context pattern subset, extract the feature vector representation of each context pattern in the relevant context pattern subset, and combine the positional relationship closeness between the context pattern and the query feature, and generate an enhanced global context vector through feature fusion processing; Step 4, using the enhanced global context vector to perform secondary correction on the preliminary corrected fusion feature representation to obtain a secondary corrected fusion feature representation; including: based on the enhanced global context vector, determining the reference direction of feature correction, establishing a correction direction reference, and performing feature space mapping on the preliminary corrected fusion feature representation according to the correction direction reference to generate a feature representation line segment; calculating the intersection position of the feature representation line segment and the correction direction reference in the feature space, determining the feature projection point, and positioning the final correction point on the feature representation line segment based on the feature projection point; based on the final correction point, adjusting the position of the preliminary corrected fusion feature representation to generate an adjusted feature representation, and verifying the feature integrity of the adjusted feature representation to obtain a secondary corrected fusion feature representation; Step 5, inputting the secondary corrected fusion feature representation and the global context vector into the prediction model to generate an inference result for completing the missing modal data; Step 6, generating the final multi-modal completion data according to the inference result, and periodically updating the parameters of the global multi-modal pattern library and the prediction model using the completion data to realize adaptive completion of the missing data.

2. The multi-modal large model modality missing data completion method according to claim 1, characterized in that, Perform missing value detection and encoding processing on multi-modal input data, analyze the association between internal features of the data, generate structured feature representations through feature reorganization, and generate feature calibration parameters based on the correlation between local features to obtain numerical feature vectors and mask matrices for each modality, including: Perform missing value detection on multi-modal input data to identify missing data positions in each modality and generate an initial mask matrix; Based on the initial mask matrix, the multi-modal input data is encoded and processed to convert the original data into numerical feature representation; The correlation between the internal features of the numerical feature representation is analyzed, and the numerical features are structurally reorganized through feature reorganization to generate a unified structured feature representation; Based on the correlation between the local features in the structured feature representation, the feature calibration parameters for adjusting the feature representation are calculated; Based on the feature calibration parameters, the structured feature representation is adjusted, and the numerical feature vectors and mask matrices of each modality are obtained.

3. The multi-modal large model modality missing data completion method according to claim 2, characterized in that, The elements in the mask matrix are used to identify the missing state of the corresponding data position.

4. The multi-modal large model modality missing data completion method according to claim 3, characterized in that, The secondary corrected fusion feature representation and the global context vector are input into the prediction model to generate an inference result for completing the missing modality data, including: Based on the secondary corrected fusion feature representation and the enhanced global context vector, an integrated feature representation is generated through feature dimension splicing operation; The integrated feature representation is input into the input layer of the neural network prediction model, and the integrated feature representation is initially feature mapped through the input layer to obtain an initial feature mapping result; Based on the initial feature mapping result, a multi-layer nonlinear transformation is performed on the hidden layer of the neural network prediction model to extract and generate deep-level feature representation layer by layer; Based on the deep-level feature representation, the output layer of the neural network prediction model is used for final mapping calculation to generate an inference result of the missing modality data.

5. The multi-modal large model modality missing data completion method according to claim 4, characterized in that, According to the inference result, the final multi-modal completion data is generated, and the parameters of the global multi-modal mode library and the prediction model are periodically updated using the completion data to realize adaptive completion of the missing data, including: Integrate the inference result with the valid modality data in the original multi-modal input data to generate complete multi-modal completion data; Based on the complete multi-modal completion data, start the mode library update process according to the preset time period, and obtain new mode features through feature extraction; Based on the new mode features, the global multi-modal mode library is expanded and improved, new mode features are added, and the distribution of existing mode features is adjusted; Based on the complete multi-modal completion data and the updated global multi-modal mode library, start the prediction model retraining process to adjust the prediction model parameters; Based on the retrained prediction model and the updated global multi-modal mode library, form an adaptive completion system to realize continuous adaptive completion of the input missing data.

6. A computing device, comprising: including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, so that the one or more processors implement the method of any one of claims 1 to 5.

7. A computer readable storage medium characterized by The computer readable storage medium stores a program which is executed by the processor to implement the method of any one of claims 1 to 5.

Images