A Multimodal Fusion Method and System for Multivariate Classification and Grading of Unstructured Data

Abstract

This invention relates to the field of data security classification and grading technology, and discloses a multimodal fusion method and system for multivariate classification and grading of unstructured data. The classification method includes: multimodal data fusion, including document data multimodal fusion, image data multimodal fusion, audio data multimodal fusion, and video data multimodal fusion, to obtain a fused representation vector; inputting the fused representation vector into a neural network module to output an M-dimensional vector, where M is the number of categories; transforming the M-dimensional vector to obtain a probability vector P; obtaining the multivariate data categories of the multimodal data based on the probability vector P; and calculating the maximum data level and expected data level of the unstructured data. This invention introduces multimodal fusion technology, fully explores and utilizes various effective information in unstructured data, and employs multivariate classification and grading technology to assign multiple categories, a maximum data level, and an expected data level to unstructured data, significantly improving the classification and grading effect of unstructured data.

Description

Technical Field

[0001] This invention relates to the field of data security classification and grading technology, and in particular to a multimodal fusion method and system for multi-dimensional classification and grading of unstructured data. Background Technology

[0002] Currently, data classification and grading efforts in various organizations primarily focus on structured data, with less attention paid to unstructured data. However, unstructured data (such as text, images, audio, and video) accounts for over 80% of the total data volume in various organizations. Classifying and grading unstructured data is a crucial means to improve data management efficiency, reduce security risks, ensure compliance, and unlock data value. Due to the irregular and non-standard characteristics of unstructured data, its classification and grading are more challenging than those of structured data. Currently, mainstream unstructured data classification and grading mainly relies on traditional methods, such as rule matching and simple text analysis. These methods suffer from low information extraction efficiency and accuracy when dealing with complex and variable unstructured data. In recent years, with the development of deep learning technology, deep learning methods have also been applied to the classification and grading of unstructured data, but this method still faces challenges such as high data annotation costs and poor model interpretability.

[0003] Many unstructured data contain a wealth of information that is difficult for humans to describe using rules. For example, a piece of text may contain a variety of information such as emotion, theme, and opinion, and the relationships between these information are intricate. Rule-based systems struggle to capture and handle this ambiguity. Similarly, images contain various irregular entities, making it difficult to describe their inherent patterns or characteristics using a single rule system. Therefore, classifying and grading unstructured data using rule-based techniques is extremely challenging. Even when some unstructured data, such as documents, can be classified and graded using rules, the length and sparse information distribution of documents often make it difficult for traditional rule-matching methods to capture the complex semantic relationships and contextual information, resulting in low information extraction efficiency and accuracy. Furthermore, rule-matching methods lack flexibility and scalability, making it difficult to adapt to constantly changing data environments and classification needs. Therefore, classifying and grading document data requires exploring more efficient, accurate, and flexible techniques to address its unique challenges.

[0004] While deep learning-based methods have shown great potential in processing unstructured data, they also have significant limitations. The primary issue is that effective training of deep learning models heavily relies on large-scale, high-quality labeled datasets. However, in many applications involving the classification and grading of sensitive or confidential information, obtaining a sufficient number of labeled samples is not only difficult but also costly. This is mainly because data ownership and privacy protection are major obstacles to data sharing. Furthermore, most existing deep learning methods focus on directly analyzing the content of unstructured data, such as text, images, and audio. However, this approach ignores a crucial characteristic of unstructured data—the sparsity of information distribution. Coupled with the limited amount of sample data available for training, even state-of-the-art deep learning models cannot achieve ideal performance levels in all situations. It is worth noting that in real-world business environments, unstructured data not only contains valuable information within its intrinsic content, but also its surrounding metadata, such as filenames, file descriptions, file sources, file sizes, and the identity of the file creators, often contains rich contextual information. This information is equally crucial for understanding and utilizing the data. However, most current deep learning-based unstructured data analysis solutions do not fully consider or utilize this additional metadata information, thus limiting their effectiveness and efficiency in practical applications.

[0005] Furthermore, most current methods for classifying and grading unstructured data tend to simply categorize data into a single category or level. While this approach may be effective when dealing with relatively simple information, it struggles to meet the diverse needs of practical applications for complex and varied unstructured data. The complexity and diversity of unstructured data mean that it may belong to multiple categories simultaneously. Therefore, when processing such data, a more flexible and nuanced strategy is urgently needed to accurately capture its multifaceted attributes. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal fusion method and system for classifying and grading unstructured data. It innovatively introduces multimodal fusion technology to fully mine and utilize various effective information within unstructured data. Furthermore, by employing multimodal classification and grading technology, multiple categories, maximum data levels, and desired data levels are assigned to unstructured data, significantly improving the classification and grading effectiveness of unstructured data.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] In a first aspect, the present invention provides a multimodal fusion method for classifying and grading unstructured data, comprising the following steps:

[0009] S1. Multimodal data fusion to obtain a fused representation vector;

[0010] S2. The fused representation vector is input to the neural network module to output an M-dimensional vector, where M is the number of categories;

[0011] S3. Transform the M-dimensional vector to obtain the probability vector P;

[0012] S4. Obtain the multivariate data categories of multimodal data based on the probability vector P, specifically including:

[0013] S40. Set parameters k and t, where k is the maximum value of the selectable categories and t is the probability threshold;

[0014] S41. Select categories from the probability vector P whose probability values ​​are greater than or equal to the probability threshold t;

[0015] S42. Sort the selected categories in descending order of probability value. When the number of selected categories is greater than or equal to k, take the first k categories as the multivariate data categories of the unstructured data; when the number of selected categories is less than k, take all categories as the multivariate data categories of the unstructured data.

[0016] S5. Calculate the maximum data level and the expected data level for unstructured data; the maximum data level is calculated using the following method:

[0017] S50. Calculate the data level vector corresponding to each category in the multivariate data categories;

[0018] S51. Select the largest data level value from the data level vector as the largest data level for unstructured data;

[0019] The expected data level is calculated using the following method:

[0020] S52. Based on the index of each category in the probability vector in the multivariate data categories, obtain its corresponding logits from the M-dimensional vector, and further represent it as CLogits;

[0021] S53. Convert CLogits into a probability vector CP, and calculate the expected data level based on the probability vector CP.

[0022] As one possible implementation, S1 includes:

[0023] S10. Apply a semantic understanding model to represent the metadata of unstructured data to obtain a metadata representation vector;

[0024] S11. Apply a semantic understanding model to represent the unstructured data itself to obtain an unstructured data representation vector;

[0025] S12. Stack the metadata representation vector and the unstructured data representation vector in the channel dimension;

[0026] S13. Apply the convolution kernel of the convolutional neural network to fuse the data in the above multiple channels to obtain a fused representation vector.

[0027] As one possible implementation, metadata should include at least the filename, directory name, creator, descriptive information, data source, and creation date; represented using the RoBERTa semantic understanding model.

[0028] As one possible implementation, unstructured data includes document data, which has three modalities: metadata, text data, and images. Metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors; text data is represented using the Longformer semantic understanding model to obtain text data representation vectors; images are first represented using the ViT semantic understanding model, and then the Transformer encoder fuses the representation vectors of all images to obtain image representation vectors. The metadata representation vectors, text data representation vectors, and image representation vectors are stacked along the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vectors, text data representation vectors, and image representation vectors from the three channels to obtain a fused representation of the document data.

[0029] As one possible implementation, unstructured data includes image data, which has two modalities: metadata and images. The metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors. Each image is first represented using the ViT semantic understanding model, and then the Transformer encoder is used to fuse the representation vectors of all images to obtain image representation vectors. The metadata representation vectors and image representation vectors are stacked in the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vectors and image representation vectors in the two channels to obtain a fused representation of the image data.

[0030] As one possible implementation, unstructured data includes audio data, which has two modalities: metadata and audio signal. The metadata is represented using the RoBERTa semantic understanding model to obtain a metadata representation vector. The audio signal is preprocessed to obtain Mel-frequency inverse coefficients, which are then represented using the Conformer semantic understanding model to obtain an audio signal representation vector. The metadata representation vector and the audio signal representation vector are stacked along the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vector and the audio signal representation vector from both channels to obtain a fused representation of the audio data.

[0031] As one possible implementation, unstructured data includes video data, which has four modalities: metadata, subtitle sequences, audio signals, and frame sequences. The metadata is represented using the RoBERTa semantic understanding model to obtain a metadata representation vector; the subtitle sequences are represented using the RoBERTa semantic understanding model to obtain a subtitle sequence representation vector; the audio signals are represented using the Conformer semantic understanding model to obtain an audio signal representation vector; and the frame sequences are represented using the VTN-ENCODER semantic understanding model to obtain a frame sequence representation vector. The metadata representation vector, subtitle sequence representation vector, audio signal representation vector, and frame sequence representation vector are stacked along the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vector, subtitle sequence representation vector, audio signal representation vector, and frame sequence representation vector from the four channels to obtain a fused representation of the video data.

[0032] As one possible implementation, the expected data level calculation formula is:

[0033]

[0034] Where edl represents the expected data level; 1≤l≤m, l is the l-th category in the multivariate data categories of unstructured data, and m is the number of categories in the multivariate data categories of unstructured data; cp l This represents the data category DC. l The probability, idx l Let be the index of the l-th category in the probability vector P of the multivariate data categories of unstructured data. It is the logit corresponding to the l-th category in the multivariate data categories of unstructured data, where j is a variable representing 1, 2, 3, ..., m. It is the logit corresponding to the j-th category in a multivariate dataset of unstructured data. The relevant data attribute of the unstructured data is attr. For a data category dc l Its corresponding data level is dl l dl l =f(dc) l ,attr).

[0035] As one possible implementation, the relevant data attribute of unstructured data is defined as attr, and for one of its data categories, dc... l Its corresponding data level dl l It can be represented as:

[0036] dl l =f(dc)l ,attr),

[0037] According to dl l Calculate the data level vector (DL) corresponding to each category and represent it as follows:

[0038]

[0039] Selecting the largest data level value from the DL as the maximum data level (mdl) for unstructured data is represented as follows:

[0040]

[0041] Secondly, the present invention provides a multimodal fusion system for classifying and grading unstructured data, comprising:

[0042] The multimodal data fusion module receives unstructured data, represents the metadata of the unstructured data and stacks it along the channel dimension, and applies the convolution kernel of the convolutional neural network to fuse the data in multiple channels to obtain a fused representation vector.

[0043] The neural network module receives the fused representation vector and outputs an M-dimensional vector, where M is the number of categories;

[0044] The transformation module transforms the M-dimensional vector to obtain the probability vector P;

[0045] The multivariate data category determination module is used to perform the first aspect S40 to S42 to obtain the multivariate data categories;

[0046] The data level calculation module is used to execute S50 to S51 of the first aspect to obtain the maximum data level; it is also used to execute S52 to S53 of the first aspect to obtain the desired data level.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. The multimodal fusion method for classifying and grading unstructured data provided by this invention combines the content of unstructured data itself with its metadata using multimodal fusion technology, making full use of available information, reducing the requirement for the number of samples, and improving the accuracy of classification and grading of unstructured data.

[0049] 2. The multimodal fusion method for classifying and grading unstructured data provided by this invention can simultaneously assign multiple categories to unstructured data and obtain the maximum data level and the desired data level, enabling more refined and flexible management and protection of the data.

[0050] 3. The multimodal fusion-based unstructured data classification and grading method provided by this invention allows for flexible selection of the maximum or desired data level based on specific application scenarios, significantly enhancing the flexibility of data management and protection. It also ensures that organizations can effectively utilize existing resources while safeguarding information security. The set maximum security level aims to provide comprehensive data protection in extreme situations. For example, in practical operation, any modification to unstructured data can be protected using MDL (Multi-Level Documentation) to prevent unauthorized tampering of sensitive information. During routine data access, corresponding security measures can be implemented based on EDL (Electronic Data Definition Documentation) to maintain operational convenience and work efficiency. Attached Figure Description

[0051] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0052] Figure 1 This is a schematic diagram illustrating the principle of the multimodal fusion method for unstructured data classification and grading in this embodiment of the invention.

[0053] Figure 2 This is a flowchart of the multimodal fusion method for unstructured data classification and grading in an embodiment of the present invention;

[0054] Figure 3 This is a flowchart illustrating the process of obtaining a fused representation vector through multimodal data fusion in an embodiment of the present invention.

[0055] Figure 4 This is a schematic diagram of the document data multimodal fusion model architecture in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the multimodal fusion model architecture for image data in an embodiment of the present invention;

[0057] Figure 6 This is a schematic diagram of the audio data multimodal fusion model architecture in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the video data multimodal fusion model architecture in an embodiment of the present invention;

[0059] Figure 8 This is a flowchart illustrating the process of obtaining multi-modal data categories based on probability vector P in an embodiment of the present invention. Detailed Implementation

[0060] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0061] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0062] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0063] In the description of this invention, it should be understood that the terms "upper" and "lower" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0064] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral connection; it can refer to a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0065] Existing deep learning methods suffer from insufficient information extraction capabilities, low accuracy, and poor flexibility when dealing with the classification and grading of unstructured data. To address these issues, this invention proposes a multimodal fusion method and system for the classification and grading of unstructured data. It innovatively introduces multimodal fusion technology to fully mine and utilize various effective information within unstructured data. Furthermore, by employing multimodal classification and grading technology, multiple categories, maximum data levels, and desired data levels are assigned to unstructured data, significantly improving the classification and grading performance.

[0066] In a first aspect, the present invention provides a multimodal fusion method for multivariate classification and grading of unstructured data, see [link to relevant documentation]. Figures 1 to 2 It includes the following steps:

[0067] S1. Multimodal data fusion to obtain a fused representation vector;

[0068] As one possible implementation, see Figure 3 S1 includes:

[0069] S10. Apply a semantic understanding model to represent the metadata of the unstructured data to obtain a metadata representation vector; for example, the metadata includes at least the file name, directory name, creator, descriptive information, data source, and creation date; apply the RoBERTa semantic understanding model for representation.

[0070] S11. Apply a semantic understanding model to represent the unstructured data itself to obtain an unstructured data representation vector;

[0071] S12. Stack the metadata representation vector and the unstructured data representation vector in the channel dimension;

[0072] S13. Apply the convolution kernel of the convolutional neural network to fuse the data in the above multiple channels to obtain a fused representation vector.

[0073] In practical applications, unstructured data, in addition to the data itself, also encompasses a wealth of metadata that aids in data classification, such as filenames, directory names, creators, descriptive information, data sources, and creation dates. The semantic content inherent in this information, combined with the unstructured data itself, can fully utilize more contextual information, thereby improving the accuracy of data classification. This invention uses multimodal deep fusion technology to combine this information and then provides it to the classification model for prediction.

[0074] Metadata is natural language in text form, while unstructured data represents different modalities and cannot be directly fused. This invention employs deep neural networks to perform deep representations of data from different modalities, and then uses convolutional neural networks to fuse the representation vectors from these different modalities. Metadata is typically short and highly information-rich, making it suitable for representation using the RoBERTa semantic understanding model. The most crucial aspect of representing metadata is its organization. This invention, leveraging the characteristics of semantic understanding models, organizes cloud data in the following way:

[0075] "File name:'{file_name}'[sep] Directory name:'{folder_name}'[sep] Creator:'{creator}'[sep] Description:'{description}'[sep] Data source:'{data_provider}'."

[0076] Here, `[sep]` is a special token used to prompt the AI ​​model to distinguish different types of metadata. `{file_name}` is the filename of the unstructured data, `{folder_name}` is the directory name where the unstructured data is located, `{creator}` is the creator of the unstructured data, `{description}` is the description or caption of the unstructured data, and `{data_provider}` is the data source. If a certain part of the metadata is unavailable, its corresponding value is set to empty. The advantages of this approach are: it maintains consistency with the RoBERTa pre-training method, maximizing the utilization of the model's capabilities; and it offers greater flexibility in handling metadata. Even if some metadata information is missing from certain unstructured data, the AI ​​model does not need to be modified and can still make classification predictions based on other metadata. However, the more comprehensive the information provided, the more accurate the model's category predictions.

[0077] Metadata for all unstructured data is organized in this way and represented using RoBERTa or other similar semantic understanding models. The representations are then fused with information representations from other modalities via convolutional neural networks.

[0078] As an example, unstructured data includes document data, which includes formats such as txt, word, ppt, and pdf. Except for txt, other document formats not only contain text content but also many images. Therefore, document data has three modalities: metadata, text data, and images.

[0079] See the document data multimodal fusion model architecture. Figure 4 In this process, metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors; text data is represented using the Longformer semantic understanding model to obtain text data representation vectors; images are first represented using the ViT semantic understanding model, and then the Transformer encoder is used to fuse the representation vectors of all images to obtain image representation vectors; the metadata representation vectors, text data representation vectors, and image representation vectors are stacked in the channel dimension and then input into a convolutional neural network; the convolutional kernel of the convolutional neural network fuses the metadata representation vectors, text data representation vectors, and image representation vectors in the three channels to obtain a fused representation of the document data.

[0080] As an example, unstructured data includes image data, which has both metadata and image modalities. See the image data multimodal fusion model architecture for details. Figure 5 In this process, metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors; images are first represented using the ViT semantic understanding model, and then the Transformer encoder is used to fuse the representation vectors of all images to obtain image representation vectors; the metadata representation vectors and image representation vectors are stacked in the channel dimension and then input into the convolutional neural network; the convolution kernel of the convolutional neural network fuses the metadata representation vectors and image representation vectors in the two channels to obtain the fused representation of the image data.

[0081] As an example, unstructured data includes audio data, which has two modalities: metadata and audio signal; see the architecture of the audio data multimodal fusion model for details. Figure 6 In this process, metadata is represented using the RoBERTa semantic understanding model to obtain a metadata representation vector; the audio signal is preprocessed to obtain Mel-frequency inverse coefficients, which are then represented using the Conformer semantic understanding model to obtain an audio signal representation vector; the metadata representation vector and the audio signal representation vector are stacked in the channel dimension and then input into a convolutional neural network; the convolution kernel of the convolutional neural network fuses the metadata representation vector and the audio signal representation vector on the two channels to obtain a fused representation of the audio data.

[0082] As an example, unstructured data includes video data, which has four modalities: metadata, subtitle sequences, audio signals, and frame sequences; see the architecture of the video data multimodal fusion model for details. Figure 7 In this process, metadata is represented using the RoBERTa semantic understanding model to obtain a metadata representation vector; subtitle sequences are represented using the RoBERTa semantic understanding model to obtain a subtitle sequence representation vector; audio signals are represented using the Conformer semantic understanding model to obtain an audio signal representation vector; and frame sequences are represented using the VTN-ENCODER semantic understanding model to obtain a frame sequence representation vector. The metadata representation vector, subtitle sequence representation vector, audio signal representation vector, and frame sequence representation vector are stacked along the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vector, subtitle sequence representation vector, audio signal representation vector, and frame sequence representation vector on the four channels to obtain a fused representation of the video data.

[0083] S2. The fused representation vector is input into a neural network module to output an M-dimensional vector, where M is the number of categories. For example, the fused representation vector is input into a neural network module for classification. This neural network module can be a multilayer perceptron, a convolutional neural network, etc., as specifically defined in this embodiment. The output of this neural network module is an M-dimensional vector called Logits, where M is the number of categories in the classification task. Logits are expressed by the following formula:

[0084]

[0085] S3. Transform the M-dimensional vector to obtain the probability vector P; for example, the softmax function is used to transform the Logits vector to obtain a probability vector P, expressed by the following formula:

[0086]

[0087] S4. Obtaining the multivariate data categories of multimodal data based on the probability vector P; for example, P represents a probability distribution, where each probability value represents the probability that the input data belongs to the corresponding category, and the sum of all probabilities is 1. Because unstructured data contains complex and diverse information, a file may simultaneously possess multiple categories. Therefore, this embodiment of the invention obtains the multivariate data categories of multimodal data based on the probability vector P. See also Figure 8 Specifically, it includes:

[0088] S40. Set parameters k and t, where k is the maximum value of the selectable categories and t is the probability threshold; for example, k can be set according to the actual business situation, and t can be obtained by statistically analyzing the predicted probabilities of data categories in the training set and test set based on the classification model.

[0089] S41. Select categories from the probability vector P whose probability values ​​are greater than or equal to the probability threshold t;

[0090] S42. Sort the selected categories in descending order of probability value. When the number of selected categories is greater than or equal to k, take the first k categories as the multivariate data categories of the unstructured data; when the number of selected categories is less than k, take all categories as the multivariate data categories of the unstructured data. For example, define DC to represent the multivariate data categories of unstructured data. Assuming that m categories are finally assigned to a piece of unstructured data, DC can be represented as:

[0091]

[0092] Correspondingly, the index of the class in the probability vector P in the DC is represented as:

[0093]

[0094] S5. Calculate the maximum data level and the expected data level for unstructured data;

[0095] Before performing multivariate classification on unstructured data to determine its maximum and expected data levels, it is necessary to calculate the corresponding level for each potential category. According to GB / T 43697-2024 "Data Security Technology - Data Classification and Grading Rules," the data level is jointly determined by the data's impact object and data classification elements (such as domain, group, region, precision, scale, depth, coverage, importance, etc.). These elements can be obtained based on data category and data attributes through certain rules. Therefore, it can be considered that there is a mapping relationship between data level and data category and data attributes. This invention does not discuss the specific mapping relationship but defines it as a function.

[0096] As one possible implementation, the relevant data attribute of unstructured data is defined as attr, and for one of its data categories, dc... l Its corresponding data level dl l It can be represented as:

[0097] dl l =f(dcl,atte) (6)

[0098] The maximum data level is calculated using the following method:

[0099] S50. Calculate the data level vector corresponding to each category in the multivariate data categories;

[0100] For example, according to dl l Calculate the data level vector (DL) corresponding to each category and represent it as follows:

[0101]

[0102] S51. Select the largest data level value from the data level vector as the largest data level for unstructured data;

[0103] For example, selecting the value with the largest data level from the DL as the maximum data level mdl for unstructured data is represented as:

[0104]

[0105] The expected data level is calculated using the following method:

[0106] S52. Based on the index of each category in the probability vector in the multivariate data categories, obtain its corresponding logits from the M-dimensional vector, and further represent it as CLogits;

[0107] For example, based on CIndex, the corresponding logits are obtained from the Logits vector, denoted as CLogits:

[0108]

[0109] S53. Convert CLogits into a probability vector CP, and calculate the expected data level based on the probability vector CP.

[0110] For example, the softmax function is used to convert CLogits into a probability vector CP, as shown in the following equation:

[0111]

[0112] As one possible implementation, the expected data level calculation formula is:

[0113]

[0114]

[0115] Where edl represents the expected data level; 1≤l≤m, l is the l-th category in the multivariate data categories of unstructured data, and m is the number of categories in the multivariate data categories of unstructured data; cp l This represents the data category DC. l The probability, idx l Let be the index of the l-th category in the probability vector P of the multivariate data categories of unstructured data. It is the logit corresponding to the l-th category in the multivariate data categories of unstructured data, where j is a variable representing 1, 2, 3, ..., m. It is the logit corresponding to the j-th category in a multivariate dataset of unstructured data. The relevant data attribute of the unstructured data is attr. For a data category dc l Its corresponding data level is dl l dl l =f(dc) l ,attr).

[0116] Thus, each unstructured data category has a maximum data level (MDL) and a desired data level (EDL). The appropriate level can be flexibly chosen based on the specific application scenario. This strategy not only enhances the flexibility of data management and protection but also ensures that organizations can effectively utilize existing resources while safeguarding information security. The maximum security level is designed to provide comprehensive data protection in extreme situations. For example, in practice, any modification to unstructured data can be protected using the MDL to prevent unauthorized alteration of sensitive information. During routine data access, appropriate security measures can be implemented based on the EDL, thereby maintaining operational convenience and work efficiency.

[0117] Secondly, embodiments of the present invention provide a multimodal fusion-based unstructured data multivariate classification and grading system, comprising:

[0118] The multimodal data fusion module receives unstructured data, represents the metadata of the unstructured data and stacks it along the channel dimension, and applies the convolution kernel of the convolutional neural network to fuse the data in multiple channels to obtain a fused representation vector.

[0119] The neural network module receives the fused representation vector and outputs an M-dimensional vector, where M is the number of categories;

[0120] The transformation module transforms the M-dimensional vector to obtain the probability vector P;

[0121] The multivariate data category determination module is used to perform the first aspect S40 to S42 to obtain the multivariate data categories;

[0122] The data level calculation module is used to execute S50 to S51 of the first aspect to obtain the maximum data level; it is also used to execute S52 to S53 of the first aspect to obtain the desired data level.

[0123] In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A multimodal fusion method for multivariate classification and grading of unstructured data, characterized in that, Includes the following steps: S1. Multimodal data fusion to obtain a fused representation vector; S2. The fused representation vector is input to the neural network module to output an M-dimensional vector, where M is the number of categories; S3. Transform the M-dimensional vector to obtain the probability vector P; S4. Obtain the multivariate data categories of multimodal data based on the probability vector P, specifically including: Metadata should include at least the filename, directory name, creator, descriptive information, data source, and creation date; S40. Set parameters k and t, where k is the maximum value of the selectable categories and t is the probability threshold; S41. Select categories from the probability vector P whose probability values ​​are greater than or equal to the probability threshold t; S42. Sort the selected categories in descending order of probability value. When the number of selected categories is greater than or equal to k, take the first k categories as the multivariate data categories of the unstructured data; when the number of selected categories is less than k, take all categories as the multivariate data categories of the unstructured data. The unstructured data includes document data, which has three modalities: metadata, text data, and images. The metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors; the text data is represented using the Longformer semantic understanding model to obtain text data representation vectors; the images are first represented using the ViT semantic understanding model, and then the representation vectors of all images are fused using a Transformer encoder to obtain image representation vectors; the metadata representation vectors, text data representation vectors, and image representation vectors are stacked in the channel dimension and then input into a convolutional neural network; the convolutional kernel of the convolutional neural network fuses the metadata representation vectors, text data representation vectors, and image representation vectors in the three channels to obtain a fused representation of the document data; S5. Calculate the maximum data level and the expected data level for unstructured data; the maximum data level is calculated using the following method: S50. Calculate the data level vector corresponding to each category in the multivariate data categories; S51. Select the largest data level value from the data level vector as the largest data level for unstructured data; The expected data level is calculated using the following method: S52. Based on the index of each category in the probability vector in the multivariate data categories, obtain its corresponding logits from the M-dimensional vector, and further represent it as CLogits; S53. Convert CLogits into a probability vector CP, and calculate the expected data level based on the probability vector CP; the formula for calculating the expected data level is: ， in, For the expected data level; , The first category of unstructured data in multivariate data Categories The number of categories in the multivariate data categories of unstructured data; This indicates the data category. The probability, , The first category of unstructured data in multivariate data The index of each category in the probability vector P It is the first in the multivariate data category of unstructured data. Logit corresponding to each category j It is a variable representing 1, 2, 3, ... m , It is the first in the multivariate data category of unstructured data. For each category, the corresponding logit, and the relevant data attribute of the unstructured data, is attr. Its corresponding data level is , .

2. The multimodal fusion method for unstructured data classification and grading according to claim 1, characterized in that, S1 includes: S10. Apply a semantic understanding model to represent the metadata of unstructured data to obtain a metadata representation vector; S11. Apply a semantic understanding model to represent the unstructured data itself to obtain an unstructured data representation vector; S12. Stack the metadata representation vector and the unstructured data representation vector along the channel dimension; S13. The convolution kernel of the convolutional neural network is used to fuse the data in the above multiple channels to obtain a fused representation vector.

3. The multimodal fusion method for unstructured data classification and grading according to claim 1, characterized in that, The unstructured data includes image data, which has two modalities: metadata and images. The metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors. The images are first represented using the ViT semantic understanding model, and then the representation vectors of all images are fused using a Transformer encoder to obtain image representation vectors. The metadata representation vectors and image representation vectors are stacked in the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vectors and image representation vectors from both channels to obtain a fused representation of the image data.

4. The multimodal fusion method for unstructured data classification and grading according to claim 1, characterized in that, The unstructured data includes audio data, which has two modalities: metadata and audio signal. The metadata is represented using the RoBERTa semantic understanding model to obtain a metadata representation vector. The audio signal is preprocessed to obtain Mel-frequency inverse coefficients, which are then represented using the Conformer semantic understanding model to obtain an audio signal representation vector. The metadata representation vector and the audio signal representation vector are stacked along the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vector and the audio signal representation vector from both channels to obtain a fused representation of the audio data.

5. The multimodal fusion method for unstructured data classification and grading according to claim 1, characterized in that, The unstructured data includes video data, which has four modalities: metadata, subtitle sequences, audio signals, and frame sequences. The metadata is represented using the RoBERTa semantic understanding model to obtain a metadata representation vector; the subtitle sequences are represented using the RoBERTa semantic understanding model to obtain a subtitle sequence representation vector; the audio signals are represented using the Conformer semantic understanding model to obtain an audio signal representation vector; and the frame sequences are represented using the VTN-ENCODER semantic understanding model to obtain a frame sequence representation vector. The metadata representation vector, subtitle sequence representation vector, audio signal representation vector, and frame sequence representation vector are stacked along the channel dimension and then input into a convolutional neural network. The convolutional kernel of the convolutional neural network fuses the metadata representation vector, subtitle sequence representation vector, audio signal representation vector, and frame sequence representation vector from the four channels to obtain a fused representation of the video data.

6. The multimodal fusion method for unstructured data classification and grading according to claim 1, characterized in that, Define the relevant data attribute of unstructured data as attr, for one of its data categories. Its corresponding data level Represented as: ， according to Calculate the data level vector corresponding to each category And expressed as: ， from Select the value with the largest data level as the maximum data level for unstructured data. , is represented as: 。 7. A multimodal fusion system for classifying and grading unstructured data, characterized in that, include: The multimodal data fusion module receives unstructured data, represents the metadata of the unstructured data and stacks it along the channel dimension, and applies the convolution kernel of the convolutional neural network to fuse the data in multiple channels to obtain a fused representation vector. The neural network module receives the fused representation vector and outputs an M-dimensional vector, where M is the number of categories; The transformation module transforms the M-dimensional vector to obtain the probability vector P; Metadata should include at least the filename, directory name, creator, descriptive information, data source, and creation date; A multi-data category determination module is used to execute S40-S42 as described in claim 1 to obtain multi-data categories; Unstructured data includes document data, which has three modalities: metadata, text data, and images. The metadata is represented using the RoBERTa semantic understanding model to obtain metadata representation vectors; the text data is represented using the Longformer semantic understanding model to obtain text data representation vectors; the images are first represented using the ViT semantic understanding model, and then the representation vectors of all images are fused using a Transformer encoder to obtain image representation vectors; the metadata representation vectors, text data representation vectors, and image representation vectors are stacked in the channel dimension and then input into a convolutional neural network; the convolutional kernel of the convolutional neural network fuses the metadata representation vectors, text data representation vectors, and image representation vectors in the three channels to obtain a fused representation of the document data; The data level calculation module is used to execute S50-S51 as described in claim 1 to obtain the maximum data level; it is also used to execute S52-S53 as described in claim 1 to obtain the desired data level, wherein the formula for calculating the desired data level is: ， in, For the expected data level; , The first category of unstructured data in multivariate data Categories The number of categories in the multivariate data categories of unstructured data; This indicates the data category. The probability, , The first category of unstructured data in multivariate data The index of each category in the probability vector P It is the first in the multivariate data category of unstructured data. Logit corresponding to each category j It is a variable representing 1, 2, 3, ... m , It is the first in the multivariate data category of unstructured data. For each category, the corresponding logit, and the relevant data attribute of the unstructured data, is attr. Its corresponding data level is , .

Images