A cloud platform auditing method based on multimodal data processing

Abstract

This invention discloses a cloud platform review method based on multimodal data processing, belonging to the field of cloud platform review technology. This invention extracts independent features from text, image, audio, video, and log data, enabling detailed analysis of each modality's information in its corresponding feature space. In the cross-modal feature fusion stage, the Cross-Modal Consistency Network (CMCN) is used to calculate the correlation between different modalities, and an attention mechanism is used to optimize feature alignment, allowing multimodal information to mutually verify each other and reducing misjudgments and omissions caused by independent reviews. In the review decision-making stage, a deep learning-based review model is constructed, combined with anomaly detection and adaptive risk control thresholds, to achieve end-to-end violation content judgment and improve the intelligence level of the review system. Furthermore, by visually annotating suspicious areas in images and videos and using a Natural Language Generator (NLG) model to generate review conclusions, the review results are made more transparent.

Description

Technical Field

[0001] This invention relates to the field of cloud platform auditing technology, and in particular to a cloud platform auditing method based on multimodal data processing. Background Technology

[0002] Cloud platforms involve a large number of data storage and computing tasks, including various types of data such as text, images, videos, and audio. As the demand for content regulation increases, platforms need to review stored files, streaming media, and user-uploaded content for violations.

[0003] Current cloud platform review methods mostly employ deep learning combined with rule engines, such as keyword matching or natural language processing (NLP) detection, and computer vision technology to identify copyright infringement. However, the review processes for various types of data are independent, lacking correlation analysis, which easily leads to misjudgments or omissions. For example, text logs may appear normal, but the accompanying images or videos may be in violation, or the video footage may be compliant, but the background audio content may be in violation. In addition, current review methods also suffer from computational resource consumption issues. In a large-scale cloud platform environment, the computational cost of processing massive amounts of data in parallel is very high, resulting in limited review efficiency.

[0004] To address the aforementioned issues, some traditional review solutions employ hierarchical optimization strategies, increase manual review, or use lightweight pre-screening models for rapid filtering. However, these methods are insufficient to fundamentally improve review efficiency and accuracy. Therefore, a cloud platform review method based on multimodal data processing is urgently needed to solve these problems. Summary of the Invention

[0005] In view of the aforementioned existing problems, the present invention is proposed.

[0006] This invention provides a cloud platform auditing method based on multimodal data processing to solve the problems of traditional auditing methods, such as weak cross-modal correlation, high computational resource consumption, serious false positives and false negatives, lack of interpretability of audit results, and difficulty in meeting the needs of efficient and accurate auditing.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] This invention provides a cloud platform auditing method based on multimodal data processing, comprising:

[0009] Step S1: Collect the storage and transmission data of the cloud platform and perform data preprocessing, including format standardization, noise removal, integrity verification and metadata extraction.

[0010] Step S2: Based on the preprocessed data from step S1, feature extraction is performed on the text, image, audio, video, and log data respectively to obtain the features of each modality;

[0011] Step S3: Based on the modal features extracted in step S2, perform cross-modal feature fusion to obtain unified multimodal features;

[0012] Step S4: Based on the multimodal features generated in step S3, construct an audit model to determine the violations of the data;

[0013] Step S5: Based on the audit results of step S4, perform interpretability analysis and generate an audit report.

[0014] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the stored and transmitted data includes text, images, audio, video, and log data.

[0015] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the feature extraction includes:

[0016] Natural language processing models are used for text analysis to identify keywords, named entities, and sentiment.

[0017] Computer vision models are used for image and video analysis to detect target objects, optical characters, and illegal elements.

[0018] Automatic speech recognition technology is used to transcribe audio and perform semantic, emotion, and timbre analysis.

[0019] Analyze log data to extract cloud platform call and access records and abnormal traffic characteristics.

[0020] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the step of extracting features from text, image, audio, video, and log data is as follows:

[0021] Natural language processing methods are used to extract key features from text data and convert the text data into word vectors.

[0022]

[0023] in, The text global vector representation of time step t. The number of text words at time step t. Let be the weight of the i-th word. This is the embedding vector corresponding to the word.

[0024] Keyword weights are calculated based on TF-IDF and named entity recognition, using the following formula:

[0025]

[0026] in, For the score of the j-th keyword, M represents the frequency of the keyword in the text. T The total number of all texts, The number of texts containing that keyword.

[0027] Sentiment bias is calculated using a bidirectional long short-term memory network, represented as:

[0028]

[0029] in, This represents the hidden state at time step t. This is the hidden state from the previous time step. Here is the hidden state weight matrix. The word vector at the current time step t. Here, σ is the bias term, and σ is the activation function.

[0030] Image and video features are extracted using computer vision models.

[0031] Object detection using convolutional neural networks can be represented as follows:

[0032]

[0033] in, Represents the feature mapping of an image. For convolution kernel, For input image data, The term represents the bias term, and * indicates the convolution operation.

[0034] Connection-time classification computation:

[0035]

[0036] in, The character probability predicted by OCR. For the set of possible character paths, The character path for time step t. This represents the total number of time steps. The predicted probability of the character corresponding to time step t;

[0037] Audio features are extracted using automatic speech recognition.

[0038] Speech features are calculated based on Mel-frequency coefficients, using the following formula:

[0039]

[0040] in, For the nth A MFCC coefficients, The k-th frequency component represents the signal energy after Mel-filtering. To determine the size of the Fourier transform window,

[0041] Sentiment bias is calculated using deep neural networks. The calculation process is as follows:

[0042]

[0043] in, This represents the hidden state at time step t. This is the weight matrix. For input features, For bias terms;

[0044] Analyze cloud platform call and access records and abnormal traffic characteristics.

[0045] Access modeling is performed using time series prediction LSTM computation:

[0046]

[0047] in, The hidden state at the current time step. This is the hidden state from the previous time step. Characteristics of log events, and This is the weight matrix. For bias,

[0048] Calculate outlier distributions using probability density estimation of the KDE:

[0049]

[0050] Where p(x) L (x) represents the data point. L The probability density, h represents the number of log data samples. L For bandwidth, Let i be the value of sample i, and K(·) be the kernel function.

[0051] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the cross-modal feature fusion method includes:

[0052] A cross-modal consistency comparison network is used to calculate the correlation between features of different modalities;

[0053] An attention-based feature alignment method is used to extract important features across modalities.

[0054] A shared feature encoder is used to uniformly represent features of different modalities.

[0055] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the step of performing cross-modal feature fusion is as follows:

[0056] A cross-modal consistency comparison network is used to calculate the similarity of features from different modalities in a common feature space. The calculation formula is as follows:

[0057]

[0058] in, Let represent the Euclidean distance between modal p and modal q, where p,q∈{T,I,A,V,L} correspond to the five modalities: text, image, audio, video, and log. The feature vector extracted for mode p. Let ||·||^2 represent the feature vector extracted from mode q, where ||·||^2 represents the squared L2 norm.

[0059] Cross-modal consistency is defined using a similarity function, the formula of which is:

[0060]

[0061] in, The similarity between mode p and mode q. For the temperature parameters of mode p,

[0062] An attention mechanism is used to extract important cross-modal features. The extraction formula is as follows:

[0063]

[0064] in, R represents the attention weights for mode r. M The total number of all modes, Let r be the attention weight matrix. The feature vector extracted for mode r.

[0065] The cross-modal feature fusion representation is calculated using the following formula:

[0066]

[0067] in, For cross-modal fusion feature representation;

[0068] The shared feature encoder unifies the features from different modalities to obtain the final multimodal features, which are represented as follows:

[0069]

[0070] in, For the final multimodal feature representation, To share the feature encoding matrix, This is a bias term.

[0071] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the auditing model includes the following auditing methods:

[0072] Combine with a rules engine for rapid content screening;

[0073] A deep learning model is used for end-to-end classification, and a violation score is output.

[0074] Analyze log behavior data using anomaly detection models to identify potential anomalies;

[0075] Set dynamic risk thresholds and conduct secondary reviews of high-risk content.

[0076] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the step of constructing an auditing model based on the multimodal features generated in step S3 and determining data violations is as follows:

[0077] The review model uses deep learning to calculate violation scores, and the scoring formula is as follows:

[0078]

[0079] in, For violating the rating rules, This is the output layer weight matrix. Here, σ is the bias term, and σ is the activation function.

[0080] Define the basis for determining violations:

[0081]

[0082] in, The result of the violation judgment, Let I(·) be the threshold for determining violations, and let I(·) be the indicator function. but otherwise

[0083] The threshold for determining abnormal behavior is calculated based on dynamic threshold adjustment. The calculation formula is as follows:

[0084]

[0085] in, For dynamic risk thresholds, The average of historical data. To adjust the coefficient, The standard deviation of the data characteristics

[0086] The anomaly detection model is used to calculate data anomalies, and the formula is as follows:

[0087]

[0088] in, For anomaly detection, I(·) is the indicator function; when hour otherwise

[0089] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the audit report includes:

[0090] Visualization technology is used to mark suspicious areas in images and videos;

[0091] Generate review conclusions and evidence of violations by combining language models;

[0092] Add a time index to the audio and video review results.

[0093] As a preferred embodiment of the cloud platform auditing method based on multimodal data processing described in this invention, the step of performing interpretability analysis and generating an audit report based on the audit results of step S4 is as follows:

[0094] For images and videos, suspicious areas are marked using the following method:

[0095]

[0096] in, This is a set of coordinates for the suspicious area. These are the pixel coordinates in an image or video frame. For pixel violation probability, Risk threshold for suspicious areas

[0097] Generate review conclusions using natural language generation models

[0098]

[0099] in, For the audit report text, For multimodal features The generated language description function,

[0100] Add an approval time index for audio and video files:

[0101]

[0102] in, This is a set of time indexes in the audit report. For audio violation timestamps, For the video violation timestamp, and For time step and The result of the violation determination.

[0103] The beneficial effects of this invention are as follows: This invention extracts independent features from text, image, audio, video, and log data, enabling detailed analysis of each modality's information in its corresponding feature space. In the cross-modal feature fusion stage, the Cross-Modal Consistency Network (CMCN) is used to calculate the correlation between different modal data, and the feature alignment is optimized through an attention mechanism. This allows text, image, audio, and video data to be represented in the same feature space, enabling multimodal information to mutually verify each other and reducing misjudgments and omissions caused by independent review.

[0104] This invention constructs a deep learning-based review model in the review decision-making process, and combines anomaly detection and adaptive risk control thresholds to achieve end-to-end judgment of illegal content, thereby improving the intelligence level of the review system. In addition, by visually annotating suspicious areas of images and videos, generating review conclusions using a natural language processing (NLG) model, and adding time indexes to audio and video review results, the review results are made more transparent, facilitating manual review and adjustment of review strategies.

[0105] In summary, compared with traditional methods, this invention reduces computational resource consumption, improves review efficiency, and lowers the false positive and false negative rates. Attached Figure Description

[0106] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 This is a flowchart illustrating the cloud platform auditing method based on multimodal data processing according to the present invention. Detailed Implementation

[0108] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0109] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0110] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0111] Example 1, referring to Figure 1 This embodiment provides a cloud platform auditing method based on multimodal data processing, including the following steps:

[0112] Step S1: Collect the storage and transmission data of the cloud platform and perform data preprocessing, including format standardization, noise removal, integrity verification and metadata extraction.

[0113] The stored and transmitted data includes text, images, audio, video, and log data;

[0114] Step S2: Based on the preprocessed data from step S1, feature extraction is performed on the text, image, audio, video, and log data respectively to obtain the features of each modality;

[0115] The feature extraction includes:

[0116] Natural language processing models are used for text analysis to identify keywords, named entities, and sentiment.

[0117] Computer vision models are used for image and video analysis to detect target objects, optical characters, and illegal elements.

[0118] Automatic speech recognition technology is used to transcribe audio and perform semantic, emotion, and timbre analysis.

[0119] Analyze log data to extract cloud platform call and access records and abnormal traffic characteristics;

[0120] The steps for feature extraction from text, image, audio, video, and log data are as follows:

[0121] Natural language processing methods are used to extract key features from text data and convert the text data into word vectors.

[0122]

[0123] in, The text global vector representation of time step t. The number of text words at time step t. Let be the weight of the i-th word. This is the embedding vector corresponding to the word.

[0124] Keyword weights are calculated based on TF-IDF and named entity recognition, using the following formula:

[0125]

[0126] in, For the score of the j-th keyword, M represents the frequency of the keyword in the text. T The total number of all texts, The number of texts containing that keyword.

[0127] Sentiment bias is calculated using a bidirectional long short-term memory network, represented as:

[0128]

[0129] in, This represents the hidden state at time step t. This is the hidden state from the previous time step. Here is the hidden state weight matrix. The word vector at the current time step t. Here, σ is the bias term, and σ is the activation function.

[0130] Image and video features are extracted using computer vision models.

[0131] Object detection using convolutional neural networks can be represented as follows:

[0132]

[0133] in, Represents the feature mapping of an image. For convolution kernel, For input image data, The term represents the bias term, and * indicates the convolution operation.

[0134] Connection-time classification computation:

[0135]

[0136] in, The character probability predicted by OCR. For the set of possible character paths, The character path for time step t. This represents the total number of time steps. The predicted probability of the character corresponding to time step t;

[0137] Audio features are extracted using automatic speech recognition.

[0138] Speech features are calculated based on Mel-frequency coefficients, using the following formula:

[0139]

[0140] in, For the nth A MFCC coefficients, The k-th frequency component represents the signal energy after Mel-filtering. To determine the size of the Fourier transform window,

[0141] Sentiment bias is calculated using deep neural networks. The calculation process is as follows:

[0142]

[0143] in, This represents the hidden state at time step t. This is the weight matrix. For input features, For bias terms;

[0144] Analyze cloud platform call and access records and abnormal traffic characteristics.

[0145] Access modeling is performed using time series prediction LSTM computation:

[0146]

[0147] in, The hidden state at the current time step. This is the hidden state from the previous time step. Characteristics of log events, and This is the weight matrix. For bias,

[0148] Calculate outlier distributions using probability density estimation of the KDE:

[0149]

[0150] Where p(x) L (x) represents the data point. L The probability density, h represents the number of log data samples. L For bandwidth, Let i be the value of sample i, and K(·) be the kernel function.

[0151] Specifically, in step S2, the feature calculations for different modalities are refined. For the text modality, word vector embedding, TFIDF, and BiLSTM are used to extract sentiment and semantic features. For the image and video modality, CNN and OCR are used to identify target and text information. For the audio modality, MFCC is used to extract speech features. For the log modality, LSTM is used to analyze time series patterns to avoid cross-modal confusion.

[0152] Step S3: Based on the modal features extracted in step S2, perform cross-modal feature fusion to obtain unified multimodal features;

[0153] The cross-modal feature fusion method includes:

[0154] A cross-modal consistency comparison network is used to calculate the correlation between features of different modalities;

[0155] An attention-based feature alignment method is used to extract important features across modalities.

[0156] A shared feature encoder is used to uniformly represent features of different modalities;

[0157] The steps for cross-modal feature fusion are as follows:

[0158] A cross-modal consistency comparison network is used to calculate the similarity of features from different modalities in a common feature space. The calculation formula is as follows:

[0159]

[0160] in, Let represent the Euclidean distance between modal p and modal q, where p,q∈{T,I,A,V,L} correspond to the five modalities: text, image, audio, video, and log. The feature vector extracted for mode p. Let ||·||^2 represent the feature vector extracted from mode q, where ||·||^2 represents the squared L2 norm.

[0161] Cross-modal consistency is defined using a similarity function, the formula of which is:

[0162]

[0163] in, The similarity between mode p and mode q. For the temperature parameters of mode p,

[0164] An attention mechanism is used to extract important cross-modal features. The extraction formula is as follows:

[0165]

[0166] in, R represents the attention weights for mode r. M The total number of all modes, Let r be the attention weight matrix. The feature vector extracted for mode r.

[0167] The cross-modal feature fusion representation is calculated using the following formula:

[0168]

[0169] in, For cross-modal fusion feature representation;

[0170] The shared feature encoder unifies the features from different modalities to obtain the final multimodal features, which are represented as follows:

[0171]

[0172] in, For the final multimodal feature representation, To share the feature encoding matrix, For bias terms;

[0173] Specifically, in step S3, cross-modal consistency calculation, attention mechanism alignment, and shared feature encoding are used to fuse features from different modalities, enabling the information from each modality to be represented in the same feature space.

[0174] The correlation between modes is calculated by Euclidean distance, the importance weights between modes are extracted by attention mechanism, and finally a shared feature encoder is used to unify the representation of different modes.

[0175] Step S4: Based on the multimodal features generated in step S3, construct an audit model to determine the violations of the data;

[0176] The audit methods of the audit model include:

[0177] Combine with a rules engine for rapid content screening;

[0178] A deep learning model is used for end-to-end classification, and a violation score is output.

[0179] Analyze log behavior data using anomaly detection models to identify potential anomalies;

[0180] Set dynamic risk thresholds and conduct secondary reviews of high-risk content;

[0181] The steps for constructing an audit model based on the multimodal features generated in step S3 and determining data violations are as follows:

[0182] The review model uses deep learning to calculate violation scores, and the scoring formula is as follows:

[0183]

[0184] in, For violating the rating rules, This is the output layer weight matrix. Here, σ is the bias term, and σ is the activation function.

[0185] Define the basis for determining violations:

[0186]

[0187] in, The result of the violation judgment, Let I(·) be the threshold for determining violations, and let I(·) be the indicator function. but otherwise

[0188] The threshold for determining abnormal behavior is calculated based on dynamic threshold adjustment. The calculation formula is as follows:

[0189]

[0190] in, For dynamic risk thresholds, The average of historical data. To adjust the coefficient, The standard deviation of the data characteristics

[0191] The anomaly detection model is used to calculate data anomalies, and the formula is as follows:

[0192]

[0193] in, For anomaly detection, I(·) is the indicator function; when hour otherwise

[0194] Specifically, in step S4, a deep learning review model is constructed based on the multimodal features generated in step S3 to determine data violations and identify abnormal behavior. The review model uses a multi-layer neural network to calculate the violation score and combines a dynamic threshold adjustment algorithm to detect anomalies. In addition, a violation judgment mechanism is set up so that the model can accurately identify the risk level of content violations.

[0195] Step S5: Based on the audit results of step S4, perform interpretability analysis and generate an audit report;

[0196] The audit report includes:

[0197] Visualization technology is used to mark suspicious areas in images and videos;

[0198] Generate review conclusions and evidence of violations by combining language models;

[0199] Add a time index to the audio and video review results;

[0200] The steps for performing interpretability analysis and generating an audit report based on the audit results of step S4 are as follows:

[0201] For images and videos, suspicious areas are marked using the following method:

[0202]

[0203] in, This is a set of coordinates for the suspicious area. These are the pixel coordinates in an image or video frame. For pixel violation probability, Risk threshold for suspicious areas

[0204] Generate review conclusions using natural language generation models

[0205]

[0206] in, For the audit report text, For multimodal features The generated language description function,

[0207] Add an approval time index for audio and video files:

[0208]

[0209] in, This is a set of time indexes in the audit report. For audio violation timestamps, For the video violation timestamp, and For time step and The result of the violation judgment,

[0210] Specifically, the output of the audit model is transformed into an interpretable audit report here, which is convenient for manual review;

[0211] In terms of visual analysis, suspicious areas are marked for illegal content in images and videos, and high-risk areas are identified through pixel-level probability mapping.

[0212] In terms of text review, natural language generation models are used to generate review conclusions.

[0213] In terms of audio and video data review, a time index has been added to enable reviewers to quickly pinpoint the specific time when a violation occurred.

[0214] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A cloud platform auditing method based on multimodal data processing, characterized in that: include, Step S1: Collect the storage and transmission data of the cloud platform and perform data preprocessing, including format standardization, noise removal, integrity verification and metadata extraction. Step S2: Based on the preprocessed data from step S1, feature extraction is performed on the text, image, audio, video, and log data respectively to obtain the features of each modality; Step S3: Based on the modal features extracted in step S2, perform cross-modal feature fusion to obtain unified multimodal features; Step S4: Based on the multimodal features generated in step S3, construct an audit model to determine the violations of the data; Step S5: Based on the audit results of step S4, perform interpretability analysis and generate an audit report; The cross-modal feature fusion method includes: A cross-modal consistency comparison network is used to calculate the correlation between features of different modalities; An attention-based feature alignment method is used to extract important features across modalities. A shared feature encoder is used to uniformly represent features of different modalities; The steps for cross-modal feature fusion are as follows: A cross-modal consistency comparison network is used to calculate the similarity of features from different modalities in a common feature space. The calculation formula is as follows: in, Let represent the Euclidean distance between modal p and modal q, where p,q∈{T,I,A,V,L} correspond to the five modalities: text, image, audio, video, and log. The feature vector extracted for mode p. Let ||·||^2 represent the feature vector extracted from mode q, where ||·||^2 represents the squared L2 norm. Cross-modal consistency is defined using a similarity function, the formula of which is: in, The similarity between mode p and mode q. For the temperature parameters of mode p, An attention mechanism is used to extract important cross-modal features. The extraction formula is as follows: in, R represents the attention weights for mode r. M The total number of all modes, Let r be the attention weight matrix. The feature vector extracted for mode r. The cross-modal feature fusion representation is calculated using the following formula: in, For cross-modal fusion feature representation; The shared feature encoder unifies the features from different modalities to obtain the final multimodal features, which are represented as follows: in, For the final multimodal feature representation, To share the feature encoding matrix, For bias terms; The audit methods of the audit model include: Combine with a rules engine for rapid content screening; A deep learning model is used for end-to-end classification, and a violation score is output. Analyze log behavior data using anomaly detection models to identify potential anomalies; Set dynamic risk thresholds and conduct secondary reviews of high-risk content; The steps for constructing an audit model based on the multimodal features generated in step S3 and determining data violations are as follows: The review model uses deep learning to calculate violation scores, and the scoring formula is as follows: in, For violating the rating rules, This is the output layer weight matrix. Here, σ is the bias term, and σ is the activation function. Define the basis for determining violations: in, The result of the violation judgment, Let I(·) be the threshold for determining violations, and let I(·) be the indicator function. but otherwise The threshold for determining abnormal behavior is calculated based on dynamic threshold adjustment. The calculation formula is as follows: in, For dynamic risk thresholds, The average of historical data. To adjust the coefficient, The standard deviation of the data characteristics The anomaly detection model is used to calculate data anomalies, and the formula is as follows: in, For anomaly detection, I(·) is the indicator function; when hour otherwise 2. The cloud platform auditing method based on multimodal data processing as described in claim 1, characterized in that: The stored and transmitted data includes text, images, audio, video, and log data.

3. The cloud platform auditing method based on multimodal data processing as described in claim 2, characterized in that: The feature extraction includes: Natural language processing models are used for text analysis to identify keywords, named entities, and sentiment. Computer vision models are used for image and video analysis to detect target objects, optical characters, and illegal elements. Automatic speech recognition technology is used to transcribe audio and perform semantic, emotion, and timbre analysis. Analyze log data to extract cloud platform call and access records and abnormal traffic characteristics.

4. The cloud platform auditing method based on multimodal data processing as described in claim 3, characterized in that: The steps for feature extraction from text, image, audio, video, and log data are as follows: Natural language processing methods are used to extract key features from text data and convert the text data into word vectors. in, The text global vector representation of time step t. The number of text words at time step t. Let be the weight of the i-th word. This is the embedding vector corresponding to the word. Keyword weights are calculated based on TF-IDF and named entity recognition, using the following formula: in, For the score of the j-th keyword, M represents the frequency of the keyword in the text. T The total number of all texts, The number of texts containing that keyword. Sentiment bias is calculated using a bidirectional long short-term memory network, represented as: in, This represents the hidden state at time step t. This is the hidden state from the previous time step. Here is the hidden state weight matrix. The word vector at the current time step t. Here, σ is the bias term, and σ is the activation function. Image and video features are extracted using computer vision models. Object detection using convolutional neural networks can be represented as follows: in, Represents the feature mapping of an image. For convolution kernel, For input image data, The term represents the bias term, and * indicates the convolution operation. Connection-time classification computation: in, The character probability predicted by OCR. For the set of possible character paths, The character path for time step t. This represents the total number of time steps. The predicted probability of the character corresponding to time step t; Audio features are extracted using automatic speech recognition. Speech features are calculated based on Mel-frequency coefficients, using the following formula: in, For the nth A MFCC coefficients, The k-th frequency component represents the signal energy after Mel-filtering. To determine the size of the Fourier transform window, Sentiment bias is calculated using deep neural networks. The calculation process is as follows: in, This represents the hidden state at time step t. This is the weight matrix. For input features, For bias terms; Analyze cloud platform call and access records and abnormal traffic characteristics. Access modeling is performed using time series prediction LSTM computation: in, The hidden state at the current time step. This is the hidden state from the previous time step. Characteristics of log events, and This is the weight matrix. For bias, Calculate outlier distributions using probability density estimation of the KDE: Where p(x) L (x) represents the data point. L The probability density, h represents the number of log data samples. L For bandwidth, Let i be the value of sample i, and K(·) be the kernel function.

5. The cloud platform auditing method based on multimodal data processing as described in claim 4, characterized in that: The audit report includes: Visualization technology is used to mark suspicious areas in images and videos; Generate review conclusions and evidence of violations by combining language models; Add a time index to the audio and video review results.

6. The cloud platform auditing method based on multimodal data processing as described in claim 5, characterized in that: The steps for performing interpretability analysis and generating an audit report based on the audit results of step S4 are as follows: For images and videos, suspicious areas are marked using the following method: in, This is a set of coordinates for the suspicious area. These are the pixel coordinates in an image or video frame. For pixel violation probability, Risk threshold for suspicious areas Generate review conclusions using natural language generation models in, For the audit report text, For multimodal features The generated language description function, Add an approval time index for audio and video files: or in, This is a set of time indexes in the audit report. For audio violation timestamps, For the video violation timestamp, and For time step and The result of the violation determination.

Images