Examination and approval risk early warning method and system based on multi-mode sensitive content identification

Abstract

The invention relates to the technical field of risk early warning, in particular to an approval risk early warning method and system based on multi-mode sensitive content recognition. The method comprises the following steps: collecting text, image and audio data from an approval related source, preprocessing, mapping the text data into a word vector sequence, and capturing local word order features to generate a text feature vector; performing hierarchical feature extraction on the image data to obtain image feature vectors, performing framing processing on the audio data and extracting an MFCC coefficient to obtain audio feature vectors, splicing and fusing the multi-modal feature vectors, inputting the spliced and fused multi-modal feature vectors into a pre-training classification model to obtain a sensitive content existence probability, calculating a risk score according to the probability, and obtaining a sensitive content risk result; and automatic risk early warning is realized by dynamically setting an early warning threshold value. According to the method, complementary information of multi-source data is fully utilized, the limitation of single-mode identification in the aspects of accuracy and robustness is effectively overcome, and the comprehensiveness and reliability of risk identification in a complex approval scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of risk warning technology, and in particular to an approval risk warning method and system based on multimodal sensitive content recognition. Background Technology

[0002] With the rapid development of information technology, the information involved in various approval processes has become increasingly diverse, encompassing not only textual information but also data in multiple modalities such as images and audio. Traditional approval risk warning systems often analyze and process only single-modal data, such as relying solely on keyword searches in text to assess risk. This approach has significant limitations. Firstly, single-modal recognition cannot fully utilize the complementary information from multiple data sources, resulting in low accuracy in risk identification. For example, some images may contain sensitive scenes or symbols, but these potential risks cannot be captured by textual information alone. Secondly, single-modal recognition systems have poor robustness; when data contains noise, interference, or format changes, false positives or false negatives are prone to occur. For instance, audio data may be affected by environmental noise during transmission, leading to inaccurate risk identification results based on audio. Furthermore, existing approval risk warning systems lack the comprehensive analytical capabilities for multimodal data when handling complex approval scenarios. For example, in some contract approval scenarios, it is necessary not only to analyze the clauses in the contract text but also to perform correlation analysis on multimodal information such as seal images and signatory voice confirmations in order to comprehensively and accurately assess approval risks. However, current systems struggle to achieve efficient fusion and in-depth analysis of multimodal data, failing to meet the increasingly complex approval business needs. Summary of the Invention

[0003] The technical problem to be solved by the embodiments of the present invention is to provide an approval risk warning method and system based on multimodal sensitive content recognition, so as to solve the problems of incomplete and low accuracy of single-modal approval risk recognition in the prior art.

[0004] This invention discloses an approval risk warning method based on multimodal sensitive content identification, comprising: Text data, image data, and audio data are collected from relevant approval sources, and the collected data are preprocessed. The words in the preprocessed text data are mapped to a high-dimensional word vector sequence, and the local word order features in the word vector sequence are captured and aggregated to generate a text feature vector; Hierarchical feature extraction of low-level and high-level features is performed on the preprocessed image data to obtain a fixed-dimensional image feature vector; The preprocessed audio data is divided into frames, and the MFCC coefficients of each frame are extracted based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. The text feature vector, the image feature vector, and the audio feature vector are concatenated and fused, and the fused feature vector is input into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content. Risk scores are calculated based on the probability of the presence of various types of sensitive content, and warning thresholds are dynamically set for different approval scenarios. When the risk score exceeds the set warning threshold, a warning is automatically triggered.

[0005] Optionally, the step of collecting text data, image data, and audio data from approval-related sources, and preprocessing the collected data, includes: Configure a web crawler to collect the text data from approval-related websites, databases, and document management systems; convert documents of different formats in the text data into plain text; and remove HTML tags and special characters from the converted plain text using regular expression rules. Image data, including enterprise qualification certificates, engineering construction site photos and design drawings, is acquired by an image sensor or obtained from a pre-established image database. The image data is then filtered, and the filtered image data is further enhanced based on histogram equalization. The audio data, which includes meeting approval and discussion records and customer voice complaints, is recorded using a microphone array or extracted from a pre-established audio repository. The audio data is then subjected to sampling rate standardization and noise removal processing.

[0006] Optionally, the step of mapping words in the preprocessed text data into a high-dimensional word vector sequence, capturing local word order features in the word vector sequence, and aggregating to generate a text feature vector includes: Based on word embedding operations, each word in the preprocessed text data is mapped to a high-dimensional word vector, and all the word vectors are combined to generate the word vector sequence; Using a pre-trained convolutional neural network with multiple convolutional kernels of different sizes, a sliding window convolution operation is performed on the word vector sequence to extract several local word order features; Based on the extracted local word order features, the contextual relationships between the words are captured using an attention mechanism, and a corresponding weight is generated for each word. Based on the generated weights, the word vectors of all the words are weighted and aggregated to generate the text feature vector. The function expression for aggregating and generating the text feature vector is as follows:

[0007] In the formula, This represents the text feature vector generated by aggregation. Indicates the first The weight of each word, Indicates the first Word vectors of 1 word, Indicating the index of words, This indicates the total number of words.

[0008] Optionally, the hierarchical feature extraction of low-level and high-level features from the preprocessed image data to obtain a fixed-dimensional image feature vector includes: A first convolutional layer group consisting of multiple convolutional kernels is added. The preprocessed image data is input into the first convolutional layer group, and feature extraction is performed through convolution operations to obtain a low-level feature map containing edge and texture information. A second convolutional layer group is added, which consists of multiple convolutional kernels and has more convolutional layers than the first convolutional layer group. The low-level feature map is input into the second convolutional layer group. By increasing the number of feature channels, feature abstraction is performed to obtain an intermediate feature map containing local shape and combined features. A third convolutional layer group is added, consisting of multiple convolutional kernels and having more convolutional layers than the second convolutional layer group. The intermediate feature map is input into the third convolutional layer group, and deep feature extraction is performed through multi-layer nonlinear transformation to obtain a high-level feature map that represents the semantic content of the image. Pooling layers are added after the first convolutional layer group, the second convolutional layer group, and the third convolutional layer group to perform spatial downsampling on the corresponding feature maps and flatten the downsampled high-level feature maps into one-dimensional vectors. The flattened one-dimensional vector feature map is mapped to the image feature vector of fixed dimension through a fully connected layer.

[0009] Optionally, the step of performing frame segmentation on the preprocessed audio data and extracting the MFCC coefficients of each frame of data based on the Mel-frequency cepstral coefficient algorithm to obtain the audio feature vector includes: The preprocessed audio data is divided into frames according to a preset frame length and frame shift to obtain multiple audio frames. Windowing is applied to each audio frame after frame segmentation, and the time-domain signal of the windowed audio frame is converted into a frequency-domain signal by Fourier transform to obtain the spectrum of each audio frame. The spectrum is filtered and logarithmically processed based on the Mel frequency cepstral coefficient algorithm to obtain the log-Mel spectrum, and the log-Mel spectrum is subjected to discrete cosine transform. A predetermined number of coefficients are extracted from the transform result as the MFCC coefficients of the audio frame. The MFCC coefficients of all the audio frames are combined in chronological order to form a feature vector sequence, and a fixed-dimensional audio feature vector is generated through statistical analysis of the feature vector sequence.

[0010] Optionally, the step of concatenating and fusing the text feature vector, the image feature vector, and the audio feature vector, and inputting the fused feature vector into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content includes: An early fusion strategy is adopted to directly concatenate the text feature vector, the image feature vector, and the audio feature vector along the feature dimension to obtain a fused feature vector. A classification model consisting of a long short-term memory network and fully connected layers was constructed, and the classification model was trained using a multimodal dataset with sensitive content annotations. The fused feature vector is input into the trained classification model. The long short-term memory network captures the long-term dependencies of the fused feature vector to obtain temporal features. The temporal features are then mapped to the probability space through a fully connected layer to obtain the probability of the existence of various types of sensitive content.

[0011] Optionally, the step of calculating a risk score based on the probability of the presence of the sensitive content, and dynamically setting a warning threshold for different approval scenarios, and automatically triggering a warning when the risk score exceeds the set warning threshold, includes: Based on historical approval data statistical analysis, risk weights are preset for various types of sensitive content. Combined with the probability of each type of sensitive content existing, a risk score is calculated through weighted summation. The function expression for calculating the risk score is as follows:

[0012] In the formula, Indicates the risk score. Indicates the first Risk weighting of sensitive content Indicates the first The probability of sensitive content existing. Index representing sensitive content, Indicates the total number of sensitive contents; Extract approval business scenario features from the preset approval system database and collect historical risk data from the approval system logs. Dynamically set early warning thresholds for different approval scenarios based on the approval business scenario features and the historical risk data. The risk score is compared with the warning threshold for the corresponding approval scenario, and a risk warning is automatically triggered when the risk score is greater than the warning threshold. Based on the triggered warning level, a warning message containing the approval ID, risk type, risk level, and risk details is generated and sent through multiple channels.

[0013] This invention also discloses an early warning system that employs the aforementioned approval risk early warning method based on multimodal sensitive content recognition. The early warning system includes: The data preprocessing module is used to collect text data, image data, and audio data from approval-related sources, and to preprocess the collected data. The text feature extraction module is used to map the words in the preprocessed text data into a high-dimensional word vector sequence, capture the local word order features in the word vector sequence, and aggregate them to generate a text feature vector; The image feature extraction module is used to perform hierarchical feature extraction of low-level and high-level features on the preprocessed image data to obtain a fixed-dimensional image feature vector. The audio feature extraction module is used to perform frame-by-frame processing on the preprocessed audio data and extract the MFCC coefficients of each frame of data based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. The feature processing module is used to concatenate and fuse the text feature vector, the image feature vector, and the audio feature vector, and input the fused feature vector into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content. The early warning module is used to calculate and obtain a risk score based on the probability of the presence of various types of sensitive content, and dynamically set early warning thresholds for different approval scenarios. When the risk score exceeds the set early warning threshold, an early warning is automatically triggered.

[0014] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described approval risk warning method based on multimodal sensitive content identification.

[0015] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned approval risk warning method based on multimodal sensitive content recognition.

[0016] Compared with the prior art, the beneficial effects of the approval risk warning method and system based on multimodal sensitive content identification provided in this invention are as follows: Based on the principle of multimodal sensitive content recognition, this method collects and preprocesses text, image, and audio data from approval-related sources. Words in the text data are mapped to word vector sequences, and local word order features are captured to generate text feature vectors. Image data undergoes low-level and high-level feature extraction to obtain image feature vectors. Audio data is processed by frame segmentation, and MFCC coefficients are extracted based on the Mel-frequency cepstral coefficient algorithm to obtain audio feature vectors. These feature vectors are concatenated and fused to form a fused feature vector, which is input into a pre-trained classification model to output the probability of the presence of sensitive content, thereby calculating a risk score. By dynamically setting warning thresholds and automatically triggering warnings, the complementary information from multiple data sources is fully utilized, effectively overcoming the limitations of single-modal recognition in terms of accuracy and robustness. This comprehensively captures potential risks in text, images, and audio, significantly improving the comprehensiveness and reliability of risk identification in complex approval scenarios. Attached Figure Description

[0017] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic flowchart illustrating the steps of an approval risk warning method based on multimodal sensitive content identification provided in an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] This invention discloses an approval risk warning method based on multimodal sensitive content identification, comprising: S1. Collect text data, image data, and audio data from the relevant approval sources, and preprocess the collected data. S2. Map the words in the preprocessed text data into a high-dimensional word vector sequence, capture the local word order features in the word vector sequence, and aggregate them to generate a text feature vector; S3. Perform hierarchical feature extraction of low-level and high-level features on the preprocessed image data to obtain image feature vectors of fixed dimensions. S4. Perform frame segmentation on the preprocessed audio data, and extract the MFCC coefficients of each frame of data based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. S5. Concatenate and fuse the text feature vector, image feature vector, and audio feature vector, and input the fused feature vector into the pre-trained classification model to obtain the probability of the existence of various types of sensitive content. S6. Calculate risk scores based on the probability of the existence of various sensitive contents, and dynamically set early warning thresholds for different approval scenarios. When the risk score exceeds the set early warning threshold, an early warning will be automatically triggered.

[0020] By implementing the above-described approval risk warning method, based on the principle of multimodal sensitive content recognition, and by collecting and preprocessing text data, image data, and audio data from approval-related sources, the system can effectively solve the problems of incomplete and low-accuracy single-modal approval risk identification. The comprehensive utilization of multimodal data enables the system to capture potential risk information from different dimensions, such as keywords in text data, visual elements in image data, and speech features in audio data, thereby compensating for potential blind spots or noise interference that may exist in single-modal approaches.

[0021] In the feature extraction stage, words in the preprocessed text data are mapped to high-dimensional word vector sequences, and local word order features in the word vector sequences are captured and aggregated to generate text feature vectors. This helps to deeply understand the semantic context and subtle sensitive expressions of the text. Simultaneously, hierarchical feature extraction of low-level and high-level features is performed on the preprocessed image data to obtain fixed-dimensional image feature vectors. Low-level features such as edges and textures can identify basic visual patterns, while high-level features such as object shapes and scene context improve the ability to analyze complex image content. The preprocessed audio data is segmented into frames, and the MFCC coefficients of each frame are extracted based on the Mel-frequency cepstral coefficient algorithm to obtain audio feature vectors. This process effectively captures the spectral characteristics and temporal variations of the audio, enhancing the detection accuracy of sensitive content in speech.

[0022] The extraction of these feature vectors ensures the complementarity of multi-source data. When text data is difficult to identify due to formatting issues or noise, image and audio features can provide auxiliary information, and vice versa, thereby significantly improving the robustness and adaptability of the system.

[0023] Subsequently, the text feature vectors, image feature vectors, and audio feature vectors are concatenated and fused to form a fused feature vector. This early fusion strategy fully utilizes the inherent correlation of multimodal data, avoiding information loss. The fused feature vector is then input into a pre-trained classification model, which uses deep learning technology to output the probability of the presence of various types of sensitive content, thereby achieving comprehensive analysis of complex approval scenarios. Risk scores are then calculated based on the probability of the presence of various types of sensitive content, and warning thresholds are dynamically set according to different approval scenarios. This allows the system to flexibly adapt to diverse needs such as high-risk financial approvals or general administrative affairs. When the risk score exceeds the set warning threshold, an automatic warning is triggered. This mechanism not only improves the real-time performance and accuracy of risk identification but also reduces errors and delays caused by manual intervention through automated processing.

[0024] Therefore, the approval risk warning method of this invention enhances the comprehensiveness and reliability of risk identification through the synergistic effect of multimodal data, enabling the system to maintain stable performance even when faced with data noise, format changes, or complex scenarios. At the same time, the design of dynamic warning thresholds improves the practicality and scalability of the application, providing more solid support for approval decisions.

[0025] Furthermore, text data, image data, and audio data are collected from relevant approval sources, and the collected data are preprocessed, including: Configure a web crawler to collect text data from approval-related websites, databases, and document management systems. Convert documents of different formats in the text data into plain text and remove HTML tags and special characters from the converted plain text using regular expression rules. Image data, including enterprise qualification certificates, engineering construction site photos and design drawings, is acquired by image sensors or obtained from a pre-established image database. The image data is then filtered, and the contrast of the filtered image data is enhanced based on histogram equalization. Audio data, including meeting approval and discussion records and customer voice complaints, is recorded using a microphone array or extracted from a pre-established audio repository. The audio data is then standardized in terms of sampling rate and noise is removed from the standardized audio data.

[0026] By implementing the above-described approval risk warning method, text, image, and audio data are collected from approval-related sources and systematically preprocessed, significantly improving the data quality and processing efficiency of the multimodal risk identification system. Specifically, configuring web crawlers (such as Scrapy) to collect text data from approval-related websites (such as government approval service platforms, enterprise internal approval system websites, etc.), databases (such as MySQL, Oracle, etc.), and document management systems (such as SharePoint) ensures the comprehensiveness and real-time nature of the data sources. Furthermore, converting documents of different formats in the text data into plain text using a format conversion tool (such as pandoc) and removing HTML tags and special characters using regular expressions effectively eliminates data format heterogeneity and noise interference. For example, converting a .docx contract document into plain text format facilitates subsequent text cleaning and feature extraction.

[0027] In image data processing, image data such as enterprise qualification certificates, construction site photos, and design drawings are acquired through image sensors or obtained from a pre-established image database. Noise is removed using filtering (e.g., Gaussian filtering), and contrast enhancement is performed based on histogram equalization. This process not only improves the visual clarity of the image but also enhances the saliency of low-level features such as edges and textures, creating favorable conditions for the hierarchical extraction of low- and high-level features by convolutional neural networks. For example, histogram equalization expands the dynamic range of the image by redistributing pixel intensity values, making it easier to detect text details in qualification certificates or safety hazards at construction sites. Preferably, the images after noise removal and contrast enhancement can also be normalized to a standard size (e.g., 224×224 pixels) set by the system for subsequent input into the image feature extraction model. Image transformation functions (such as the cv2.resize function in OpenCV) are used to adjust the image size.

[0028] For example, in a business qualification approval scenario, images of business licenses, production permits, and other certificates are captured by connecting to an image acquisition device (such as a high-definition camera). In a construction project approval scenario, photos of the construction site are obtained from surveillance cameras at the project site, or design drawings are downloaded from a design drawing management system. For the acquired images, functions from an image editing library (such as OpenCV) are used for noise removal and contrast enhancement. For instance, the Gaussian filter function `cv2.GaussianBlur` is used to remove noise from construction site photos, and the histogram equalization function `cv2.equalizeHist` is used to enhance image contrast. Regarding audio data, meeting approval and discussion records and customer voice complaints are extracted from microphone array recordings or pre-established audio repositories. The audio data undergoes sampling rate standardization (e.g., uniform to 16kHz) and noise removal processing (e.g., spectral subtraction) to ensure the temporal consistency and signal-to-noise ratio optimization of the audio signal. This enables the Mel frequency cepstral coefficient algorithm to extract the MFCC feature vector of each frame of audio more stably, avoiding feature distortion caused by environmental noise or sampling differences.

[0029] For example, in a meeting approval scenario, professional recording equipment (such as a digital microphone array) is used to record the meeting discussion process. In a customer complaint handling approval scenario, audio files of customer voice complaints are obtained from the customer service system. The collected audio data is then processed using functions in an audio processing library (such as Librosa) to perform sample rate conversion and frequency domain transformation. For instance, the `librosa.resample` function is used to convert the audio sample rate from 44100Hz to the system-set 16000Hz, and the `librosa.stft` function is used to convert the audio's time-domain signal to its frequency-domain signal.

[0030] Overall, these preprocessing steps significantly reduce the risk of error propagation in subsequent feature extraction and fusion by cleaning and enhancing multimodal data early on. This not only improves the representational capabilities of text, image, and audio feature vectors but also enhances the system's adaptability to complex approval scenarios through overall data quality optimization. For example, in contract approval, clean text data can accurately parse the semantics of clauses, enhanced image data can clearly identify the authenticity of seals, and denoised audio data can reliably verify voice confirmation content. This provides highly consistent, low-noise input for the multimodal fusion and sensitive content recognition modules, laying a solid foundation for improving the accuracy of risk identification.

[0031] Furthermore, the words in the preprocessed text data are mapped to a high-dimensional word vector sequence, and local word order features in the word vector sequence are captured and aggregated to generate a text feature vector, including: Based on word embedding operations, each word in the preprocessed text data is mapped to a high-dimensional word vector, and all word vectors are combined to generate a word vector sequence; Using a pre-trained convolutional neural network with multiple convolutional kernels of different sizes, a sliding window convolution operation is performed on the word vector sequence to extract several local word order features; Based on the extracted local word order features, the contextual relationships between words are captured using an attention mechanism, and a corresponding weight is generated for each word. Based on the generated weights, the word vectors of all words are weighted and aggregated to generate a text feature vector. The function expression for aggregating and generating the text feature vector is as follows:

[0032] In the formula, This represents the text feature vector generated by aggregation. Indicates the first The weight of each word, Indicates the first Word vectors of 1 word, Indicating the index of words, This indicates the total number of words.

[0033] Through the implementation of the above embodiments of the approval risk warning method, the words in the preprocessed text data are mapped into a high-dimensional word vector sequence, and the local word order features in the word vector sequence are captured. Finally, a text feature vector is aggregated and generated, significantly enhancing the feature representation ability of the text modality in the approval risk warning system. Among them, each word is mapped into a high-dimensional word vector based on the word embedding operation, and a word vector sequence is generated, enabling the semantic information of the text to be efficiently encoded in a continuous vector space. For example, using the Word2Vec model can transform the semantic relationship between words into a vector distance, laying a foundation for subsequent feature extraction.

[0034] Use a pre-trained convolutional neural network to apply multiple convolutional kernels of different sizes to perform a sliding window convolution operation on the word vector sequence to extract several local word order features. This process captures the local patterns at the phrase level and syntactic level in the text through convolutional kernels of different scales (such as kernels with sizes of 3, 4, and 5), effectively identifying key information such as sensitive word combinations or specific expression structures. Further, based on the extracted local word order features, the context relationship between words is captured based on the attention mechanism, and a corresponding weight is generated for each word. This enables the model to dynamically focus on the words highly relevant to sensitive content (such as key terms or risk warning phrases in contract approval), reducing the interference of irrelevant words. Finally, according to the generated weights, the word vectors of all words are weighted and aggregated to generate a text feature vector. This weighted aggregation method ensures that the text feature vector not only retains the details of the local word order features but also incorporates the importance distribution of the global context, thereby generating a more discriminative representation.

[0035] Overall, this method fully excavates the deep sensitive information in the text data through semantic encoding of word embedding, local feature extraction of convolutional neural network, context weighting of attention mechanism, and vector generation of weighted aggregation. It not only improves the feature quality of the text modality itself but also enhances the risk recognition accuracy and robustness of the system for complex approval scenarios (such as contract text analysis) through multi-modal complementarity.

[0036] As described above, preferably, use a regular expression library (such as re) to perform word segmentation on the cleaned text data, splitting the text into individual words or phrases. For example, for the text sentence "This project involves sensitive information", the word segmentation result is ["This", "project", "involves", "sensitive", "information"]. Then, use a stop word list (such as the English stop word list in the NLTK library or a custom Chinese stop word list) to remove the stop words in the text (such as meaningless words like "of", "is", "in", etc.), and HTML tags can be accurately removed, making the text content standardized and further streamlining the text data, providing a clean and consistent data basis for subsequent word vector mapping and local word order feature capture.

[0037] If a custom word embedding model is used (such as word embedding training based on the Skip-Gram model), a large amount of approval-related text data is used for training. During training, an appropriate window size (e.g., 5) is set, considering the context information of the five words before and after the current word. By continuously adjusting the model parameters (such as the dimension of the word vectors, the learning rate, etc.), the model can better learn the semantic relationships between words. For example, after multiple experiments, setting the dimension of the word vectors to 100 and the learning rate to 0.01, a relatively stable word embedding model is obtained after training for [X] rounds. Furthermore, hierarchical feature extraction of low-level and high-level features is performed on the preprocessed image data to obtain a fixed-dimensional image feature vector, including: Add a first convolutional layer group composed of multiple convolutional kernels, input the preprocessed image data into the first convolutional layer group, perform feature extraction through convolution operation to obtain a low-level feature map containing edge and texture information; Add a second convolutional layer group consisting of multiple convolutional kernels and having more convolutional layers than the first convolutional layer group. Input the low-level feature map into the second convolutional layer group, and perform feature abstraction by increasing the number of feature channels to obtain an intermediate feature map containing local shape and combined features. A third convolutional layer group is added, consisting of multiple convolutional kernels and having more convolutional layers than the second convolutional layer group. The intermediate feature map is input into the third convolutional layer group, and deep feature extraction is performed through multi-layer nonlinear transformation to obtain a high-level feature map that represents the semantic content of the image. Pooling layers are added after the first, second, and third convolutional layers to spatially downsample the corresponding feature maps and flatten the downsampled high-level feature maps into one-dimensional vectors. The flattened one-dimensional vector feature map is mapped to a fixed-dimensional image feature vector through a fully connected layer.

[0038] Through the implementation of the above-described approval risk warning method, hierarchical feature extraction of low-level and high-level features is performed on the preprocessed image data to obtain fixed-dimensional image feature vectors, significantly improving the feature representation and discriminative power of image modalities in the approval risk warning system. Specifically, by adding a first convolutional layer group composed of multiple convolutional kernels to perform convolution operations on the input image, low-level feature maps containing edge and texture information are extracted, providing basic details for feature abstraction. Further, a second convolutional layer group with a larger number of convolutional layers is used to increase the number of feature channels, and feature abstraction is performed to obtain intermediate feature maps containing local shape and combined features, enhancing the ability to capture the local structure of the image. Finally, a third convolutional layer group with the largest number of convolutional layers performs multi-layer nonlinear transformations to extract high-level feature maps representing the semantic content of the image, enabling the features to reflect the overall semantic information of the image.

[0039] Adding pooling layers after each convolutional layer group spatially downsamples the feature map, which not only reduces computational complexity but also enhances the translation invariance of the features. Finally, the downsampled high-level feature map is flattened into a one-dimensional vector and mapped to a fixed-dimensional image feature vector through a fully connected layer, ensuring the uniformity and fusionability of the feature dimensions. This hierarchical feature extraction method, through alternating convolutional and pooling layer operations, gradually abstracts features from low to high levels, enabling the image feature vector to effectively represent key information in approval-related images (such as enterprise qualification certificates and construction site photos).

[0040] As described above, preferably, before feature extraction from image data, the images that have undergone noise removal and contrast enhancement can be normalized to a uniform size (e.g., 224×224 pixels) set by the system, so that they can be input into the image feature extraction model later. Image size adjustment is achieved using an image transformation function (e.g., the cv2.resize function in OpenCV). Furthermore, a CNN-based image feature extraction model (e.g., the VGG16 model) is preferred. If a pre-trained model is used, fine-tuning can be performed on an approval-related image dataset. The image dataset is divided into a training set, a validation set, and a test set, with proportions of 70%, 15%, and 15%, respectively. During fine-tuning, the first few convolutional layers of the model (e.g., the first 10 layers) are frozen, and only the parameters of the subsequent convolutional and fully connected layers are updated. An appropriate learning rate (e.g., 1e - 5) and number of iterations (e.g., 50 times) are set, and the model parameters are continuously optimized through backpropagation, enabling the model to better extract features from approval-related images.

[0041] Furthermore, the preprocessed audio data is segmented into frames, and the MFCC coefficients of each frame are extracted using the Mel-frequency cepstral coefficient algorithm to obtain the audio feature vector, including: The preprocessed audio data is divided into frames according to a preset frame length and frame shift to obtain multiple audio frames. Windowing is applied to each audio frame after frame segmentation, and the time-domain signal of the windowed audio frame is converted into the frequency-domain signal through Fourier transform to obtain the spectrum of each audio frame. The spectrum is filtered and logarithmically processed based on the Mel frequency cepstral coefficient algorithm to obtain the log-Mel spectrum, and then the log-Mel spectrum is subjected to discrete cosine transform. A predetermined number of coefficients are extracted from the transform result as the MFCC coefficients of the audio frame. The MFCC coefficients of all audio frames are combined in chronological order to form a feature vector sequence, and a fixed-dimensional audio feature vector is generated through statistical analysis of the feature vector sequence.

[0042] Through the implementation of the above-described approval risk warning method, the preprocessed audio data is segmented into frames, and the MFCC coefficients of each frame are extracted based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. This significantly improves the feature representation capability and robustness of audio modes in the approval risk warning system. Frame segmentation divides the audio data into multiple short-term stationary audio frames by setting the frame length and frame shift, effectively capturing the temporal dynamic changes of the audio signal and providing a stable foundation for subsequent feature extraction. Windowing (such as using the Hanning window or Hamming window function) reduces spectral leakage, ensuring a clearer spectral representation when the Fourier transform converts the time-domain signal to the frequency-domain signal, thus enhancing frequency resolution accuracy. Based on the Mel frequency cepstral coefficient algorithm, the spectrum is filtered by a Mel filter bank (e.g., using a set of triangular filters covering the frequency band sensitive to the human ear) and logarithmic processing is performed to obtain the log-Mel spectrum, simulating the nonlinear perception characteristics of the human ear, highlighting key frequency band information in the audio, and then the spectral envelope is compressed by discrete cosine transform to extract a predetermined number of MFCC coefficients (e.g., preferentially selecting the first 12-13 coefficients to represent the vocal tract features), effectively reducing data dimensionality and retaining discriminative features.

[0043] The MFCC coefficients of all audio frames are combined in chronological order to form a feature vector sequence. Statistical analysis (such as calculating the mean and variance) is then used to generate fixed-dimensional audio feature vectors, ensuring feature consistency and facilitating multimodal integration. For example, in analyzing meeting approval and discussion records and customer voice complaint content, audio features can effectively capture sensitive intonations or keywords in speech, cross-validating with text and image modalities, thereby comprehensively improving the system's adaptability to complex approval scenarios and the accuracy of risk identification.

[0044] As described above, preferably, the audio data converted to a frequency domain signal is segmented into frames, with each frame having a certain duration (e.g., 25ms) and a certain overlap between frames (e.g., 10ms). Then, feature extraction is performed on each frame of audio data to obtain the corresponding MFCC coefficients. Functions in an audio processing library (e.g., Librosa) are used for audio framing and MFCC coefficient extraction. For example, the librosa.feature.mfcc function is used to calculate the number of MFCC coefficients in each frame of audio.

[0045] For the MFCC algorithm, its parameters generally do not need to be trained, but some parameters can be adjusted and optimized according to the characteristics of the approved audio data. For example, the number of Mel filter banks can be adjusted (e.g., from the default 26 to 32) to better adapt to the frequency distribution characteristics of the approved audio data, thereby improving the representativeness of the MFCC coefficients.

[0046] Furthermore, the text feature vector, image feature vector, and audio feature vector are concatenated and fused, and the fused feature vector is input into a pre-trained classification model to obtain the probability of the presence of various types of sensitive content, including: An early fusion strategy is adopted to directly concatenate the text feature vector, image feature vector and audio feature vector along the feature dimension to obtain the fused feature vector; A classification model consisting of a long short-term memory network and fully connected layers was constructed, and the classification model was trained using a multimodal dataset with sensitive content annotations. The fused feature vectors are input into the trained classification model. The long short-term memory network captures the long-term dependencies of the fused feature vectors to obtain temporal features. The temporal features are then mapped to the probability space through a fully connected layer to obtain the probability of the existence of various types of sensitive content.

[0047] Through the implementation of the above-described approval risk warning method, text feature vectors, image feature vectors, and audio feature vectors are concatenated and fused. The fused feature vector is then input into a pre-trained classification model to obtain the probability of the presence of various sensitive contents, achieving efficient integration and accurate analysis of multimodal data. Specifically, the early fusion strategy directly concatenates multimodal feature vectors along the feature dimensions, fully utilizing the complementary information of text, image, and audio data and avoiding information loss. For example, in a contract approval scenario, text feature vectors (generated by aggregating local word order features extracted using word embedding technology and convolutional neural networks) can capture the semantics of the clauses; image feature vectors (based on low-level and high-level features extracted hierarchically from a convolutional neural network architecture) can identify the authenticity of seals; and audio feature vectors (generated by statistical analysis of MFCC coefficient sequences extracted using the Mel-frequency cepstral coefficient algorithm) can verify speech content. The fusion of these three features forms a more discriminative fused feature vector.

[0048] A classification model consisting of a Long Short-Term Memory (LSTM) network and fully connected layers is constructed and trained using a multimodal dataset labeled with sensitive content (e.g., using labeled data such as politically sensitive data and data related to leaked trade secrets to optimize model parameters), enabling the model to learn multimodal association patterns. After inputting the fused feature vectors into the trained classification model, the LTM network effectively captures the long-term dependencies between multimodal features (e.g., the temporal association between text descriptions and image content), outputting temporal features. These features are then mapped to a probability space through fully connected layers to obtain the probability of the presence of various types of sensitive content, thus providing accurate input for the risk assessment unit. This fusion and classification mechanism significantly improves the comprehensiveness and accuracy of sensitive content identification.

[0049] As mentioned above, the fused feature vector is represented as: , Represents the text feature vector. Represents the image feature vector. This represents the audio feature vector. The dimension of the fused feature vector is represented as: , This represents the dimension of the text feature vector. The dimension of the image feature vector. This represents the dimension of the audio feature vector.

[0050] Preferably, the server configuration used to implement this method should include a multi-core CPU, large-capacity memory and a high-performance GPU, with Linux being the preferred operating system, and TensorFlow or PyTorch deep learning framework, as well as data processing libraries such as NumPy, Pandas, and OpenCV installed.

[0051] During model training, the multimodal dataset labeled with sensitive content is divided into training, validation, and test sets in a ratio of 70%, 15%, and 15%, respectively. The classification model is trained using the backpropagation algorithm, and Dropout and L2 regularization are applied to optimize the model, prevent overfitting, and improve its generalization ability. Those skilled in the art will understand that the above hardware and software configurations and model training methods are merely examples and are not intended to limit the scope of this invention.

[0052] Furthermore, a risk score is calculated based on the probability of sensitive content being present, and warning thresholds are dynamically set for different approval scenarios. When the risk score exceeds the set warning threshold, an warning is automatically triggered, including: Based on historical approval data statistical analysis, risk weights are preset for various types of sensitive content. Combined with the probability of each type of sensitive content existing, a risk score is calculated through weighted summation. The function expression for calculating the risk score is as follows:

[0053] In the formula, Indicates the risk score. Indicates the first Risk weighting of sensitive content Indicates the first The probability of sensitive content existing. Index representing sensitive content, Indicates the total number of sensitive contents; Extract approval business scenario characteristics from the pre-set approval system database and collect historical risk data from the approval system logs. Dynamically set early warning thresholds for different approval scenarios based on the approval business scenario characteristics and historical risk data. The risk score is compared with the warning threshold for the corresponding approval scenario, and a risk warning is automatically triggered when the risk score is greater than the warning threshold. Based on the triggered warning level, a warning message containing the approval ID, risk type, risk level, and risk details is generated and sent through multiple channels.

[0054] Through the implementation of the above-described approval risk warning method, a risk score is calculated based on the probability of the presence of sensitive content, and a warning threshold is dynamically set for different approval scenarios. When the risk score exceeds the set warning threshold, an automatic warning is triggered, achieving precision and adaptability in approval risk warning. Specifically, risk weights for various types of sensitive content are preset based on statistical analysis of historical approval data, and the risk score is calculated by weighted summation combining the probability of the presence of each type of sensitive content. The risk weights can be dynamically adjusted according to the type of sensitive content (such as political sensitivity or commercial secrets). For example, in contract approval scenarios, a higher weight is given to the disclosure of commercial terms, enabling the risk score to objectively reflect the overall risk level of the multimodal recognition results.

[0055] The system extracts approval business scenario characteristics (such as approval type and amount) from a pre-set approval system database and collects historical risk data (such as risk occurrence frequency) from the approval system logs. Based on this data, it dynamically sets early warning thresholds for different approval scenarios (such as high-risk financial approvals or low-risk administrative affairs), ensuring that the thresholds match the actual business situation and improving the targeting of early warnings. The system compares the risk score with the corresponding early warning threshold for the approval scenario. When the risk score exceeds the early warning threshold, a risk warning is automatically triggered, enabling real-time response. Based on the triggered warning level, an early warning message containing the approval ID, risk type, risk level, and risk details is generated and sent through multiple channels (such as email, SMS, or system notifications) to enhance the timeliness and coverage of the warnings. This mechanism is closely integrated with the multimodal fusion and sensitive content identification module. Text feature vectors (extracted through word embedding and CNN), image feature vectors (extracted through hierarchical CNN), and audio feature vectors (extracted through MFCC) are fused and input into a classification model to obtain the probability of sensitive content presence. These probabilities are used to calculate a risk score. Combined with dynamic threshold settings, this allows the system to not only comprehensively utilize complementary information from multimodal data (such as cross-validation of text, images, and audio) but also adapt to diverse approval scenarios. Ultimately, the approval risk warning module achieves efficient and accurate risk warnings, improving the overall system's reliability and usability. Preferably, a visual interface can also be provided for approvers to view risk details and process them, recording processing operations and results. Approval operations include further verifying information, requesting supplementary materials from the applicant, or directly rejecting the approval.

[0056] This invention also discloses an early warning system that employs the aforementioned approval risk early warning method based on multimodal sensitive content recognition. The early warning system includes: The data preprocessing module is used to collect text data, image data, and audio data from approval-related sources, and to preprocess the collected data. The text feature extraction module is used to map words in the preprocessed text data into a high-dimensional word vector sequence, capture local word order features in the word vector sequence, and aggregate them to generate text feature vectors. The image feature extraction module is used to perform hierarchical feature extraction of low-level and high-level features on the preprocessed image data to obtain image feature vectors of fixed dimensions. The audio feature extraction module is used to perform frame-by-frame processing on the preprocessed audio data and extract the MFCC coefficients of each frame of data based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. The feature fusion processing module is used to concatenate and fuse text feature vectors, image feature vectors, and audio feature vectors, and input the fused feature vectors into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content. The early warning module is used to calculate risk scores based on the probability of the presence of various sensitive contents, and dynamically set early warning thresholds for different approval scenarios. When the risk score exceeds the set early warning threshold, an early warning is automatically triggered.

[0057] The present invention also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described approval risk warning method based on multimodal sensitive content identification.

[0058] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned approval risk warning method based on multimodal sensitive content recognition.

[0059] This invention is described based on flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to specific embodiments. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the flowcharts and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Those skilled in the art can modify the technical solutions described in the above embodiments, or make equivalent substitutions for some of the technical features; and all such modifications and substitutions should fall within the protection scope of the present invention.

Claims

1. A method for early warning of approval risks based on multimodal sensitive content identification, characterized in that, The aforementioned approval risk warning method includes: Text data, image data, and audio data are collected from relevant approval sources, and the collected data are preprocessed. The words in the preprocessed text data are mapped to a high-dimensional word vector sequence, and the local word order features in the word vector sequence are captured and aggregated to generate a text feature vector; Hierarchical feature extraction of low-level and high-level features is performed on the preprocessed image data to obtain a fixed-dimensional image feature vector; The preprocessed audio data is divided into frames, and the MFCC coefficients of each frame are extracted based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. The text feature vector, the image feature vector, and the audio feature vector are concatenated and fused, and the fused feature vector is input into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content. Risk scores are calculated based on the probability of the presence of various types of sensitive content, and warning thresholds are dynamically set for different approval scenarios. When the risk score exceeds the set warning threshold, a warning is automatically triggered.

2. The approval risk warning method based on multimodal sensitive content identification according to claim 1, characterized in that, The process involves collecting text data, image data, and audio data from approval-related sources, and preprocessing the collected data, including: Configure a web crawler to collect the text data from approval-related websites, databases, and document management systems; convert documents of different formats in the text data into plain text; and remove HTML tags and special characters from the converted plain text using regular expression rules. Image data, including enterprise qualification certificates, engineering construction site photos and design drawings, is acquired by an image sensor or obtained from a pre-established image database. The image data is then filtered, and the filtered image data is further enhanced based on histogram equalization. The audio data, which includes meeting approval and discussion records and customer voice complaints, is recorded using a microphone array or extracted from a pre-established audio repository. The audio data is then subjected to sampling rate standardization and noise removal processing.

3. The approval risk early warning method based on multimodal sensitive content identification according to claim 1, characterized in that, The process of mapping words in the preprocessed text data into a high-dimensional word vector sequence, capturing local word order features in the word vector sequence, and aggregating them to generate a text feature vector includes: Based on word embedding operations, each word in the preprocessed text data is mapped to a high-dimensional word vector, and all the word vectors are combined to generate the word vector sequence; Using a pre-trained convolutional neural network with multiple convolutional kernels of different sizes, a sliding window convolution operation is performed on the word vector sequence to extract several local word order features; Based on the extracted local word order features, the contextual relationships between the words are captured using an attention mechanism, and a corresponding weight is generated for each word. Based on the generated weights, the word vectors of all the words are weighted and aggregated to generate the text feature vector. The function expression for aggregating and generating the text feature vector is as follows: In the formula, This represents the text feature vector generated by aggregation. Indicates the first The weight of each word, Indicates the first Word vectors of 1 word, Indicating the index of words, Indicates the total number of words.

4. The approval risk early warning method based on multimodal sensitive content identification according to claim 1, characterized in that, The step of performing hierarchical feature extraction on the preprocessed image data, including low-level and high-level features, to obtain a fixed-dimensional image feature vector includes: A first convolutional layer group consisting of multiple convolutional kernels is added. The preprocessed image data is input into the first convolutional layer group. Feature extraction is performed through convolution operations to obtain a low-level feature map containing edge and texture information. A second convolutional layer group is added, consisting of multiple convolutional kernels and having more convolutional layers than the first convolutional layer group. The low-level feature map is input into the second convolutional layer group, and feature abstraction is performed by increasing the number of feature channels to obtain an intermediate feature map containing local shape and combined features. A third convolutional layer group is added, consisting of multiple convolutional kernels and having more convolutional layers than the second convolutional layer group. The intermediate feature map is input into the third convolutional layer group, and deep feature extraction is performed through multi-layer nonlinear transformation to obtain a high-level feature map representing the semantic content of the image. Pooling layers are added after the first convolutional layer group, the second convolutional layer group, and the third convolutional layer group to perform spatial downsampling on the corresponding feature maps and flatten the downsampled high-level feature maps into one-dimensional vectors. The flattened one-dimensional vector feature map is mapped to the image feature vector of fixed dimension through a fully connected layer.

5. The approval risk early warning method based on multimodal sensitive content identification according to claim 1, characterized in that, The process of segmenting the preprocessed audio data into frames and extracting the MFCC coefficients of each frame based on the Mel-frequency cepstral coefficient algorithm to obtain the audio feature vector includes: The preprocessed audio data is divided into frames according to a preset frame length and frame shift to obtain multiple audio frames. Windowing is applied to each audio frame after frame segmentation, and the time-domain signal of the windowed audio frame is converted into a frequency-domain signal by Fourier transform to obtain the spectrum of each audio frame. The spectrum is filtered and logarithmically processed based on the Mel frequency cepstral coefficient algorithm to obtain the log-Mel spectrum, and the log-Mel spectrum is subjected to discrete cosine transform. A predetermined number of coefficients are extracted from the transform result as the MFCC coefficients of the audio frame. The MFCC coefficients of all the audio frames are combined in chronological order to form a feature vector sequence, and a fixed-dimensional audio feature vector is generated through statistical analysis of the feature vector sequence.

6. The approval risk early warning method based on multimodal sensitive content identification according to claim 1, characterized in that, The step of concatenating and fusing the text feature vector, the image feature vector, and the audio feature vector, and inputting the fused feature vector into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content includes: An early fusion strategy is adopted to directly concatenate the text feature vector, the image feature vector, and the audio feature vector along the feature dimension to obtain a fused feature vector. A classification model consisting of a long short-term memory network and fully connected layers was constructed, and the classification model was trained using a multimodal dataset with sensitive content annotations. The fused feature vector is input into the trained classification model. The long short-term memory network captures the long-term dependencies of the fused feature vector to obtain temporal features. The temporal features are then mapped to the probability space through a fully connected layer to obtain the probability of the existence of various types of sensitive content.

7. The approval risk warning method based on multimodal sensitive content identification according to claim 1, characterized in that, The process involves calculating a risk score based on the probability of the presence of the sensitive content, dynamically setting warning thresholds for different approval scenarios, and automatically triggering a warning when the risk score exceeds the set warning threshold. This includes: Based on historical approval data statistical analysis, risk weights are preset for various types of sensitive content. Combined with the probability of each type of sensitive content existing, a risk score is calculated through weighted summation. The function expression for calculating the risk score is as follows: In the formula, Indicates the risk score. Indicates the first Risk weighting of sensitive content Indicates the first The probability of sensitive content existing. Index representing sensitive content, Indicates the total number of sensitive contents; Extract approval business scenario features from the preset approval system database and collect historical risk data from the approval system logs. Dynamically set early warning thresholds for different approval scenarios based on the approval business scenario features and the historical risk data. The risk score is compared with the warning threshold for the corresponding approval scenario, and a risk warning is automatically triggered when the risk score is greater than the warning threshold. Based on the triggered warning level, a warning message containing the approval ID, risk type, risk level, and risk details is generated and sent through multiple channels.

8. An early warning system, employing the approval risk early warning method based on multimodal sensitive content recognition as described in any one of claims 1-7, characterized in that, The early warning system includes: The data preprocessing module is used to collect text data, image data, and audio data from approval-related sources, and to preprocess the collected data. The text feature extraction module is used to map the words in the preprocessed text data into a high-dimensional word vector sequence, capture the local word order features in the word vector sequence, and aggregate them to generate a text feature vector; The image feature extraction module is used to perform hierarchical feature extraction of low-level and high-level features on the preprocessed image data to obtain a fixed-dimensional image feature vector. The audio feature extraction module is used to perform frame-by-frame processing on the preprocessed audio data and extract the MFCC coefficients of each frame of data based on the Mel frequency cepstral coefficient algorithm to obtain the audio feature vector. The feature processing module is used to concatenate and fuse the text feature vector, the image feature vector, and the audio feature vector, and input the fused feature vector into a pre-trained classification model to obtain the probability of the existence of various types of sensitive content. The early warning module is used to calculate and obtain a risk score based on the probability of the presence of various types of sensitive content, and dynamically set early warning thresholds for different approval scenarios. When the risk score exceeds the set early warning threshold, an early warning is automatically triggered.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the approval risk warning method based on multimodal sensitive content recognition as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the approval risk warning method based on multimodal sensitive content recognition as described in any one of claims 1-7.

Images