Abnormity identification method, device and equipment for multi-modal data and storage medium
By preprocessing and fusion of multimodal data, combined with the recognition methods of support vector machines and convolutional neural networks, the problem that traditional methods are difficult to deal with multiple types of data is solved, and efficient exception recognition and accuracy are achieved.
Patent Information
- Application Number
- CN202411382097.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2024-12-31
AI Technical Summary
Traditional abnormal data recognition methods are difficult to process multiple types of data at the same time and cannot meet actual needs.
A method of abnormal identification of multimodal data is proposed. By obtaining multimodal data, data preprocessing is generated, and abnormal identification is performed using support vector machines and convolutional neural networks, the abnormal identification results are finally determined and based on the recognition. As a result, early warning measures will be implemented.
It improves the accuracy and processing speed of abnormal identification of multimodal data, and can effectively identify abnormal situations in various types of data.
Smart Images
Figure CN119226981A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of financial technology, and in particular to a method, device, equipment and storage medium for identifying anomalies in multimodal data. Background Art
[0002] With the development of computer technology, the amount of data used in various industries has increased greatly. These data often contain multiple forms, such as text, images, voice, etc. There may be various abnormalities in these data, such as noise, missing, errors, etc. Traditional abnormal data recognition methods can usually only process one type of data, which is difficult to meet actual needs. Therefore, there is a need for an abnormal data recognition system and method that can simultaneously process multiple types of data. Summary of the invention
[0003] The purpose of the embodiments of the present application is to provide a method, apparatus, computer device and storage medium for anomaly identification of multimodal data, the main purpose of which is to improve the accuracy of anomaly identification of multimodal data.
[0004] In order to solve the above technical problems, the present application provides a method for identifying anomalies in multimodal data, which adopts the following technical solutions:
[0005] Acquire multimodal data;
[0006] Performing data preprocessing on the multimodal data to obtain basic data corresponding to each modality;
[0007] Performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data;
[0008] Abnormal data is identified based on the fusion vector to obtain an abnormal identification result.
[0009] Furthermore, the multimodal data includes text data, image data and voice data, and the data preprocessing is performed on the multimodal data to obtain basic data corresponding to each modality, including:
[0010] Performing word segmentation and vectorization on the text data to obtain basic data corresponding to the text data;
[0011] Performing feature extraction and dimensionality reduction on the image data to obtain basic data corresponding to the image data;
[0012] Mel-frequency cepstral coefficient features are extracted from the speech data to obtain basic data corresponding to the speech data.
[0013] Furthermore, the step of performing abnormal data identification based on the fusion vector to obtain an abnormal identification result specifically includes:
[0014] Aggregate the fused vector to obtain N data clusters, where N is a category of the data cluster and is a positive integer;
[0015] Perform statistical analysis on each data cluster to obtain analysis results;
[0016] A classifier is used to identify abnormal data on the analysis result to obtain an abnormal identification result.
[0017] Furthermore, the step of using a classifier to identify abnormal data on the analysis result to obtain an abnormal identification result specifically includes:
[0018] Using a support vector machine and a convolutional neural network to perform abnormality recognition on the analysis results, respectively, to obtain a first recognition result and a second recognition result;
[0019] Based on the first recognition result and the second recognition result, an abnormal recognition result is determined.
[0020] Furthermore, the step of determining an abnormal recognition result based on the first recognition result and the second recognition result specifically includes:
[0021] If there is abnormal vector data in the first recognition result and the second recognition result, obtaining multimodal data corresponding to the abnormal vector data;
[0022] An abnormality recognition result is generated according to the multimodal data corresponding to the multiple abnormality vector data.
[0023] Furthermore, after the step of using a classifier to identify abnormal data on the analysis result to obtain an abnormal identification result, the method further includes:
[0024] Determining a target abnormality level according to the abnormality identification result;
[0025] Obtain the correspondence between the preset warning level and the warning measure, and execute the warning measure corresponding to the target abnormal level to perform warning processing.
[0026] In order to solve the above technical problems, the embodiment of the present application further provides a device for identifying anomalies in multimodal data, which adopts the following technical solution:
[0027] An acquisition module, used to acquire multimodal data;
[0028] A preprocessing module, used to perform data preprocessing on the multimodal data to obtain basic data corresponding to each modality;
[0029] A fusion module, used for performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data;
[0030] The recognition module is used to identify abnormal data based on the fusion vector to obtain an abnormal recognition result.
[0031] Furthermore, the preprocessing module includes:
[0032] A first processing unit, configured to perform word segmentation and vectorization on the text data to obtain basic data corresponding to the text data;
[0033] A second processing unit is used to perform feature extraction and dimension reduction on the image data to obtain basic data corresponding to the image data;
[0034] The third processing unit is used to extract Mel-frequency cepstral coefficient features from the speech data to obtain basic data corresponding to the speech data.
[0035] Furthermore, the identification module includes:
[0036] an aggregation unit, configured to perform aggregation processing on the fused vector to obtain N data clusters, where N is a category of the data cluster and is a positive integer;
[0037] An analysis unit, used for performing statistical analysis on each data cluster to obtain analysis results;
[0038] The identification unit is used to use a classifier to identify abnormal data on the analysis result to obtain an abnormal identification result.
[0039] Furthermore, the identification unit includes:
[0040] An abnormality identification subunit is used to identify abnormalities on the analysis results using a support vector machine and a convolutional neural network respectively, to obtain a first identification result and a second identification result;
[0041] The result determination subunit is used to determine an abnormal recognition result based on the first recognition result and the second recognition result.
[0042] Furthermore, the result determination subunit includes:
[0043] A discriminating component, configured to obtain multimodal data corresponding to abnormal vector data if abnormal vector data exists in the first recognition result and the second recognition result;
[0044] A generating component is used to generate an anomaly recognition result according to the multi-modal data corresponding to the multiple anomaly vector data.
[0045] Furthermore, the multimodal data anomaly identification device comprises:
[0046] A grading module, used to determine a target abnormality level according to the abnormality identification result;
[0047] The early warning module is used to obtain the corresponding relationship between the preset early warning level and the early warning measure, and execute the early warning measure corresponding to the target abnormal level to perform early warning processing.
[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0049] Acquire multimodal data;
[0050] Performing data preprocessing on the multimodal data to obtain basic data corresponding to each modality;
[0051] Performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data;
[0052] Abnormal data is identified based on the fusion vector to obtain an abnormal identification result.
[0053] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0054] Acquire multimodal data;
[0055] Performing data preprocessing on the multimodal data to obtain basic data corresponding to each modality;
[0056] Performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data;
[0057] Abnormal data is identified based on the fusion vector to obtain an abnormal identification result.
[0058] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0059] By processing and fusing data from different modalities, and then performing anomaly identification on the fused data, the processing speed and accuracy of anomaly identification are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0061] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0062] Figure 2 A flowchart of an embodiment of a method for identifying abnormalities in multimodal data according to the present application;
[0063] Figure 3 is a structural schematic diagram of an embodiment of a device for identifying abnormalities in multimodal data according to the present application;
[0064] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0066] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0068] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0069] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0070] The terminal devices 101, 102, and 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.
[0071] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .
[0072] It should be noted that the method for identifying anomalies in multimodal data provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the device for identifying anomalies in multimodal data is generally arranged in a server / terminal device.
[0073] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.
[0074] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for identifying anomalies in multimodal data according to the present application. The method for identifying anomalies in multimodal data comprises the following steps:
[0075] Step S201, acquiring multimodal data.
[0076] Among them, multimodal data refers to data or information that contains multiple forms of expression, including but not limited to text data, image data, and voice data.
[0077] In this embodiment, the method for identifying abnormalities in multimodal data is executed on an electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the multimodal data sent by the client through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, Wi-Fi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (ultrawi deband) connection, and other wireless connection methods currently known or to be developed in the future.
[0078] Taking the field of financial risk control as an example, text data includes user information, transaction records, etc., image data includes user photos, ID card photos, etc., and voice data includes user voice commands, etc.
[0079] Step S202: performing data preprocessing on the multimodal data to obtain basic data corresponding to each modality.
[0080] Specifically, first, corresponding data preprocessing is performed for different types of data, such as word segmentation and vectorization for text data, feature extraction and dimensionality reduction for image data, and Mel-cephalogram feature extraction for speech data.
[0081] It should be noted that this embodiment can also process video type data, specifically, parsing the video data into audio data and image data, establishing an association between the audio data and the image data through a time frame, and then preprocessing the audio data and the image data.
[0082] Preferably, the basic data format corresponding to each modality in this embodiment is a vector format.
[0083] Step S203: performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data.
[0084] Specifically, in this embodiment, a vector splicing method is used to perform data fusion on the basic data.
[0085] Step S204, performing abnormal data identification based on the fusion vector to obtain an abnormal identification result.
[0086] Specifically, after the fusion vector is obtained, a training classifier is constructed through a neural network, and then the classifier is used to perform abnormality recognition and classification on the fusion vector to obtain an abnormality recognition result.
[0087] In this embodiment, the processing speed and the accuracy of abnormality identification are improved by processing different modal data, splicing and fusing them, and then performing abnormality identification on the fused data.
[0088] In some optional implementations, in step 202, the multimodal data includes text data, image data, and voice data, and performing data preprocessing on the multimodal data to obtain basic data corresponding to each modality includes the following steps:
[0089] Perform word segmentation and vectorization on text data to obtain the basic data corresponding to the text data;
[0090] Perform feature extraction and dimensionality reduction on image data to obtain basic data corresponding to the image data;
[0091] Mel-frequency cepstral coefficient feature extraction is performed on the speech data to obtain the basic data corresponding to the speech data.
[0092] Specifically, word segmentation and vectorization of text data are two basic steps in natural language processing (NLP). Word segmentation is to divide text into meaningful units (usually words or phrases), while vectorization is to convert these text units into numerical forms that can be processed by machine learning algorithms. It mainly includes two stages: word segmentation and vectorization.
[0093] Among them, word segmentation is to break down a text string into smaller pieces, usually words or phrases. In Chinese, word segmentation is especially important because Chinese text does not have obvious word separators (such as spaces) like English. The purpose of word segmentation is to enable computers to understand the structure of the text.
[0094] Vectorization is the process of converting text into numerical vectors that can be used as input by machine learning models.
[0095] Common vectorization methods include but are not limited to: Bag of Words, TF-IDF (Term Frequency-Inverse Document Frequency), Word2Vec, GloVe, and BERT, etc. These methods convert text into a fixed-length numerical vector, where each element in the vector represents the importance of a word or phrase in the text.
[0096] Specifically, feature extraction and dimensionality reduction of image data are common tasks in image processing and computer vision, which help to extract useful information from raw images and reduce the dimension of data, thereby improving the efficiency of subsequent processing.
[0097] Among them, feature extraction (Feature Ext ract i on) is to extract meaningful information from an image, which can represent important attributes of the image. Features include but are not limited to color, texture, shape, edge, etc. Common feature extraction methods include but are not limited to SI FT (Scale Invariant Feature Transform), SURF (Speeded Robust Features), HOG (Histogram of Oriented Gradients), etc.
[0098] Among them, dimensionality reduction is to reduce the dimension of data while retaining the most important features. Dimensionality reduction can improve data processing speed, reduce memory usage, and help prevent overfitting. Common dimensionality reduction techniques include but are not limited to principal component analysis (PCA), linear discriminant analysis (LDA), t-SNE (t-distributed stochastic neighbor embedding) and autoencoders.
[0099] Specifically, Mel-Frequency Cepstrum Coefficients (MFCC) are a common feature in speech processing. They are extracted from the power spectrum of the speech signal and can describe the spectral envelope of the speech signal. MFCC features are very useful for tasks such as speech recognition, speaker recognition, and sentiment analysis because they can capture the human ear's perception of sound. Extracting Mel-Frequency Cepstrum Coefficients from speech data includes the following steps:
[0100] Step 1: preprocessing, including denoising, endpoint detection, etc., to improve the quality of subsequent processing.
[0101] Step 2, framing, divides the speech signal into short time frames (usually 20-40 milliseconds) because the speech signal changes over time.
[0102] Step three: windowing. Each frame of data is multiplied by a window function (such as a Hamming window) to reduce the impact of the boundary between frames.
[0103] Step 4: Fast Fourier Transform (FFT) converts the time domain signal into a frequency domain signal.
[0104] Step 5: Calculate the power spectrum, which is usually obtained by taking the square of the absolute value of the FFT result.
[0105] Step 6, Mel filter bank, filters the power spectrum through a series of Mel filters, and the response of each filter corresponds to the human ear's perception of different frequencies.
[0106] Step seven, take the logarithm of the output of the Mel filter to simulate the nonlinear perception of sound intensity by the human ear.
[0107] Step 8: Discrete Cosine Transform (DCT): perform DCT on the output of the logarithmic Mel filter bank to obtain MFCC.
[0108] Step nine, feature extraction, usually taking the first 13 coefficients of DCT as MFCC features.
[0109] In this embodiment, by separating and preprocessing data of different modalities, vector morphological data corresponding to various modal data are obtained, which is beneficial to the subsequent fusion processing of different data.
[0110] In some optional implementations, in step 204, the step of performing abnormal data identification based on the fusion vector to obtain an abnormal identification result specifically includes the following steps:
[0111] Aggregate the fused vector to obtain N data clusters, where N is a category of the data cluster and is a positive integer;
[0112] Perform statistical analysis on each data cluster to obtain analysis results;
[0113] A classifier is used to identify abnormal data on the analysis result to obtain an abnormal identification result.
[0114] In this embodiment, a vector fusion method is adopted to aggregate data of different rule types to the same dimension for identification and classification, which is beneficial to improving the efficiency and accuracy of identification and classification.
[0115] In some optional implementations, the step of using a classifier to identify abnormal data on the analysis result to obtain an abnormal identification result specifically includes the following steps:
[0116] Using a support vector machine and a convolutional neural network to perform abnormality recognition on the analysis results, respectively, to obtain a first recognition result and a second recognition result;
[0117] Based on the first recognition result and the second recognition result, an abnormal recognition result is determined.
[0118] Among them, support vector machine (SVM) is a supervised learning algorithm commonly used for classification problems. It can distinguish different categories of data by using hyperplanes. In the field of anomaly detection, one-class support vector machine (One-Class SVM or OCSVM) is widely used. It is able to define what is "normal" by learning the pattern of normal data, thereby identifying abnormal data that is significantly different from the normal pattern. This model is particularly suitable for those scenarios where we only have normal data samples, such as financial fraud detection, intrusion detection in network security, defect detection in industrial production, etc.
[0119] Among them, convolutional neural networks (CNNs) are a type of deep learning model that excels in the fields of image and sound recognition. CNNs extract features of input data through convolutional layers, which are then used for classification or other tasks. In terms of anomaly detection, CNNs can be used to identify abnormal patterns in images, videos, or sounds. For example, they can be used in medical image analysis to identify abnormal tissues, or in industrial applications to detect product defects. CNNs have also shown good performance in abnormal sound recognition tasks, especially in noisy conditions, by extracting features of sound signals and using these features for effective anomaly detection.
[0120] In addition, CNNs are also used for abnormal behavior detection, such as identifying unusual activities or behavior patterns by analyzing video data. These networks learn the characteristics of normal behavior and then identify behaviors that differ significantly from these characteristics as abnormal.
[0121] In this embodiment, the support vector machine and the convolutional neural network are trained in advance.
[0122] Among them, the support vector machine training is detailed as follows:
[0123] Sample label: Assign a label to each sample in the training set (usually 1 for positive samples and -1 for negative samples) to indicate the category of the sample;
[0124] Training set construction: Take the feature vector and the corresponding label as the input data of the training set to construct a data set for training the classifier;
[0125] Kernel function selection: Select a suitable kernel function to map the input feature vector to a high-dimensional feature space. Common kernel functions include linear kernel, polynomial kernel, and Gaussian kernel.
[0126] Model construction: Based on the training set data, an SVM classifier is constructed using an optimization algorithm. The main goal is to find an optimal hyperplane to separate samples of different categories;
[0127] Optimization problem: Convert the training problem of the SVM classifier into a quadratic programming problem, and obtain the optimal solution by solving the optimization problem;
[0128] Support vector selection: During the training process, sample points located on the decision boundary, i.e. support vectors, are selected. They play a key role in defining the optimal hyperplane;
[0129] Determine classifier parameters: Calculate classifier parameters based on support vectors, including the size of decision boundaries and intervals;
[0130] Model evaluation: Use the test set to evaluate the model performance and calculate the classifier's accuracy, recall, F1 value and other indicators.
[0131] Among them, the neural network model training is detailed as follows:
[0132] Construct a Convolutional Neural Networks (CNN) model: Design a CNN model, including input layer, convolution layer, pooling layer, fully connected layer and output layer. Select the appropriate network structure and number of layers according to the complexity of the task and the characteristics of the dataset.
[0133] Initialization parameters: Randomly initialize the weight parameters and bias items of the convolution kernel for subsequent model training;
[0134] Forward propagation: The preprocessed training samples are passed through the CNN network, and convolution operations, nonlinear activation function (such as ReLU) processing, pooling operations, etc. are performed in sequence to obtain the final feature representation;
[0135] Loss function: Select an appropriate loss function to measure the difference between the model's prediction results and the true label. Common loss functions include cross entropy loss function, mean square error loss function, etc.
[0136] Back propagation: Based on the loss function, the gradient is calculated through the back propagation algorithm, and the model parameters are updated according to the gradient. These updates will make the model's prediction results closer to the true label;
[0137] Parameter optimization: Use optimization algorithms (such as stochastic gradient descent algorithm) to optimize parameters and continuously adjust the weights and bias terms of the convolution kernel to minimize the loss function.
[0138] Iterative training: Repeat the above steps until a preset stop condition is reached, such as reaching the maximum number of iterations or the loss function converges;
[0139] Model evaluation: Use validation sets or cross-validation methods to evaluate the performance of the trained CNN model, calculate indicators such as accuracy, recall, F1 value, and draw learning curves for model selection and adjustment;
[0140] Prediction: Use the trained CNN classifier to predict new samples, input the samples into the model through forward propagation, and obtain the output result, that is, the class probability or class label of the sample. It should be noted that during the training process, batch gradient descent algorithm or stochastic gradient descent algorithm can be used to update parameters. In addition, regularization techniques (such as L2 regularization) can be used to prevent overfitting, and strategies such as learning rate scheduling and early stopping can be used to improve training results. According to the characteristics of specific tasks and data sets, the network structure can also be adjusted and optimized.
[0141] Further, based on the first recognition result and the second recognition result, determining the abnormal recognition result includes:
[0142] The data with abnormalities in the first recognition result and the second recognition result are screened out as abnormal data, and an abnormal recognition result is generated according to the abnormal data.
[0143] In this embodiment, anomalies are identified through trained support vector machines and convolutional neural networks, thereby improving recognition accuracy and efficiency.
[0144] Furthermore, if there is abnormal vector data in the first recognition result and the second recognition result, multimodal data corresponding to the abnormal vector data is obtained; and an abnormal recognition result is generated according to the multimodal data corresponding to the multi-abnormal vector data.
[0145] In this embodiment, the abnormality recognition result is generated by using the recognition results determined in two different ways, which is beneficial to improving the accuracy of abnormality recognition.
[0146] In some optional implementations of this embodiment, after step 204, the electronic device may further perform the following steps:
[0147] Determining a target abnormality level according to the abnormality identification result;
[0148] Obtain the correspondence between the preset warning level and the warning measure, and execute the warning measure corresponding to the target abnormal level to perform warning processing.
[0149] Specifically, after obtaining the abnormality identification result, the preset early warning measures are executed according to the actual application needs, wherein the preset early warning measures include but are not limited to warning, deleting abnormal data, backup, etc.
[0150] In this embodiment, by classifying the abnormal recognition results, the abnormal level is obtained, and then corresponding warning measures are taken according to the abnormal level, and timely warnings are issued for multimodal data with abnormalities, which is conducive to improving the security and timeliness of data recognition processing.
[0151] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned abnormal identification results, the above-mentioned abnormal identification results can also be stored in a node of a blockchain.
[0152] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0153] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0154] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0155] This application can be applied in the field of smart finance, thereby promoting the construction of smart cities.
[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0157] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0158] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for identifying abnormalities in multimodal data. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0159] like Figure 3 As shown, the multimodal data anomaly identification device 300 described in this embodiment includes: an acquisition module 301, a preprocessing module 302, a fusion module 303 and an identification module 304. Among them:
[0160] An acquisition module 301 is used to acquire multimodal data;
[0161] A preprocessing module 302 is used to perform data preprocessing on the multimodal data to obtain basic data corresponding to each modality;
[0162] A fusion module 303 is used to perform data fusion on the basic data corresponding to each modality to generate a fusion vector of multi-modal data;
[0163] The identification module 304 is used to identify abnormal data based on the fusion vector to obtain an abnormal identification result.
[0164] In this embodiment, the processing speed and the accuracy of abnormality identification are improved by processing different modal data, splicing and fusing them, and then performing abnormality identification on the fused data.
[0165] Furthermore, the preprocessing module 302 includes:
[0166] A first processing unit is used to perform word segmentation and vectorization on the text data to obtain basic data corresponding to the text data;
[0167] The second processing unit is used to perform feature extraction and dimension reduction on the image data to obtain basic data corresponding to the image data;
[0168] The third processing unit is used to extract Mel-frequency cepstral coefficient features from the speech data to obtain basic data corresponding to the speech data.
[0169] In this embodiment, by separating and preprocessing data of different modalities, vector morphological data corresponding to various modal data are obtained, which is beneficial to the subsequent fusion processing of different data.
[0170] Further, the identification module 304 includes:
[0171] An aggregation unit, used for performing aggregation processing on the fused vector to obtain N data clusters, where N is the category of the data cluster and N is a positive integer;
[0172] An analysis unit, used for performing statistical analysis on each data cluster to obtain analysis results;
[0173] The identification unit is used to use a classifier to identify abnormal data from the analysis results to obtain an abnormal identification result.
[0174] In this embodiment, anomalies are identified through trained support vector machines and convolutional neural networks, thereby improving recognition accuracy and efficiency.
[0175] Furthermore, the result determination subunit includes:
[0176] A discriminating component, configured to obtain multimodal data corresponding to abnormal vector data if abnormal vector data exists in the first recognition result and the second recognition result;
[0177] A generating component is used to generate an anomaly recognition result according to the multi-modal data corresponding to the multiple anomaly vector data.
[0178] In this embodiment, the abnormality recognition result is generated by using the recognition results determined in two different ways, which is beneficial to improving the accuracy of abnormality recognition.
[0179] Furthermore, the multimodal data anomaly identification device comprises:
[0180] A grading module, used to determine a target abnormality level according to the abnormality identification result;
[0181] The early warning module is used to obtain the corresponding relationship between the preset early warning level and the early warning measure, and execute the early warning measure corresponding to the target abnormal level to perform early warning processing.
[0182] In this embodiment, by classifying the abnormality recognition results to obtain the abnormality level, and then taking corresponding warning measures according to the abnormality level, it is beneficial to give priority to warnings of important matters and improve the timeliness and flexibility of warnings.
[0183] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.
[0184] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital processors (DSPs), embedded devices, etc.
[0185] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0186] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the abnormal identification method of multimodal data, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.
[0187] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the anomaly identification method of the multimodal data.
[0188] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.
[0189] This embodiment improves the processing speed and the accuracy of abnormality identification by processing different modal data, splicing and fusing them, and then performing abnormality identification on the fused data.
[0190] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the multimodal data anomaly identification method as described above.
[0191] This embodiment improves the processing speed and the accuracy of abnormal identification by processing different modal data, splicing and fusing them, and then identifying abnormalities on the fused data. Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions to enable a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0192] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for identifying anomalies in multimodal data, characterized in that: The steps include: Acquire multimodal data; Performing data preprocessing on the multimodal data to obtain basic data corresponding to each modality; Performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data; Abnormal data is identified based on the fusion vector to obtain an abnormal identification result.
2. The method for identifying anomalies in multimodal data according to claim 1, characterized in that: The multimodal data includes text data, image data and voice data, and the data preprocessing is performed on the multimodal data to obtain basic data corresponding to each modality, including: Performing word segmentation and vectorization on the text data to obtain basic data corresponding to the text data; Performing feature extraction and dimensionality reduction on the image data to obtain basic data corresponding to the image data; Mel-frequency cepstral coefficient features are extracted from the speech data to obtain basic data corresponding to the speech data.
3. The method for identifying anomalies in multimodal data according to claim 1, characterized in that: The step of performing abnormal data identification based on the fusion vector to obtain an abnormal identification result specifically includes: Aggregate the fused vector to obtain N data clusters, where N is a category of the data cluster and is a positive integer; Perform statistical analysis on each data cluster to obtain analysis results; A classifier is used to identify abnormal data on the analysis result to obtain an abnormal identification result.
4. The method for identifying anomalies in multimodal data according to claim 3, characterized in that: The step of using a classifier to identify abnormal data on the analysis result to obtain an abnormal identification result specifically includes: Using a support vector machine and a convolutional neural network to perform abnormality recognition on the analysis results, respectively, to obtain a first recognition result and a second recognition result; Based on the first recognition result and the second recognition result, an abnormal recognition result is determined.
5. The method for identifying anomalies in multimodal data according to claim 4, characterized in that: The step of determining an abnormal recognition result based on the first recognition result and the second recognition result specifically includes: If there is abnormal vector data in the first recognition result and the second recognition result, obtaining multimodal data corresponding to the abnormal vector data; An abnormality recognition result is generated according to the multimodal data corresponding to the multiple abnormality vector data.
6. The method for identifying anomalies in multimodal data according to claim 3, characterized in that: After the step of using a classifier to identify abnormal data on the analysis result to obtain an abnormal identification result, the method further includes: Determining a target abnormality level according to the abnormality identification result; Obtain the correspondence between the preset warning level and the warning measure, and execute the warning measure corresponding to the target abnormal level to perform warning processing.
7. A device for identifying anomalies in multimodal data, characterized in that: include: An acquisition module, used to acquire multimodal data; A preprocessing module, used to perform data preprocessing on the multimodal data to obtain basic data corresponding to each modality; A fusion module, used for performing data fusion on the basic data corresponding to each modality to generate a fusion vector of the multimodal data; The recognition module is used to identify abnormal data based on the fusion vector to obtain an abnormal recognition result.
8. The device for identifying abnormalities in multimodal data according to claim 7, characterized in that: The multimodal data includes text data, image data and voice data, and the preprocessing module includes: A first processing unit, configured to perform word segmentation and vectorization on the text data to obtain basic data corresponding to the text data; A second processing unit is used to perform feature extraction and dimension reduction on the image data to obtain basic data corresponding to the image data; The third processing unit is used to extract Mel-frequency cepstral coefficient features from the speech data to obtain basic data corresponding to the speech data.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the method for identifying anomalies in multimodal data according to any one of claims 1 to 6 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for identifying anomalies in multimodal data according to any one of claims 1 to 6 are implemented.