Intelligent detection method, system and equipment based on multi-modal fusion and machine storage medium

Abstract

The invention provides an intelligent detection method, system and device based on multi-modal fusion and a machine storage medium. The method comprises the following steps: acquiring multi-modal data in a target area, the multi-modal data at least comprising image data, sensor data, sound data and text data; preprocessing the multi-modal data, wherein the preprocessing at least comprises denoising, standardization and feature extraction; performing feature fusion through a multi-modal fusion technology by using the preprocessed multi-modal data to generate a fusion feature vector; and based on the fusion feature vector and the obtained target data, performing classification or regression analysis by using a machine learning algorithm to obtain a detection result corresponding to the target data. According to the technical scheme provided by the invention, the accuracy and reliability of the detection result are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent detection technology, and in particular to an intelligent detection method, system, device and storage medium based on multimodal fusion. Background Art

[0002] With the development of IoT technology and big data, more and more application scenarios require real-time monitoring and intelligent analysis of multi-source heterogeneous data in complex environments. For example, in the fields of industrial production monitoring, environmental monitoring, medical diagnosis, etc., traditional single-modality data processing methods are difficult to meet the growing demand. These scenarios often require the integration of multiple types of data (such as images, sounds, sensor readings, text records, etc.) in order to more comprehensively understand the on-site situation and make accurate judgments.

[0003] Currently, most intelligent detection methods rely on data from a single modality for analysis. For example, image recognition technology is used to detect specific objects or sensor data is used to monitor changes in the physical environment. Although these methods perform well in certain scenarios, they may be limited by data limitations in complex environments, resulting in inaccurate or incomplete detection results. Summary of the invention

[0004] The embodiments of the present application provide an intelligent detection method, system, device and storage medium based on multimodal fusion to solve the problem of inaccurate detection results in the prior art.

[0005] In a first aspect, an embodiment of the present application provides an intelligent detection method based on multimodal fusion, comprising:

[0006] Collecting multimodal data in the target area, wherein the multimodal data includes at least image data, sensor data, sound data, and text data;

[0007] Preprocessing the multimodal data, wherein the preprocessing at least includes denoising, standardization, and feature extraction;

[0008] Using the pre-processed multimodal data, feature fusion is performed through multimodal fusion technology to generate a fused feature vector;

[0009] Based on the fused feature vector and the acquired target data, a machine learning algorithm is used to perform classification or regression analysis to obtain a detection result corresponding to the target data.

[0010] Optionally, the multimodal fusion technology includes at least a deep learning model, feature splicing, an attention mechanism, and a gated recursive unit.

[0011] Optionally, the feature fusion comprises the following steps:

[0012] Mapping data of different modalities into a unified feature space to form feature representations of each modality;

[0013] Apply deep learning network to extract features of each modality;

[0014] The features of each modality are selectively fused through the gating mechanism to generate a fused feature vector.

[0015] Optionally, the preprocessing the multimodal data further includes:

[0016] Applying a Gaussian blur filter to the image data for denoising, wherein the Gaussian blur filter has a kernel size of k×k and a standard deviation of σ;

[0017] Wavelet transform is applied to the sensor data to perform noise reduction processing, wherein the wavelet transform adopts Daubechies wavelet basis and the number of decomposition layers is j.

[0018] Optionally, the using a machine learning algorithm to perform classification or regression analysis based on the fused feature vector and the acquired target data to obtain a detection result corresponding to the target data includes:

[0019] Construct a support vector machine model, the support vector machine model is used to use the kernel function K(x i , x j ) to deal with nonlinear separable problems;

[0020] Calculate the fusion feature vector X = {x1, x2, ..., x n}The mapping value Φ(X) under the kernel function;

[0021] Using the soft margin maximization principle to determine the optimal classification hyperplane to complete the training of the support vector machine model;

[0022] The optimal classification hyperplane is defined by the following formula:

[0023]

[0024] subject to 0≤α i ≤C, i=1,…,n

[0025]

[0026] Among them, α i and α j is the Lagrange multiplier, y i is the sample label, y j For sample x j The category label, C is the penalty coefficient, n is the number of training samples, K(x i , x j) is the kernel function, Φ(X) is the mapping function, x i and x j : Sample points in the training set;

[0027] Inputting the acquired target data into a trained support vector machine model to determine the detection result corresponding to the target data;

[0028] The step of inputting the acquired target data into a trained support vector machine model to determine a detection result corresponding to the target data includes:

[0029] By formula: Determine a detection result corresponding to the target data, wherein the detection result at least includes a category to which the target data belongs;

[0030] Among them, f(x new ) is a prediction function used to output the detection result corresponding to the target data, x new is the target data, K(x i , x new ) is the kernel function, which is used to measure the training sample x i and target data x new Similarity in high-dimensional space, b is the bias term.

[0031] Optionally, the step of performing feature fusion by using a multimodal fusion technology to generate a fused feature vector includes:

[0032] Use adaptive weighting mechanism to adjust the importance of different modal features;

[0033] The adaptive weighting mechanism comprises the following steps:

[0034] Define the eigenvectors of mode m

[0035] Calculate the eigenvector F m The importance weight w m ,in:

[0036]

[0037] Among them, F m is the eigenvector of the mth mode, is the i-th eigenvalue in the m-th modal eigenvector, n m is the dimension of the mth modal eigenvector, w m is the importance weight of the mth modal feature vector, M is the number of modes, β m is a hyperparameter used to adjust the importance of mode m, is the eigenvector F mThe information quantity measurement includes at least information entropy or variance, β m' is a hyperparameter used to adjust the importance of the m'th mode, is the eigenvector F m' The information content measurement of

[0038] According to the weight w m Adjust the importance of different modal features to obtain the fused feature vector

[0039] Optionally, the information quantity metric Calculated by the following formula:

[0040]

[0041] in, is the eigenvector F m The average value of is the information measure of the mth modal eigenvector, such as information summary or variance, is the i-th eigenvalue in the m-th modal eigenvector, n m The dimension of the mth modal eigenvector, The mean of the mth mode eigenvectors.

[0042] In a second aspect, an embodiment of the present application provides an intelligent detection system based on multimodal fusion, comprising:

[0043] A data acquisition module, used to acquire multimodal data in a target area, wherein the multimodal data includes at least image data, sensor data, sound data and text data;

[0044] A data preprocessing module, used for preprocessing the multimodal data, wherein the preprocessing includes at least denoising, standardization, and feature extraction;

[0045] The feature fusion module is used to utilize the pre-processed multimodal data to perform feature fusion through multimodal fusion technology to generate a fused feature vector;

[0046] The intelligent detection module is used to perform classification or regression analysis using a machine learning algorithm based on the fused feature vector and the acquired target data to obtain a detection result corresponding to the target data.

[0047] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the multimodal fusion-based intelligent detection methods described in the first aspect.

[0048] In a fourth aspect, an embodiment of the present application provides a computer storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the intelligent detection method based on multimodal fusion described in any one of the first aspects is implemented.

[0049] In the embodiment of the present application, multimodal data in the target area is collected, and the multimodal data includes at least image data, sensor data, sound data and text data; the multimodal data is preprocessed, and the preprocessing includes at least denoising, standardization, and feature extraction; the preprocessed multimodal data is used to perform feature fusion through multimodal fusion technology to generate a fusion feature vector; based on the fusion feature vector and the acquired target data, a machine learning algorithm is used to perform classification or regression analysis to obtain the detection result corresponding to the target data. The technical solution provided by the present application improves the accuracy and reliability of the detection results.

[0050] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of an intelligent detection method based on multimodal fusion provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of the structure of an intelligent detection system based on multimodal fusion provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0056] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0058] Figure 1 A flowchart of an intelligent detection method based on multimodal fusion is provided for an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0059] 101. Collect multimodal data in a target area, where the multimodal data includes at least image data, sensor data, sound data, and text data;

[0060] In this step, collecting multimodal data in the target area refers to obtaining different types of raw data through various sensors and technical means. These data can come from different senses or measurement methods, including but not limited to image data, sensor data, sound data and text data. Multimodal data refers to different forms of data obtained from the same target or event. By combining these different types of data, a more comprehensive understanding and analysis of the target or event can be achieved.

[0061] Image data: refers to the visual information obtained through a camera or other imaging device, which can be a static picture or a dynamic video stream.

[0062] Sensor data: refers to the measured values ​​of physical or chemical quantities obtained through various sensors (such as temperature sensors, humidity sensors, pressure sensors, etc.).

[0063] Sound data: refers to sound signals captured by a microphone or other recording equipment.

[0064] Text data: refers to text information obtained through natural language processing or text mining technology, which can be text data collected from sources such as social media, news reports, research reports, etc.

[0065] In the embodiment of this application, assuming that our goal is to monitor the safety status of a bridge, we need to collect multimodal data for comprehensive analysis. The specific steps are as follows:

[0066] Collecting image data: Installing high-definition cameras at key locations on the bridge regularly captures images of the bridge’s exterior. These images can help us check for obvious cracks or structural damage.

[0067] Collecting sensor data: Various sensors are deployed at different locations on the bridge, such as vibration sensors, stress sensors, and temperature and humidity sensors (Vibration Sensor, VS; Stress Sensor, SS; Temperature and Humidity Sensor, THS). These sensors can continuously monitor the vibration frequency, stress distribution, and environmental conditions of the bridge.

[0068] Acoustic Sensors (AS) are placed near bridges to record the noise generated by vehicles passing by. By analyzing the noise pattern, it is possible to indirectly infer whether the bridge has abnormal vibration or damage.

[0069] Collect text data: Collect text descriptions of bridge conditions from relevant maintenance reports, user feedback, and social media platforms. These text data provide qualitative information that helps understand the public's views on bridge safety and possible problem reports.

[0070] Assume that we have collected the following data:

[0071] Image data: A picture of the bridge is taken once every morning at 8 o'clock.

[0072] Sensor data: Vibration data (e.g., vibration frequency of 10 Hz), stress data (e.g., maximum stress value of 100 MPa), and temperature and humidity data (e.g., temperature of 20°C and humidity of 60%) of the bridge are recorded every hour.

[0073] Sound data: Ambient noise is recorded for 1 minute every half hour.

[0074] Text data: Monthly reports on bridge status and comments on social media were collected.

[0075] Next, we preprocess the data, such as denoising and edge enhancement of images, filtering and smoothing of sensor data, spectral analysis of sound data, and keyword extraction of text data.

[0076] The following are numerical examples:

[0077] Vibration frequency data recorded by the vibration sensor: 10Hz.

[0078] The maximum stress value recorded by the stress sensor: 100MPa.

[0079] Data recorded by the temperature and humidity sensor: temperature 20℃, humidity 60%.

[0080] Noise level in sound data: The spectrum is obtained through Fourier transform, and the energy peak at a specific frequency is found to be 100dB.

[0081] Keywords in text data: Through text mining technology, keywords such as "cracks" and "repairs" are extracted from the report.

[0082] Through the collection and preprocessing of these multimodal data, we have laid the foundation for subsequent feature fusion and intelligent detection.

[0083] 102. Preprocess the multimodal data, wherein the preprocessing at least includes denoising, standardization, and feature extraction;

[0084] Optionally, the preprocessing of the multimodal data in step 102 further includes:

[0085] Applying a Gaussian blur filter to the image data for denoising, wherein the Gaussian blur filter has a kernel size of k×k and a standard deviation of ρ;

[0086] Wavelet transform is applied to the sensor data to perform noise reduction processing, wherein the wavelet transform adopts Daubechies wavelet basis and the number of decomposition layers is j.

[0087] In this step, multimodal data is preprocessed to ensure that subsequent data analysis and feature fusion can be based on high-quality data. Preprocessing includes at least denoising, standardization, and feature extraction. These steps help remove noise from the data, make the data more consistent, and extract useful features from it.

[0088] Denoising: Remove random noise from data to improve data quality.

[0089] Standardization: Make the data have the same scale to facilitate subsequent processing.

[0090] Feature extraction: Extract features from raw data that are useful for subsequent analysis.

[0091] Here is the denoising method:

[0092] Image data denoising: You can use a Gaussian Blur Filter to perform denoising. The Gaussian Blur Filter smoothes the image by applying a Gaussian Kernel to reduce the impact of noise.

[0093] Sensor data denoising: Wavelet transform can be used for denoising. Wavelet transform is a time-frequency analysis tool that can effectively separate the noise components in the signal and remove these noises through threshold processing and other methods.

[0094] In the embodiment of the present application, it is assumed that we have captured an image from a camera, and the image has certain noise interference and needs to be denoised.

[0095] Gaussian blur filter: Apply a Gaussian blur filter to denoise the image. The kernel size of the Gaussian blur filter is k×k and the standard deviation is σ.

[0096] Kernel size: Set k = 5, that is, use a 5×5 Gaussian kernel.

[0097] Standard deviation: Let σ = 1.4. The standard deviation determines the width of the Gaussian kernel. A smaller standard deviation will make the kernel more concentrated, and a larger standard deviation will make the kernel more dispersed.

[0098] The calculation formula of Gaussian kernel is:

[0099]

[0100] For each pixel, the weighted average of all pixel values ​​in the surrounding 5×5 area is calculated, and the weight is determined by the Gaussian kernel.

[0101] Sensor Data Denoising

[0102] Suppose we have collected a series of vibration data from a vibration sensor on a bridge and need to perform noise reduction on it.

[0103] Wavelet transform: Use wavelet transform to reduce the noise of sensor data. The wavelet transform uses Daubechies Wavelet Basis and the number of decomposition layers is j.

[0104] Daubechies wavelet basis: Select Daubechies4 wavelet basis (db4), which is a commonly used wavelet basis with good time-frequency localization characteristics.

[0105] Decomposition level: Set j = 2, that is, two-layer wavelet decomposition is performed. Wavelet transform decomposes the original signal into an approximation signal (approximate component) and a detail signal (detailed component). The detail signal is processed by setting an appropriate threshold to remove the noise component and then reconstruct the signal.

[0106] The following are the normalization and feature extraction:

[0107] Assuming that we have completed the denoising of image data and sensor data, we need to normalize and extract features from these data.

[0108] Normalization: The denoised image data and sensor data are normalized to have zero mean and unit variance.

[0109] Image data: Calculate the mean and standard deviation of the image data and normalize each pixel value.

[0110] Sensor Data: Calculate the mean and standard deviation of the sensor data and normalize each data point.

[0111] The standardized formula is:

[0112]

[0113] Among them, x is the original data point, μ is the mean of the data, and σ is the standard deviation of the data.

[0114] Feature extraction: Extract useful features from standardized data.

[0115] Image Data: Edge features in images can be extracted using edge detection algorithms such as the Sobel operator.

[0116] Sensor data: The frequency domain features of the signal can be extracted using Fourier Transform (FT).

[0117] The following is a complete example:

[0118] Suppose we install a camera and a vibration sensor on a bridge to collect image data and vibration data respectively. To ensure the accuracy of subsequent analysis, we first preprocess these two types of data.

[0119] Image data denoising: Use a 5×55×5 Gaussian kernel (standard deviation σ=1.4σ=1.4) to denoise the image and obtain the denoised image.

[0120] Sensor data denoising: Use Daubechies4 wavelet basis to perform two-layer wavelet decomposition, perform threshold processing on detail signals, remove noise components and reconstruct the signal to obtain denoised vibration data.

[0121] Standardization: The denoised image data and vibration data are standardized to have zero mean and unit variance.

[0122] Feature extraction: Extract edge features from the standardized image data and extract frequency domain features from the vibration data.

[0123] Through the above preprocessing steps, we obtain high-quality image data and sensor data, which provide a reliable foundation for subsequent feature fusion and intelligent detection.

[0124] 103. Using the pre-processed multimodal data, perform feature fusion through multimodal fusion technology to generate a fused feature vector;

[0125] Optionally, the multimodal fusion technology in step 103 includes at least a deep learning model, feature splicing, an attention mechanism, and a gated recursive unit.

[0126] Optionally, the feature fusion in step 103 includes the following steps: mapping data of different modalities to a unified feature space to form feature representations of each modality; applying a deep learning network to extract features of each modality; and selectively fusing features of each modality through a gating mechanism to generate a fused feature vector.

[0127] Optionally, the step 103 of performing feature fusion by using a multimodal fusion technology to generate a fused feature vector includes:

[0128] Use adaptive weighting mechanism to adjust the importance of different modal features;

[0129] The adaptive weighting mechanism comprises the following steps:

[0130] Define the eigenvectors of mode m

[0131] Calculate the eigenvector F m The importance weight w m ,in:

[0132]

[0133] Among them, F m is the eigenvector of the mth mode, is the i-th eigenvalue in the m-th modal eigenvector, n m is the dimension of the mth modal eigenvector, w m is the importance weight of the mth modal feature vector, M is the number of modes, β m is a hyperparameter used to adjust the importance of mode m, is the eigenvector Fm The information quantity measurement includes at least information entropy or variance, β m' is a hyperparameter used to adjust the importance of the m'th mode, is the eigenvector F m' The information quantity measurement.

[0134] According to the weight w m Adjust the importance of different modal features to obtain the fused feature vector

[0135] Among them, the information quantity measurement Calculated by the following formula:

[0136]

[0137] in, is the eigenvector F m The average value of is the information measure of the mth modal eigenvector, such as information summary or variance, is the i-th eigenvalue in the m-th modal eigenvector, n m The dimension of the mth modal eigenvector, The mean of the mth mode eigenvectors.

[0138] In this step, the preprocessed multimodal data is used for feature fusion to generate a fused feature vector. This process mainly includes the following aspects:

[0139] Feature mapping: Map data of different modalities into a unified feature space to form feature representations of each modality.

[0140] Feature extraction: Apply deep learning networks to extract features of each modality.

[0141] Feature fusion: The features of each modality are selectively fused through a gating mechanism to generate a fused feature vector.

[0142] Adaptive weighting mechanism: By calculating the importance weights of different modal feature vectors, the importance of different modal features is adjusted.

[0143] In the embodiment of the present application, it is assumed that we have completed the preprocessing of image data, sensor data, sound data and text data. Now it is necessary to perform feature fusion on these data to generate a fused feature vector.

[0144] Feature Mapping:

[0145] Mapping data from different modalities into a unified feature space. For example, converting image data into a set of feature vectors, converting sensor data into another set of feature vectors, and so on.

[0146] Feature extraction:

[0147] Deep learning networks (such as convolutional neural networks (CNN) and recurrent neural networks (RNN)) are used to extract the features of each modality.

[0148] For example, a convolutional neural network (CNN) is used to extract features from image data; a recurrent neural network (RNN) is used to extract features from sound data.

[0149] Feature Fusion:

[0150] The features of each modality are selectively fused through the gating mechanism to generate a fused feature vector.

[0151] For example, feature fusion is performed using a Gated Recurrent Unit (GRU) or a Long Short-Term Memory (LSTM).

[0152] Adaptive weighting mechanism:

[0153] Define the eigenvectors of mode m

[0154] Calculate the eigenvector F m The importance weight w m ,in:

[0155]

[0156] Among them, β m is a hyperparameter used to adjust the importance of mode m, I(F m ) is the eigenvector F m The amount of information is measured according to the weight w m Adjust the importance of different modal features to obtain the fused feature vector

[0157] Information quantity measurement I(F m ) is calculated by the following formula:

[0158]

[0159] in, is the eigenvector F m The average value of .

[0160] Assume that we have collected the following multimodal data:

[0161] Image data: Bridge images acquired through a camera.

[0162] Sensor data: Vibration data obtained through the vibration sensor.

[0163] Sound data: Sound data obtained through Sukphon.

[0164] Text data: text descriptions of bridge status obtained through social media.

[0165] Feature Mapping:

[0166] Image data: Use a pre-trained convolutional neural network (such as VGG16) to extract image features and obtain the feature vector F image

[0167] Sensor data: Use wavelet transform to extract the features of vibration data and obtain the feature vector F sensor .

[0168] Sound data: Use Fourier transform to extract the spectral features of the sound data and obtain the feature vector F audio。

[0169] Text data: Use word embedding (such as Word2Vec) to extract the features of text data and obtain the feature vector F text .

[0170] Feature extraction

[0171] Image data: Use convolutional neural network (CNN) to extract image features and obtain feature vector F image .

[0172] Sensor data: Use a recurrent neural network (RNN) to extract the features of the vibration data and obtain the feature vector F sensor .

[0173] Sound data: Use a recurrent neural network (RNN) to extract the features of the sound data and obtain the feature vector F audio .

[0174] Text data: Use the long short-term memory network (LSTM) to extract the features of the text data and obtain the feature vector F text .

[0175] The following are the feature fusions:

[0176] Gating mechanism: Use the gated recurrent unit (GRU) to fuse the features of each modality.

[0177] Adaptive weighting mechanism: Calculate the importance weight w of each modal feature vector m .

[0178] Adaptive weighting mechanism

[0179] Assume that we have four modal data, namely image data, sensor data, sound data and text data. We denote the feature vectors of these modalities as F image , F sensor , F audio , F text .

[0180] Information quantity measure: Calculate the information quantity measure I(F m ).

[0181] For image data:

[0182]

[0183] For sensor data:

[0184]

[0185] For sound data:

[0186]

[0187] For text data:

[0188]

[0189] Weight calculation: Calculate the importance weight w of each mode based on the information measurement m .

[0190]

[0191] Fusion feature vector: according to weight w m Adjust the importance of different modal features and obtain the fused feature vector F.

[0192] F=w image ·F image +w sensor ·F sensor +w audio ·F audio +w text ·F text

[0193] 104. Based on the fused feature vector and the acquired target data, a machine learning algorithm is used to perform classification or regression analysis to obtain a detection result corresponding to the target data.

[0194] Optionally, the step 104 uses a machine learning algorithm to perform classification or regression analysis based on the fused feature vector and the acquired target data to obtain a detection result corresponding to the target data, including: constructing a support vector machine model, wherein the support vector machine model is used to use a kernel function K(x i , x j ) to handle the nonlinear separable problem; calculate the fusion feature vector X = {x1, x2, ..., x n} mapping value Φ(X) under the kernel function; using the soft margin maximization principle to determine the optimal classification hyperplane to complete the training of the support vector machine model;

[0195] The optimal classification hyperplane is defined by the following formula:

[0196]

[0197] subject to 0≤α i ≤C, i=1,…,n

[0198]

[0199] Among them, α i and α j is the Lagrange multiplier, y i is the sample label, y j For sample x j The category label, C is the penalty coefficient, n is the number of training samples, K(x i , x j ) is the kernel function, Φ(X) is the mapping function, x i and x j : Sample points in the training set;

[0200] Inputting the acquired target data into a trained support vector machine model to determine the detection result corresponding to the target data;

[0201] The step of inputting the acquired target data into a trained support vector machine model to determine a detection result corresponding to the target data includes:

[0202] By formula: Determine a detection result corresponding to the target data, wherein the detection result at least includes a category to which the target data belongs;

[0203] Among them, f(x new ) is a prediction function used to output the detection result corresponding to the target data, x new is the target data, K(x i , x new ) is the kernel function, which is used to measure the training sample xi and target data x new Similarity in high-dimensional space, b is the bias term.

[0204] In this step, based on the fused feature vector and the acquired target data, a machine learning algorithm is used for classification or regression analysis to obtain the detection results corresponding to the target data. The specific steps include building a support vector machine (SVM) model, processing nonlinear separable problems through kernel functions, calculating the mapping value of the feature vector under the kernel function, determining the optimal classification hyperplane using the soft margin maximization principle, and finally inputting the target data into the trained model to determine the detection results.

[0205] Support Vector Machine (SVM) model: A supervised learning model used for classification and regression analysis, particularly for data in high-dimensional space.

[0206] Kernel Function (K(x_i, x_j)): It is used to map data in low-dimensional space to high-dimensional space, making it easier to find the classification hyperplane in high-dimensional space.

[0207] Mapping Function (Φ(X)): A function that maps the original feature vector to a high-dimensional space.

[0208] Soft margin maximization principle: used in SVM to find the optimal classification hyperplane, allowing a certain degree of misclassification to improve the generalization ability of the model.

[0209] Lagrange Multipliers (α_i): used to solve the dual problem in the SVM optimization problem and determine the degree of influence of the training samples on the classification boundary.

[0210] Bias Term (b): used to adjust the position of the classification hyperplane.

[0211] In the embodiment of the present application, a support vector machine model is constructed:

[0212] Assume that we have completed the collection and feature fusion of multimodal data and obtained the fused feature vector X = {x1, x2, ..., x n}. Now we need to use the support vector machine model to classify the target data.

[0213] Build a support vector machine model:

[0214] Kernel function: Select the Gaussian kernel function (Gaussian Kernel), also known as the radial basis function (Radial Basis Function, RBF).

[0215] K(x i , x j )=exp(-γ||x i -x j || 2 )

[0216] Among them, γ is the parameter of the kernel function, which is used to control the width of the kernel function.

[0217] Calculate the mapping value:

[0218] Calculate the mapping value Φ(X) of the feature vector X in the high-dimensional space. Since the Gaussian kernel function is used, there is actually no need to explicitly calculate the mapping value. Instead, the kernel function is directly used to calculate the similarity between samples.

[0219] Determine the optimal classification hyperplane:

[0220] The optimal classification hyperplane is determined using the soft margin maximization principle. The specific optimization problem is as follows:

[0221]

[0222] subject to 0≤α i ≤C, i=1,…,n

[0223]

[0224] Among them, α i is the Lagrange multiplier, y i is the sample label, C is the penalty coefficient, n is the number of training samples, K(x i , x j ) is the kernel function.

[0225] Train the model:

[0226] By solving the above optimization problem, we can obtain the optimal Lagrange multiplier α i And bias term b, thereby determining the optimal classification hyperplane.

[0227] Prediction target data:

[0228] The target data x new Input the trained support vector machine model to determine the 5 detection results corresponding to the target data.

[0229]

[0230] Among them, f(x ncw ) is the prediction function, which is used to output the target data x new The corresponding test results.

[0231] The following is a complete example:

[0232] Assume that we have completed the collection and feature fusion of multimodal data and obtained the fused feature vector X = {x1, x2, ..., x n}. Now we need to use the support vector machine model to classify the target data.

[0233] Build a support vector machine model:

[0234] Kernel function: Select Gaussian kernel function (RBF).

[0235] K(x i , x j )=exp(-γ||x i -x j || 2 )

[0236] Among them, γ=0.1.

[0237] Calculate the mapping value:

[0238] Calculate the mapping value Φ(X) of the feature vector X in the high-dimensional space. Since the Gaussian kernel function is used, there is actually no need to explicitly calculate the mapping value. Instead, the kernel function is directly used to calculate the similarity between samples.

[0239] Determine the optimal classification hyperplane:

[0240] The optimal classification hyperplane is determined using the soft margin maximization principle. The specific optimization problem is as follows:

[0241]

[0242] subject to 0≤α i ≤C, i=1,…,n

[0243]

[0244] Among them, α i is the Lagrange multiplier, y i is the sample label, C = 1 is the penalty coefficient, n = 100 is the number of training samples, K(x i , x j ) is the kernel function.

[0245] Train the model:

[0246] By solving the above optimization problem, we can obtain the optimal Lagrange multiplier α i And bias term b, thereby determining the optimal classification hyperplane.

[0247] Assume that the optimal Lagrange multiplier obtained is α i And the bias term b = -0.5.

[0248] Prediction target data:

[0249] The target data x ncw Input the trained support vector machine model to determine the detection results corresponding to the target data.

[0250]

[0251] Among them, f(x new ) is the prediction function, which is used to output the target data x new The corresponding test results.

[0252] Suppose we have the following training data and target data:

[0253] Training Data:

[0254] Sample size n=100.

[0255] Lagrange multiplier α i (Some examples):

[0256] α1=0.2,y1=+1

[0257] α2=0.1,y2=-1

[0258] α3=0.3,y3=+1

[0259] The bias term b = -0.5.

[0260] Target data: x new is the new target data point.

[0261] Calculate the prediction results:

[0262] Assume that the target data x new The kernel function value of the training data is:

[0263] K(x1,x new )=0.8

[0264] K(x2,x new )=0.2

[0265] K(x3,x new )=0.6 ...

[0267] The prediction function is:

[0268] f(x new)=sign(0.2·(+1)·0.8+0.1·(-1)·0.2+0.3·(+1)·0.6-0.5)

[0269] f(x new ) = sign (0.16-0.02+0.18-0.5)

[0270] f(x new )=sign(0.32-0.5)

[0271] f(x new ) = sign (-0.18) = -1

[0272] Therefore, the target data x new The prediction result is -1, which belongs to the negative class.

[0273] Through the above steps, we successfully used the support vector machine model to classify the target data and obtained reliable detection results.

[0274] Figure 2 A schematic diagram of the structure of an intelligent detection system based on multimodal fusion is provided for the embodiment of the present application, such as Figure 2 As shown, the system includes:

[0275] A data acquisition module 21, used to acquire multimodal data in a target area, wherein the multimodal data includes at least image data, sensor data, sound data and text data;

[0276] A data preprocessing module 22, used for preprocessing the multimodal data, wherein the preprocessing includes at least denoising, standardization, and feature extraction;

[0277] The feature fusion module 23 is used to use the pre-processed multimodal data to perform feature fusion through multimodal fusion technology to generate a fused feature vector;

[0278] The intelligent detection module 24 is used to perform classification or regression analysis using a machine learning algorithm based on the fused feature vector and the acquired target data to obtain a detection result corresponding to the target data.

[0279] Figure 2 The intelligent detection system based on multimodal fusion can perform Figure 1 The implementation principle and technical effect of the intelligent detection method based on multimodal fusion described in the illustrated embodiment will not be described in detail. The specific manner in which each module and unit performs operations in the intelligent detection system based on multimodal fusion in the above embodiment has been described in detail in the embodiment of the method, and will not be elaborated here.

[0280] In one possible design, Figure 2 An intelligent detection system based on multimodal fusion in the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0281] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32 .

[0282] The processing component 32 is used to: collect multimodal data in the target area, the multimodal data including at least image data, sensor data, sound data and text data; preprocess the multimodal data, the preprocessing including at least denoising, standardization and feature extraction; use the preprocessed multimodal data to perform feature fusion through multimodal fusion technology to generate a fused feature vector; based on the fused feature vector and the acquired target data, use a machine learning algorithm to perform classification or regression analysis to obtain a detection result corresponding to the target data.

[0283] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0284] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0285] Of course, the computing device may also include other components, such as input / output interfaces, display components, communication components, etc.

[0286] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0287] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0288] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0289] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The illustrated embodiment is an intelligent detection method based on multimodal fusion.

[0290] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0291] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0292] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0293] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An intelligent detection method based on multimodal fusion, characterized in that: include: Collecting multimodal data in the target area, wherein the multimodal data includes at least image data, sensor data, sound data, and text data; Preprocessing the multimodal data, wherein the preprocessing at least includes denoising, standardization, and feature extraction; Using the pre-processed multimodal data, feature fusion is performed through multimodal fusion technology to generate a fused feature vector; Based on the fused feature vector and the acquired target data, a machine learning algorithm is used to perform classification or regression analysis to obtain a detection result corresponding to the target data.

2. The method according to claim 1, characterized in that The multimodal fusion technology at least includes a deep learning model, feature splicing, an attention mechanism, and a gated recursive unit.

3. The method according to claim 1, characterized in that The feature fusion comprises the following steps: Mapping data of different modalities into a unified feature space to form feature representations of each modality; Apply deep learning network to extract features of each modality; The features of each modality are selectively fused through the gating mechanism to generate a fused feature vector.

4. The method according to claim 1, characterized in that: The preprocessing of the multimodal data further includes: Applying a Gaussian blur filter to the image data for denoising, wherein the Gaussian blur filter has a kernel size of k×k and a standard deviation of σ; Wavelet transform is applied to the sensor data to perform noise reduction processing, wherein the wavelet transform adopts Daubechies wavelet basis and the number of decomposition layers is j.

5. The method according to claim 1, characterized in that The method of performing classification or regression analysis based on the fused feature vector and the acquired target data using a machine learning algorithm to obtain a detection result corresponding to the target data includes: Construct a support vector machine model, the support vector machine model is used to use the kernel function K(x i , x j ) to deal with nonlinear separable problems; Calculate the fusion feature vector X = {x1, x2, ..., x m }The mapping value Φ(X) under the kernel function; Using the soft margin maximization principle to determine the optimal classification hyperplane to complete the training of the support vector machine model; The optimal classification hyperplane is defined by the following formula: Among them, α i and α j is the Lagrange multiplier, y i is the sample label, y j For sample x j The category label, C is the penalty coefficient, n is the number of training samples, K(x i , x j ) is the kernel function, Φ(X) is the mapping function, x i and x j : Sample points in the training set; Inputting the acquired target data into a trained support vector machine model to determine the detection result corresponding to the target data; The step of inputting the acquired target data into a trained support vector machine model to determine a detection result corresponding to the target data includes: By formula: Determine a detection result corresponding to the target data, wherein the detection result at least includes a category to which the target data belongs; Among them, f(x new ) is a prediction function used to output the detection result corresponding to the target data, x new is the target data, K(x i , x new ) is the kernel function, which is used to measure the training sample x i and target data x new Similarity in high-dimensional space, b is the bias term.

6. The method according to claim 1, characterized in that The method of performing feature fusion by using multimodal fusion technology to generate a fused feature vector includes: Use adaptive weighting mechanism to adjust the importance of different modal features; The adaptive weighting mechanism comprises the following steps: Define the eigenvectors of mode m Calculate the eigenvector F m The importance weight w m ,in: Among them, F m is the eigenvector of the mth mode, is the i-th eigenvalue in the m-th modal eigenvector, n m is the dimension of the mth modal eigenvector, w m is the importance weight of the mth modal feature vector, M is the number of modes, β m is a hyperparameter used to adjust the importance of mode m, is the eigenvector F m The information quantity measurement includes at least information entropy or variance, β m' is a hyperparameter used to adjust the importance of the m'th mode, is the eigenvector F m' The information content measurement of According to the weight w m Adjust the importance of different modal features to obtain the fused feature vector 7. The method according to claim 6, characterized in that The information quantity metric Calculated by the following formula: in, is the eigenvector F m The average value of is the information measure of the mth modal eigenvector, such as information summary or variance, is the i-th eigenvalue in the m-th modal eigenvector, n m The dimension of the mth modal eigenvector, The mean of the mth mode eigenvectors.

8. An intelligent detection system based on multimodal fusion, characterized in that: include: A data acquisition module, used to acquire multimodal data in a target area, wherein the multimodal data includes at least image data, sensor data, sound data and text data; A data preprocessing module, used for preprocessing the multimodal data, wherein the preprocessing includes at least denoising, standardization, and feature extraction; The feature fusion module is used to utilize the pre-processed multimodal data to perform feature fusion through multimodal fusion technology to generate a fused feature vector; The intelligent detection module is used to perform classification or regression analysis using a machine learning algorithm based on the fused feature vector and the acquired target data to obtain a detection result corresponding to the target data.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the intelligent detection method based on multimodal fusion as described in any one of claims 1 to 7.

10. A computer storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the intelligent detection method based on multimodal fusion described in any one of claims 1 to 7 is implemented.

Images