Small sample image classification method and system based on multi-modal data fusion

Abstract

This invention discloses a few-sample image classification method and system based on multimodal data fusion, relating to the field of computer vision technology. The key technical points are: acquiring multimodal data of a target power equipment and extracting image data, text data, and audio data from the multimodal data; using a convolutional neural network to extract local and global features of the image data, fusing the local and global features to obtain fused features of the image data; using a word embedding model to extract semantic features of the text data; using short-time Fourier transform to extract spectral features of the audio data; constructing a multimodal feature matrix based on the fused features, semantic features, and spectral features; employing a dynamic weight allocation algorithm to assign weights to the feature vectors of each modality in the multimodal feature matrix to obtain a weighted feature matrix; and using a support vector machine to perform classification training on the weighted feature matrix to generate image classification results for the target power equipment.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a few-sample image classification method and system based on multimodal data fusion. Background Technology

[0002] Image classification technology refers to the techniques and methods used in the fields of artificial intelligence and computer vision to automatically identify and classify target objects by analyzing and processing image data. Multimodal data fusion, applied to image classification technology, can improve the performance of classification models by integrating information from different modalities (such as images, text, and audio), which is particularly significant in scenarios with small sample sizes.

[0003] Existing technologies have the following shortcomings: Currently, classification methods mainly focus on the utilization of image features and do not involve the fusion analysis of multimodal data. In small sample scenarios, insufficient information affects the classification effect. They are limited to single-modal data processing and fail to fully explore the potential value of multimodal information, resulting in limited accuracy of classification results. Summary of the Invention

[0004] The purpose of this invention is to provide a few-sample image classification method and system based on multimodal data fusion, which solves the problem of insufficient accuracy of image classification results due to insufficient utilization of multimodal data in small-sample scenarios in existing technologies.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides a few-sample image classification method based on multimodal data fusion, the method comprising:

[0007] Acquire multimodal data of the target power equipment, and extract image data, text data, and audio data from the multimodal data;

[0008] Convolutional neural networks are used to extract local and global features from image data, and the local and global features are fused to obtain the fused features of the image data.

[0009] Extracting semantic features from text data using word embedding models;

[0010] Extracting spectral features from audio data using short-time Fourier transform;

[0011] A multimodal feature matrix is ​​constructed based on fusion features, semantic features, and spectral features;

[0012] A dynamic weight allocation algorithm is used to assign weights to the feature vectors of each mode in the multimodal feature matrix to obtain a weighted feature matrix;

[0013] Support vector machines are used to train the weighted feature matrix for classification, generating image classification results for the target power equipment.

[0014] In one implementation, a convolutional neural network is used to extract local and global features from image data, including:

[0015] The shallow structure of a convolutional neural network is used to extract local features from image data. The shallow structure includes three convolutional layers, each followed by a max pooling layer, which is used to extract local details of the edges and textures of the target electrical equipment.

[0016] The deep structure of a convolutional neural network is used to extract global features from image data; the deep structure includes five convolutional layers and two fully connected layers to capture the shape distribution and spatial distribution information of the target power equipment.

[0017] In one implementation, a dynamic weight allocation algorithm is used to assign weights to the eigenvectors of each mode in the multimodal feature matrix, resulting in a weighted feature matrix, including:

[0018] An initial weight matrix is ​​constructed based on the dimension and distribution characteristics of the feature vectors of each mode in the multimodal feature matrix; where the initial weight value of the initial weight matrix is ​​1 / N, and N is the number of modes;

[0019] The correlation between the feature vectors of each mode in the multimodal feature matrix is ​​calculated using the cosine similarity formula, and a correlation matrix is ​​generated.

[0020] The initial weight values ​​of the initial weight matrix are adjusted based on the correlation matrix. The adjustment process is as follows: feature vectors with high correlation are assigned the first weight, and feature vectors with low correlation are assigned the second weight. The first weight is greater than the second weight.

[0021] The initial weight matrix after adjusting the initial weight values ​​is normalized so that the sum of the weight values ​​in each column of the initial weight matrix is ​​1, thus obtaining the dynamic weight matrix.

[0022] Multiply the dynamic weight matrix by the multimodal feature matrix to obtain the weighted feature matrix.

[0023] In one implementation, the method further includes: setting a dynamic adjustment strategy for each weight value in the dynamic weight matrix, and performing dynamic adjustment of each weight value in the dynamic weight matrix according to the dynamic adjustment strategy.

[0024] In one implementation, the dynamic adjustment strategy includes:

[0025] Each weight value in the dynamic weight matrix is ​​compared with the preset high and low adjustment thresholds;

[0026] If the weight value exceeds the high adjustment threshold, the high adjustment mechanism is activated; if the weight value is below the low adjustment threshold, the low adjustment mechanism is activated; if the weight value is between the high and low adjustment thresholds, the default adjustment mechanism is activated.

[0027] The high adjustment mechanism works by collecting the average information gain value, calculating the positive adjustment ratio, inputting the feature vectors of each modality into the current classification model, evaluating the classification information of the feature vectors extracted from each modality, calculating the information gain of the feature dimensions of each modality as the average information gain value, multiplying the positive adjustment ratio by the weight value exceeding the high adjustment threshold, and inputting the calculation result into the dynamic weight matrix to achieve dynamic adjustment of the dynamic weight matrix.

[0028] The low adjustment mechanism collects the average mutual information value through mutual information analysis and calculates the negative adjustment ratio. It extracts the output feature vector of each mode and performs mutual information analysis with the real label of the corresponding sample to obtain the average mutual information value. The negative adjustment ratio is multiplied by the weight value below the low adjustment threshold, and the calculation result is input into the dynamic weight matrix to realize the dynamic adjustment of the dynamic weight matrix.

[0029] In one implementation scheme, a support vector machine is used to classify and train the weighted feature matrix. Specifically, the weighted feature matrix is ​​mapped from a low-dimensional space to a high-dimensional space using a kernel function mapping method. A radial basis function is selected as the kernel function, and the parameters of the kernel function are determined by cross-validation. The gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, thus completing the training of the support vector machine.

[0030] In one implementation, the word embedding model employs a bidirectional long short-term memory neural network.

[0031] In one implementation, the dynamic weight allocation algorithm employs a genetic algorithm, defining a fitness function to evaluate the effectiveness of weight allocation. The fitness function uses classification accuracy as the core indicator, and combines computational complexity for comprehensive evaluation. The weight allocation parameters are continuously optimized using crossover and mutation operations of the genetic algorithm until the fitness function reaches its optimal value.

[0032] In one implementation, the method further includes:

[0033] The confidence level of each category is calculated using the probability distribution of the image classification results;

[0034] If the confidence level of the classification category is lower than the preset confidence threshold, the hyperplane parameters of the support vector machine are adjusted and the weighted feature matrix is ​​trained again until the confidence level of the image classification result reaches the confidence threshold.

[0035] In a second aspect, the present invention provides a few-sample image classification system based on multimodal data fusion, the system comprising:

[0036] The data acquisition module is used to acquire multimodal data of the target power equipment and extract image data, text data and audio data from the multimodal data;

[0037] The image feature extraction module is used to extract local and global features of image data using a convolutional neural network, and then fuse the local and global features to obtain the fused features of the image data.

[0038] The text feature extraction module is used to extract semantic features from text data using a word embedding model;

[0039] The audio feature extraction module is used to extract the spectral features of audio data using short-time Fourier transform;

[0040] The feature matrix construction module is used to construct a multimodal feature matrix based on fused features, semantic features, and spectral features;

[0041] The weight allocation module is used to assign weights to the feature vectors of each mode in the multimodal feature matrix using a dynamic weight allocation algorithm, so as to obtain a weighted feature matrix.

[0042] The image classification module is used to train a weighted feature matrix using a support vector machine to generate image classification results for the target power equipment.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] This invention integrates multiple modal data, including images, text, and audio, and extracts features from each modality. Image data is processed using a convolutional neural network to extract multi-level local and global features, text data is processed using a word embedding model to extract semantic information, and audio data is processed using a short-time Fourier transform to extract spectral features. The multiple modal features are fused to construct a unified feature matrix, and an adaptive evaluation and weighting process is performed based on a dynamic weight allocation algorithm to assess and weight the importance of each modality feature. In the classification stage, a support vector machine is used to train and classify the fused features, and a confidence scoring mechanism and confusion matrix analysis are introduced to evaluate and optimize the classification results. This invention can fully exploit the discriminative information of each modality in small sample scenarios, make up for the expression defects of a single modality when there are insufficient samples, effectively improve the overall accuracy, stability, and generalization ability of the classification system, and significantly enhance the application effect of the system in complex environments. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 A flowchart illustrating a few-sample image classification method based on multimodal data fusion provided in this embodiment of the invention.

[0047] Figure 2 A schematic diagram of a small sample image classification system based on multimodal data fusion provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0049] It should be noted that the terms "comprising" or "may include" used in the various embodiments of this application indicate the presence of the claimed function, operation, or element, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in the various embodiments of this application, the terms "comprising," "having," and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or adding one or more combinations of the foregoing.

[0050] It should be understood that terms such as "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0051] Figure 1 A flowchart of a few-sample image classification method based on multimodal data fusion provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0052] S101, acquire multimodal data of the target power equipment, and extract image data, text data and audio data from the multimodal data.

[0053] In this embodiment, power equipment refers to all equipment and devices used for production, conversion, control, protection, measurement, and connection in the entire power system, including power generation, transmission and distribution, and power consumption. Examples include power generation equipment, transmission and distribution equipment, and power consumption equipment. After acquiring the multimodal data of the target power equipment, image data, text data, and audio data can be extracted from the multimodal data. Specifically, a high-resolution camera is installed at a fixed position directly in front of the target power equipment to capture image data, a microphone array is arranged around the target power equipment to capture omnidirectional audio data, and text data can be input through a text input device.

[0054] The acquired data is cleaned and formatted as follows: for image data, histogram equalization is used to enhance image contrast; for text data, stop word filtering and stemming are used to remove redundant information; and for audio data, noise suppression algorithms are used to reduce background noise interference.

[0055] S102, using a convolutional neural network to extract local and global features of image data, and then fusing the local and global features to obtain the fused features of the image data.

[0056] In this embodiment, the shallow structure of a convolutional neural network is used to extract local features of image data; wherein, the shallow structure includes three convolutional layers, each followed by a max pooling layer, for extracting local detail information of the edges and textures of the target power equipment;

[0057] The deep structure of a convolutional neural network is used to extract global features from image data; the deep structure includes five convolutional layers and two fully connected layers to capture the shape distribution and spatial distribution information of the target power equipment.

[0058] Finally, the extracted local and global features are fused to obtain the fused features of the image data. Here, multi-scale feature fusion provided by existing technology can be used, or a fusion method based on attention mechanism can be used. This embodiment will not describe the fusion process in detail.

[0059] S103 utilizes a word embedding model to extract semantic features from text data.

[0060] In this embodiment, the word embedding model uses a bidirectional long short-term memory neural network (BiLSTM). The BiLSTM network extracts semantic features from both forward and backward directions and generates semantic vectors of fixed length.

[0061] S104 uses short-time Fourier transform to extract the spectral features of audio data.

[0062] In this embodiment, a short-time Fourier transform combined with a sliding window mechanism is used to perform frame-by-frame processing on the audio data. The window size is set to 25 milliseconds and the step size is set to 10 milliseconds. The features of each frame are extracted and spliced ​​together to form spectral features as the feature representation of the audio data.

[0063] S105, construct a multimodal feature matrix based on fusion features, semantic features and spectral features.

[0064] In this embodiment, conventional techniques for constructing matrices from various different features will not be described in detail.

[0065] S106. A dynamic weight allocation algorithm is used to assign weights to the eigenvectors of each mode in the multimodal feature matrix to obtain a weighted feature matrix.

[0066] In this embodiment, for the multimodal feature matrix, a dynamic weight allocation algorithm is used to assign weights to the feature vectors in the multimodal feature matrix, as follows:

[0067] First, an initial weight matrix is ​​constructed based on the dimension and distribution characteristics of the feature vectors of each mode in the multimodal feature matrix; where the initial weight value of the initial weight matrix is ​​1 / N, and N is the number of modes;

[0068] Secondly, the correlation between the feature vectors of each mode in the multimodal feature matrix is ​​calculated using the cosine similarity formula to generate a correlation matrix.

[0069] Furthermore, the initial weight values ​​of the initial weight matrix are adjusted according to the correlation matrix; the adjustment process is as follows: feature vectors with high correlation are assigned the first weight, feature vectors with low correlation are assigned the second weight, and the first weight is greater than the second weight.

[0070] Then, the initial weight matrix after adjusting the initial weight values ​​is normalized so that the sum of the weight values ​​in each column of the initial weight matrix after adjusting the initial weight values ​​is 1, thus obtaining the dynamic weight matrix;

[0071] Finally, the dynamic weight matrix is ​​multiplied by the multimodal feature matrix to obtain the weighted feature matrix.

[0072] Specifically, the method also includes: setting a dynamic adjustment strategy for each weight value in the dynamic weight matrix, and performing dynamic adjustment of each weight value in the dynamic weight matrix according to the dynamic adjustment strategy.

[0073] The dynamic adjustment strategy for each weight in the weighted feature matrix is ​​as follows:

[0074] Each weight value in the dynamic weight matrix is ​​compared with the preset high and low adjustment thresholds;

[0075] If the weight value exceeds the high adjustment threshold, the high adjustment mechanism is activated; if the weight value is below the low adjustment threshold, the low adjustment mechanism is activated; if the weight value is between the high and low adjustment thresholds, the default adjustment mechanism is activated.

[0076] The high adjustment mechanism works by collecting the average information gain value, calculating the positive adjustment ratio, inputting the feature vectors of each modality into the current classification model, evaluating the classification information of the feature vectors extracted from each modality, calculating the information gain of the feature dimensions of each modality as the average information gain value, multiplying the positive adjustment ratio by the weight value exceeding the high adjustment threshold, and inputting the calculation result into the dynamic weight matrix to achieve dynamic adjustment of the dynamic weight matrix.

[0077] The low adjustment mechanism collects the average mutual information value through mutual information analysis and calculates the negative adjustment ratio. It extracts the output feature vector of each mode and performs mutual information analysis with the real label of the corresponding sample to obtain the average mutual information value. The negative adjustment ratio is multiplied by the weight value below the low adjustment threshold, and the calculation result is input into the dynamic weight matrix to realize the dynamic adjustment of the dynamic weight matrix.

[0078] It should be noted that the high and low adjustment thresholds are determined based on the discriminative distribution of each modality feature in the training set and the historical classification accuracy trend, which will not be elaborated here.

[0079] The default adjustment strategy involves fine-tuning the current weights using a linear smooth update mechanism when the corresponding weights of the feature vectors fall between the high and low adjustment thresholds. The specific fine-tuning formula is as follows: In the formula, This represents the current weight of the eigenvector of the ith modality in the t-th iteration. The smoothing coefficient is adjusted by the experimenters based on the model's convergence. This represents the gradient feedback of the current modality feature vector in the classification loss function. Here, i is the index of the current modality feature, i.e., the i-th modality input, and t is the iteration number of the current dynamic weight update.

[0080] The high adjustment mechanism calculates the positive adjustment ratio by collecting the average information gain value of the current classification model.

[0081] The low adjustment mechanism calculates the negative adjustment ratio by collecting average mutual information values ​​through mutual information analysis.

[0082] The modal features are input into the current classification model. The feature vectors extracted from each modality are evaluated for classification information. The information gain of the feature dimensions of each modality is calculated as the average information gain value.

[0083] The current classification model was designed by our team based on the specific implementation situation, and will not be elaborated here.

[0084] Specifically, the formula for calculating the average information gain is as follows: In the formula, Let be the average information gain value of the i-th mode. The dimension of this modality feature. For the j-th feature of the i-th mode, The entropy of the sample labels, The conditional entropy is given the features.

[0085] Furthermore, the average information gain value measures the contribution of a certain modality feature to the classification during the training process. The greater the contribution, the greater the average information gain value, which indicates that the modality has a stronger influence on the classification decision in the current training stage and is given a higher adjustment ratio in the dynamic weight adjustment.

[0086] By extracting the output feature vector of each modality and performing mutual information analysis with the real label of the corresponding sample, the average mutual information value is obtained.

[0087] Specifically, the formula for calculating the average mutual information value is as follows: In the formula, Let be the average mutual information value of the i-th mode. Let i be the feature vector of the i-th mode in the k-th sample. Let be the label of the k-th sample. This refers to the mutual information between modal features and labels.

[0088] The true label of the corresponding sample is the target category label labeled in the supervised training sample, which is usually represented by one-hot vector or integer encoding. Mutual information analysis is to quantify the statistical dependency between the joint probability distribution and the marginal probability distribution of the modal feature vector and the label variable by evaluating the degree of difference between them.

[0089] Furthermore, the average mutual information value measures the correlation between the current modality features and the true classification result. The smaller the mutual information, the weaker the help in classification, and the lower the adjustment ratio is given in dynamic weight adjustment.

[0090] The average information gain and average mutual information are standardized so that the values ​​remain between 0 and 1.

[0091] It should be noted that the standardization methods include, but are not limited to, standard linear transformation based on interval scaling, statistical Z-Score standardization, or normalization based on nonlinear mapping functions. The application methods of standardization will not be elaborated here.

[0092] The difference between the average information gain value and the high adjustment threshold is normalized to obtain the positive adjustment ratio. The positive adjustment ratio is then multiplied with the high adjustment threshold to obtain the weight adjustment result of the high adjustment threshold, which is then input into the dynamic weight matrix.

[0093] Specifically, the formula for calculating the difference normalization is: In the formula, It is a positive adjustment ratio. This is the average information gain value. To adjust the threshold, To adjust the sensitivity factor;

[0094] The average mutual information value is used to calculate the negative adjustment ratio by applying the low adjustment threshold. The negative adjustment ratio is then multiplied by the lower-low adjustment threshold to obtain the weighted adjustment result below the low adjustment threshold, which is then input into the weighted feature matrix.

[0095] Specifically, the formula for calculating the low adjustment mechanism is as follows: In the formula, It is a negative adjustment ratio. The average mutual information value, To adjust the threshold to a low value, To adjust the sensitivity factor;

[0096] S107 uses a support vector machine to train the weighted feature matrix for classification, generating image classification results for the target power equipment.

[0097] In this embodiment, a support vector machine is used to perform classification training on the weighted feature matrix. Specifically, during the training process, a kernel function mapping method is used to map the weighted feature matrix from a low-dimensional space to a high-dimensional space. A radial basis function is selected as the kernel function, and the parameters of the kernel function are determined by cross-validation. The gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, thus completing the training of the support vector machine.

[0098] Specifically, a kernel function mapping method is used to map the weighted feature matrix from a low-dimensional space to a high-dimensional space to enhance the linear separability of the feature matrix. The radial basis function is selected as the kernel function, and its parameters are determined by cross-validation. During training, the gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, and finally the classification result is generated and output.

[0099] The above embodiments detail the principle of the few-sample image classification method based on multimodal data fusion. By integrating information from multimodal data such as images, text, and audio, the limitations of single-modal data in small-sample scenarios are solved. At the same time, the utilization rate of multimodal features is optimized through a dynamic weight allocation algorithm, thereby improving classification performance and computational efficiency.

[0100] Furthermore, this invention introduces a confidence assessment mechanism to perform reliability analysis on the classification results, including:

[0101] The confidence level of each category is calculated using the probability distribution of the image classification results;

[0102] If the confidence level of the classification category is lower than the preset confidence threshold, the hyperplane parameters of the support vector machine are adjusted and the weighted feature matrix is ​​trained again until the confidence level of the image classification result reaches the confidence threshold.

[0103] Specifically, first, the confidence score for each category is calculated using the probability distribution of the classification results. The confidence score formula is P_max / (P_sum-P_max); where P_max is the maximum value in the probability distribution, and P_sum is the sum of the probabilities of all categories.

[0104] Secondly, a confidence threshold is set based on the distribution characteristics of the training data. When the confidence of the classification result is lower than the threshold, a reclassification mechanism is triggered.

[0105] Finally, the feature matrices corresponding to the classification results with confidence scores below the threshold are re-input into the support vector machine. The hyperplane parameters of the support vector machine are adjusted, and classification training is performed again until the confidence scores of the classification results reach the threshold requirement.

[0106] In the above process, if the confidence level of the classification result does not meet the requirement, a reclassification mechanism is triggered. At this point, the feature matrix corresponding to the classification result with a confidence level below the threshold is re-inputted, the hyperplane parameters of the support vector machine are adjusted, and classification training is performed again until the confidence level of the classification result reaches the threshold requirement. This mechanism ensures the reliability of the classification results while avoiding misclassification problems caused by insufficient sample data.

[0107] In practical applications, this technology can be deployed in intelligent monitoring scenarios, such as real-time classification of human behavior within a specific area. High-resolution cameras capture image data of people within the area, microphone arrays collect ambient sound information, and text input devices record the identity tags or behavioral descriptions of relevant personnel. This multimodal data is preprocessed and then fed into subsequent modules. Behavioral features of the target personnel are extracted from the image data, combined with identity tags from the text data and ambient sound features from the audio data, and a weighted feature matrix is ​​generated using a dynamic weight allocation algorithm. A support vector machine is then used to train the weighted feature matrix for classification, ultimately outputting the classification result of the human behavior and evaluating its confidence level. If the confidence level does not meet the requirements, a reclassification mechanism is triggered until the conditions are met.

[0108] In some embodiments, the dynamic weight allocation algorithm employs a genetic algorithm, defining a fitness function to evaluate the effectiveness of weight allocation. The fitness function uses classification accuracy as the core indicator, and combines computational complexity for comprehensive evaluation. The weight allocation parameters are continuously optimized using the crossover and mutation operations of the genetic algorithm until the fitness function reaches its optimal value.

[0109] Specifically, to optimize the performance of the dynamic weight allocation algorithm, a genetic algorithm is used to optimize the algorithm parameters. A fitness function is defined with classification accuracy as the core indicator, and computational complexity is comprehensively evaluated. The weight allocation parameters are continuously optimized using crossover and mutation operations of the genetic algorithm until the fitness function reaches its optimal value. Simultaneously, after classification training, a confusion matrix is ​​introduced to evaluate the classification results. The confusion matrix includes four indicators: true positive, false positive, true negative, and false negative. These indicators are used to calculate the precision, recall, and F1 score of the classification model to further analyze its classification performance.

[0110] In summary, the embodiments described above demonstrate that this invention effectively integrates information from multimodal data such as images, text, and audio, overcoming the limitations of single-modal data in small sample scenarios. Furthermore, the dynamic weight allocation algorithm significantly improves classification performance and computational efficiency by assigning weights to the feature vectors in the multimodal feature matrix. For example, in intelligent monitoring scenarios, the system can classify the behavior of people within a specific area in real time and generate reliable classification results by combining identity tags and environmental sound features, thereby providing technical support for practical applications.

[0111] Figure 2 A schematic diagram of a module for a few-sample image classification system based on multimodal data fusion provided in an embodiment of the present invention is shown below. Figure 2 As shown, the system includes:

[0112] The data acquisition module 210 is used to acquire multimodal data of the target power equipment and extract image data, text data and audio data from the multimodal data;

[0113] Image feature extraction module 220 is used to extract local and global features of image data using a convolutional neural network;

[0114] The text feature extraction module 230 is used to extract semantic features of text data using a word embedding model;

[0115] The audio feature extraction module 240 is used to extract the spectral features of audio data using short-time Fourier transform;

[0116] The feature matrix construction module 250 is used to construct a multimodal feature matrix based on fused features, semantic features, and spectral features;

[0117] The weight allocation module 260 is used to assign weights to the feature vectors of each mode in the multimodal feature matrix using a dynamic weight allocation algorithm to obtain a weighted feature matrix.

[0118] The image classification module 270 is used to perform classification training on the weighted feature matrix using a support vector machine to generate image classification results for the target power equipment.

[0119] In the small-sample image classification system based on multimodal data fusion provided in this embodiment of the invention, multiple modal data such as images, text, and audio are integrated, and their respective features are extracted. Image data is processed by a convolutional neural network to extract multi-level local and global features, text data is processed by a word embedding model to extract semantic information, and audio data is processed by a short-time Fourier transform to extract spectral features. The features of multiple modalities are fused to construct a unified feature matrix, and the importance of each modal feature is adaptively evaluated and weighted based on a dynamic weight allocation algorithm. In the classification stage, a support vector machine is used to train and classify the fused features, and a confidence scoring mechanism and confusion matrix analysis are introduced to evaluate and optimize the classification results. This system can fully explore the discriminative information of each modality in small-sample scenarios, make up for the expression defects of a single modality when there are insufficient samples, effectively improve the overall accuracy, stability, and generalization ability of the classification system, and significantly enhance the application effect of the system in complex environments.

[0120] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A small sample image classification method based on multi-modal data fusion, characterized in that, The method comprises: acquiring multi-modal data of a target power equipment, and extracting image data, text data and audio data from the multi-modal data; extracting local features and global features of the image data using a convolutional neural network, fusing the local features and the global features to obtain fused features of the image data; extracting semantic features of the text data using a word embedding model; extracting spectral features of the audio data using a short-time Fourier transform; constructing a multi-modal feature matrix according to the fused features, the semantic features and the spectral features; assigning weights to feature vectors of each modality in the multi-modal feature matrix using a dynamic weight distribution algorithm to obtain a weighted feature matrix; wherein the dynamic weight distribution algorithm comprises: constructing an initial weight matrix according to dimensions and distribution characteristics of the feature vectors of each modality in the multi-modal feature matrix; wherein the initial weight value of the initial weight matrix is 1 / N, and N is the number of modalities; calculating correlations between the feature vectors of each modality in the multi-modal feature matrix using a cosine similarity formula to generate a correlation matrix; adjusting the initial weight value of the initial weight matrix according to the correlation matrix; wherein the adjustment process is: assigning a first weight to a feature vector with high correlation, assigning a second weight to a feature vector with low correlation, and the first weight is greater than the second weight; normalizing the initial weight matrix after the initial weight value is adjusted, so that the weight value of each column in the initial weight matrix after the initial weight value is adjusted is 1, to obtain a dynamic weight matrix; multiplying the dynamic weight matrix and the multi-modal feature matrix to obtain the weighted feature matrix; performing classification training on the weighted feature matrix using a support vector machine to generate an image classification result of the target power equipment. The method further comprises: setting a dynamic adjustment strategy for each weight value in the dynamic weight matrix, and completing dynamic adjustment of each weight value in the dynamic weight matrix according to the dynamic adjustment strategy; wherein the dynamic adjustment strategy comprises: comparing each weight value in the dynamic weight matrix with preset high and low adjustment thresholds; if the weight value exceeds the high adjustment threshold, a high adjustment mechanism is enabled; if the weight value is lower than the low adjustment threshold, a low adjustment mechanism is enabled; and if the weight value is between the high and low adjustment thresholds, a default adjustment mechanism is enabled; the high adjustment mechanism is to collect an average information gain value, calculate a positive adjustment ratio, input the feature vectors of each modality into the current classification model, perform classification information division and evaluation on the dimensions of the feature vectors extracted for each modality, and calculate the information gain of the feature dimensions of each modality as the average information gain value; the positive adjustment ratio is multiplied by the weight value exceeding the high adjustment threshold to obtain a calculation result, which is input into the dynamic weight matrix to realize dynamic adjustment of the dynamic weight matrix; the low adjustment mechanism is to collect an average mutual information value through mutual information analysis, and calculate a negative adjustment ratio; the output feature vectors of each modality are extracted, mutual information analysis is performed on the feature vectors and the true labels of the corresponding samples to obtain the average mutual information value; the negative adjustment ratio is multiplied by the weight value lower than the low adjustment threshold to obtain a calculation result, which is input into the dynamic weight matrix to realize dynamic adjustment of the dynamic weight matrix.

2. The small sample image classification method based on multi-modal data fusion according to claim 1, characterized in that, The local features and global features of the image data are extracted by using the convolutional neural network, including: The local features of the image data are extracted by using a shallow structure of the convolutional neural network; the shallow structure includes three convolutional layers, each followed by a max-pooling layer, which is used to extract local detailed information of edges and textures of the target power equipment; The global features of the image data are extracted by using a deep structure of the convolutional neural network; the deep structure includes five convolutional layers and two fully connected layers, which are used to capture shape distribution information and spatial distribution information of the target power equipment.

3. The small sample image classification method based on multi-modal data fusion according to claim 1, characterized in that, The weighted feature matrix is classified and trained by using the support vector machine, specifically: a kernel function mapping method is used to map the weighted feature matrix from a low-dimensional space to a high-dimensional space, a radial basis function is selected as the kernel function, parameters of the kernel function are determined by a cross-validation method, a gradient descent algorithm is used to optimize hyperplane parameters of the support vector machine until a loss function converges, and training of the support vector machine is completed.

4. The small sample image classification method based on multi-modal data fusion according to claim 1, characterized in that, The word embedding model adopts a bidirectional long short-term memory neural network.

5. The small sample image classification method based on multi-modal data fusion according to claim 1, characterized in that, The dynamic weight distribution algorithm adopts a genetic algorithm, a fitness function is defined to evaluate the effect of weight distribution, the fitness function takes classification accuracy as a core index and comprehensively evaluates in combination with a calculation complexity, a crossover and mutation operation of the genetic algorithm is used to continuously optimize weight distribution parameters until the fitness function reaches an optimal value.

6. The small sample image classification method based on multi-modal data fusion according to claim 1, characterized in that, The method further comprises: A confidence of each classification category is calculated by using a probability distribution of the image classification result; If the confidence of the classification category is lower than a preset confidence threshold, the hyperplane parameters of the support vector machine are adjusted, and the weighted feature matrix is classified and trained again until the confidence of the image classification result reaches the confidence threshold.

7. A small sample image classification system based on multi-modal data fusion, configured to perform a small sample image classification method based on multi-modal data fusion according to any one of claims 1 to 6. The system comprises: The data acquisition module is configured to acquire multi-modal data of the target power equipment, and extract image data, text data and audio data from the multi-modal data. The image feature extraction module is configured to extract local features and global features of the image data by using a convolutional neural network, fuse the local features and the global features, and obtain fused features of the image data. The text feature extraction module is configured to extract semantic features of the text data by using a word embedding model. The audio feature extraction module is configured to extract spectral features of the audio data by using a short-time Fourier transform. The feature matrix construction module is configured to construct a multi-modal feature matrix according to the fused features, the semantic features and the spectral features. The weight distribution module is configured to assign weights to feature vectors of each modality in the multi-modal feature matrix by using a dynamic weight distribution algorithm, and obtain a weighted feature matrix. The image classification module is configured to perform classification training on the weighted feature matrix by using a support vector machine, and generate an image classification result of the target power equipment.

Images