Small sample image classification method and system based on multi-modal data fusion

Abstract

The invention discloses a small sample image classification method and system based on multi-modal data fusion, and relates to the technical field of computer vision, and the key points of the technical scheme are as follows: obtaining multi-modal data of target power equipment, and extracting image data, text data and audio data from the multi-modal data; extracting local features and global features of the image data by using a convolutional neural network, and fusing the local features and the global features to obtain fused features of the image data; extracting semantic features of the text data by using a word embedding model; extracting spectrum features of the audio data by using short-time Fourier transform; constructing a multi-modal feature matrix according to the fusion features, the semantic features and the spectrum features; a dynamic weight distribution algorithm is adopted to endow the feature vector of each modal in the multi-modal feature matrix with a weight, and a weighted feature matrix is obtained; and performing classification training on the weighted feature matrix by using a support vector machine to generate an image classification result of the target power equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, more particularly, it relates to a small sample image classification method and system based on multi-modal data fusion. BACKGROUND

[0002] Image classification technology refers to a technology and method in the field of artificial intelligence and computer vision, which realizes the automatic identification and classification of target objects by analyzing and processing image data. Multi-modal data fusion applied to image classification technology can integrate information of different modalities (such as images, texts, audios, etc.), improve the performance of the classification model, and is particularly important in small sample scenarios.

[0003] The prior art has the following disadvantages: At present, the classification method mainly focuses on the use of image features, and does not involve the fusion analysis of multi-modal data. In the small sample scenario, the classification effect is affected due to insufficient information, and it is limited to single modal data processing, and the potential value of multi-modal information cannot be fully tapped, resulting in limited precision of the classification result. SUMMARY

[0004] The purpose of the present application is to provide a small sample image classification method and system based on multi-modal data fusion, which solves the problem of insufficient use of multi-modal data in the prior art in small sample scenarios, resulting in insufficient precision of image classification results.

[0005] The above technical purpose of the present application is achieved by the following technical scheme: In a first aspect, the present application provides a small sample image classification method based on multi-modal data fusion, the method comprising: Obtaining multi-modal data of a target power equipment, and extracting image data, text data and audio data from the multi-modal data; Extracting local features and global features of the image data using a convolutional neural network, fusing the local features and the global features to obtain fusion features of the image data; Extracting semantic features of the text data using a word embedding model; Extracting frequency spectrum features of the audio data using a short-time Fourier transform; Constructing a multi-modal feature matrix according to the fusion features, the semantic features and the frequency spectrum features; Assigning weights to feature vectors of each modality in the multi-modal feature matrix using a dynamic weight distribution algorithm to obtain a weighted feature matrix; Classifying and training the weighted feature matrix using a support vector machine to generate an image classification result of the target power equipment.

[0006] In one implementation, the convolutional neural network is used to extract local features and global features of the image data, comprising: extracting local features of the image data by using a shallow structure of the convolutional neural network; wherein the shallow structure comprises three convolutional layers, each followed by a max-pooling layer, for extracting local detailed information of edges and textures of the target power equipment; extracting global features of the image data by using a deep structure of the convolutional neural network; wherein the deep structure comprises five convolutional layers and two fully connected layers, for capturing shape distribution information and spatial distribution information of the target power equipment.

[0007] In an implementation scheme, a dynamic weight assignment algorithm is used to assign weights to the feature vectors of each modality in the multi-modal feature matrix to obtain a weighted feature matrix, comprising: constructing an initial weight matrix according to the dimensions and distribution characteristics of the feature vectors of each modality in the multi-modal feature matrix; wherein the initial weight values of the initial weight matrix are 1 / N, and N is the number of modalities; calculating the correlation between the feature vectors of each modality in the multi-modal feature matrix by using a cosine similarity formula to generate a correlation matrix; adjusting the initial weight values of the initial weight matrix according to the correlation matrix; wherein the adjustment process is: the feature vectors with high correlation are assigned a first weight, and the feature vectors with low correlation are assigned a second weight, and the first weight is greater than the second weight; normalizing the initial weight matrix after adjusting the initial weight values, so that the weight values in each column of the initial weight matrix after adjusting the initial weight values are equal to 1, to obtain a dynamic weight matrix; multiplying the dynamic weight matrix with the multi-modal feature matrix to obtain a weighted feature matrix.

[0008] In an implementation scheme, the method further comprises: setting a dynamic adjustment strategy for each weight value in the dynamic weight matrix, and dynamically adjusting each weight value in the dynamic weight matrix according to the dynamic adjustment strategy.

[0009] In an implementation scheme, the dynamic adjustment strategy comprises: comparing each weight value in the dynamic weight matrix with a preset high adjustment threshold and a low adjustment threshold; if the weight value exceeds the high adjustment threshold, a high adjustment mechanism is enabled; if the weight value is lower than the low adjustment threshold, a low adjustment mechanism is enabled; if the weight value is between the high adjustment threshold and the low adjustment threshold, a default adjustment mechanism is enabled; The high adjustment mechanism is to collect average information gain value, calculate positive adjustment ratio, input feature vectors of each modality into the current classification model, evaluate classification information division of the feature vector dimension extracted by each modality, calculate information gain of the feature dimension of each modality as the average information gain value, multiply the positive adjustment ratio and the weight value exceeding the high adjustment threshold, and input the calculation result into the dynamic weight matrix to realize dynamic adjustment of the dynamic weight matrix. The low adjustment mechanism is to collect average mutual information value through mutual information analysis, calculate negative adjustment ratio, extract the output feature vector of each modality, and perform mutual information analysis on the real label of the corresponding sample to obtain the average mutual information value, multiply the negative adjustment ratio and the weight value lower than the low adjustment threshold, and input the calculation result into the dynamic weight matrix to realize dynamic adjustment of the dynamic weight matrix.

[0010] In an implementation scheme, the weighted feature matrix is classified and trained by using a support vector machine, specifically: the kernel function mapping method is used to map the weighted feature matrix from a low-dimensional space to a high-dimensional space, the radial basis function is selected as the kernel function, the parameters of the kernel function are determined by the cross-validation method, the gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, and the training of the support vector machine is completed.

[0011] In an implementation scheme, the word embedding model adopts a bidirectional long short-term memory neural network.

[0012] In an implementation scheme, the dynamic weight distribution algorithm adopts a genetic algorithm, a fitness function is defined to evaluate the effect of weight distribution, the fitness function takes classification accuracy as a core index, and comprehensively evaluates in combination with calculation complexity, the weight distribution parameters are continuously optimized by using the crossover and mutation operations of the genetic algorithm until the fitness function reaches an optimal value.

[0013] In an implementation scheme, the method further comprises: The probability distribution of the image classification result is used to calculate the confidence of each classification category. If the confidence of the classification category is lower than a preset confidence threshold, the hyperplane parameters of the support vector machine are adjusted, and the weighted feature matrix is classified and trained again until the confidence of the image classification result reaches the confidence threshold.

[0014] In the second aspect of the present application, a small sample image classification system based on multi-modal data fusion is provided, and the system comprises: A data acquisition module is configured to acquire multi-modal data of a target power equipment, and extract image data, text data and audio data from the multi-modal data. An image feature extraction module is configured to extract local features and global features of image data by using a convolutional neural network, and fuse the local features and the global features to obtain fused features of the image data. A text feature extraction module is configured to extract semantic features of text data by using a word embedding model. An audio feature extraction module is configured to extract spectral features of audio data by using a short-time Fourier transform. A feature matrix construction module is configured to construct a multi-modal feature matrix according to the fused features, the semantic features and the spectral features. A weight distribution module is configured to assign weights to feature vectors of each modality in the multi-modal feature matrix by using a dynamic weight distribution algorithm to obtain a weighted feature matrix. An image classification module is configured to perform classification training on the weighted feature matrix by using a support vector machine to generate an image classification result of the target power equipment.

[0015] Compared with the prior art, the present application has the following beneficial effects: The present application integrates various modal data such as images, texts and audios, extracts respective features, extracts multi-level local and global features of image data by using a convolutional neural network, extracts semantic information of text data by using a word embedding model, extracts spectral features of audio data by using a short-time Fourier transform, fuses various modal features to construct a unified feature matrix, adaptively evaluates and weights the importance of each modal feature based on a dynamic weight distribution algorithm, uses a support vector machine to train and classify the fused features in the classification stage, introduces a confidence score mechanism and a confusion matrix analysis to evaluate and optimize the classification result, can fully mine discriminative information of each modal data in a small sample scenario, makes up for the expression defects of a single modality in a sample deficiency situation, effectively improves the overall accuracy, stability and generalization ability of the classification system, and significantly enhances the application effect of the system in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings, which are included to provide a further understanding of the embodiments of the application and constitute a part of this application, illustrate embodiments of the application and are used to explain the embodiments of the application. In the drawings: Figure 1 A method flowchart of a small sample image classification method based on multi-modal data fusion provided by the embodiments of the present application; Figure 2 A module schematic diagram of a small sample image classification system based on multi-modal data fusion provided by the embodiments of the present application. DETAILED DESCRIPTION

[0017] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application with reference to the embodiments and drawings, and the schematic embodiments of the present application and the description thereof are only used for explaining the present application, but not limiting the present application.

[0018] It is noted that the term "include" or "may include" used in various embodiments of the present application indicates the existence of the claimed function, operation or element, and does not limit the addition of one or more functions, operations or elements. In addition, as used in various embodiments of the present application, the terms "include", "have" and their synonyms only mean to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing, and should not be understood as first excluding the existence or addition of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, elements, components or combinations of the foregoing.

[0019] It is understood that terms such as "first", "second" are only used for description purposes, and should not be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0020] Figure 1 The method flowchart of the small sample image classification method based on multi-modal data fusion provided by the embodiments of the present application is shown in Figure 1 As shown in the figure, the method comprises: S101, obtaining multi-modal data of a target power equipment, and extracting image data, text data and audio data from the multi-modal data.

[0021] In the present embodiment, the power equipment refers to all equipment and devices used for production, conversion, control, protection, measurement and connection in the entire power system of power generation, power transmission, power distribution and power consumption, such as power generation equipment, power transmission and distribution equipment, power consumption equipment, etc. After obtaining the multi-modal data of the target power equipment, the image data, text data and audio data can be extracted from the multi-modal data. Among them, the high-resolution camera is installed in the fixed position in front of the target power equipment to capture image data, the microphone array is arranged around the target power equipment to capture omnidirectional audio data, and the text data can be input through the text input device.

[0022] The obtained data is cleaned and formatted, for example, histogram equalization is used to enhance the contrast of image data, stop word filtering and stem extraction are used to remove redundant information from text data, and a noise suppression algorithm is used to reduce background noise interference from audio data.

[0023] In S102, local features and global features of the image data are extracted using a convolutional neural network, and the local features and the global features are fused to obtain fused features of the image data.

[0024] In this embodiment, the local features of the image data are extracted using a shallow structure of the convolutional neural network, wherein the shallow structure includes three convolutional layers, each followed by a max-pooling layer, for extracting local detailed information of edges and textures of the target power equipment. The global features of the image data are extracted using a deep structure of the convolutional neural network, wherein the deep structure includes five convolutional layers and two fully connected layers, for capturing shape distribution information and spatial distribution information of the target power equipment.

[0025] Finally, the extracted local features and global features are fused to obtain fused features of the image data, which can be fused by using a multi-scale feature fusion provided by the prior art or by using an attention mechanism-based fusion manner, and the fusion process is not described in detail in this embodiment.

[0026] In S103, semantic features of the text data are extracted using a word embedding model.

[0027] In this embodiment, the word embedding model is a bidirectional long short-term memory neural network, which extracts semantic features from both forward and backward directions and generates fixed-length semantic vectors.

[0028] In S104, spectral features of the audio data are extracted using a short-time Fourier transform.

[0029] In this embodiment, the short-time Fourier transform is combined with a sliding window mechanism to frame the audio data, the window size is set to 25 milliseconds, and the step size is set to 10 milliseconds, the features of each frame are extracted and spliced into spectral features as the feature representation of the audio data.

[0030] In S105, a multi-modal feature matrix is constructed according to the fused features, the semantic features, and the spectral features.

[0031] In this embodiment, the matrix is constructed by using conventional techniques of different features, and no redundant description is made in this embodiment.

[0032] In S106, a dynamic weight distribution algorithm is used to assign weights to the feature vectors of each modality in the multi-modal feature matrix to obtain a weighted feature matrix.

[0033] In the embodiment, the feature vectors in the multi-modal feature matrix are assigned weights by a dynamic weight distribution algorithm, and the implementation is as follows: First, an initial weight matrix is constructed according to the dimensions and distribution characteristics of the feature vectors of each modality in the multi-modal feature matrix; wherein the initial weight value of the initial weight matrix is 1 / N, and N is the number of modalities; Second, the correlation between the feature vectors of each modality in the multi-modal feature matrix is calculated using the cosine similarity formula to generate a correlation matrix; Third, the initial weight values of the initial weight matrix are adjusted according to the correlation matrix; wherein the adjustment process is as follows: the feature vectors with high correlation are assigned a first weight, and the feature vectors with low correlation are assigned a second weight, and the first weight is greater than the second weight; Then, the initial weight matrix with adjusted initial weight values is normalized so that the weight values in each column of the initial weight matrix with adjusted initial weight values sum to 1, and a dynamic weight matrix is obtained; Finally, the dynamic weight matrix is multiplied by the multi-modal feature matrix to obtain a weighted feature matrix.

[0034] Specifically, the method further includes setting a dynamic adjustment strategy for each weight value in the dynamic weight matrix, and dynamically adjusting each weight value in the dynamic weight matrix according to the dynamic adjustment strategy.

[0035] The dynamic adjustment strategy for each weight in the weighted feature matrix is as follows: Each weight value in the dynamic weight matrix is compared with a preset high adjustment threshold and a low adjustment threshold; If the weight value exceeds the high adjustment threshold, a high adjustment mechanism is enabled; if the weight value is below the low adjustment threshold, a low adjustment mechanism is enabled; if the weight value is between the high adjustment threshold and the low adjustment threshold, a default adjustment mechanism is enabled; The high adjustment mechanism is to collect average information gain values, calculate a positive adjustment ratio, input the feature vectors of each modality into the current classification model, perform classification information division evaluation on the dimensions of the feature vectors extracted for each modality, calculate the information gain of the feature dimensions of each modality as the average information gain value, multiply the positive adjustment ratio by the weight value that exceeds the high adjustment threshold, and input the calculation result into the dynamic weight matrix to realize dynamic adjustment of the dynamic weight matrix; The low adjustment mechanism is to collect average mutual information values by mutual information analysis and calculate a negative adjustment ratio; the average mutual information values are obtained by performing mutual information analysis on the output feature vectors of each modality and the true labels of the corresponding samples; the negative adjustment ratio is multiplied by the weight value that is below the low adjustment threshold, and the calculation result is input into the dynamic weight matrix to realize dynamic adjustment of the dynamic weight matrix.

[0036] It should be noted that the high adjustment threshold and the low adjustment threshold are determined according to the distribution of the discrimination degree of each modal feature in the training set and the historical classification accuracy change trend, which will not be repeated here.

[0037] Among them, the default adjustment strategy is to fine-tune the current weight by using a linear smoothing update mechanism when the corresponding weight of the feature vector is between the high adjustment threshold and the low adjustment threshold. The specific fine-tuning formula is: ; In the formula, represents the current weight of the i-th modal feature vector in the t-th iteration, is a smoothing coefficient, which is adjusted by the experimenters according to the model convergence, is the gradient feedback of the current modal feature vector in the classification loss function. Wherein, i is the number of the current modal feature, that is, the i-th modal input, t is the iteration round of the current dynamic weight update; The high adjustment mechanism is to collect the average information gain value through the current classification model, and calculate the positive adjustment proportion; The low adjustment mechanism is to collect the average mutual information value through mutual information analysis, and calculate the negative adjustment proportion; The modal feature is input into the current classification model, and the feature vector dimension extracted by each modal is classified and evaluated. The information gain of each modal feature dimension is calculated as the average information gain value; Wherein, the selection of the current classification model is designed by the experimenters according to the specific implementation, which will not be repeated here.

[0038] Specifically, the calculation formula of the average information gain value is: ; In the formula, is the average information gain value of the i-th modal, is the dimension number of the modal feature, is the j-th feature of the i-th modal, is the entropy of the sample label, is the conditional entropy under the given feature.

[0039] Further, the average information gain value measures the contribution of a modal feature to the class division in the training process. The greater the contribution, the greater the average information gain value, indicating that the modal has a stronger influence on the classification decision in the current training stage, and is given a higher adjustment proportion in the dynamic weight adjustment; The output feature vector of each modal is extracted, and mutual information analysis is performed with the real label of the corresponding sample to obtain the average mutual information value; Specifically, the calculation formula of the average mutual information value is: ; In the formula, is the average mutual information value of the i-th modal, is the feature vector of the i-th modality in the k-th sample, is the label of the k-th sample, is the mutual information between the modality feature and the label.

[0040] wherein the true label of the corresponding sample is the target class label labeled in the supervised training sample, usually represented by one-hot vector or integer coding, and the mutual information analysis is to evaluate the difference between the joint probability distribution and the marginal probability distribution between the modality feature vector and the label variable, and to quantify the statistical dependence relationship; Further, the average mutual information value is to measure the correlation between the current modality feature and the true classification result, and the smaller the mutual information is, the weaker the help to classification is, and a lower adjustment ratio is given in the dynamic weight adjustment; The average information gain value and the average mutual information value are standardized to keep the numerical interval between 0 and 1; It should be noted that the standardization methods include but are not limited to standard linear transformation based on interval scaling, Z-Score standardization method based on statistics, or normalization method based on nonlinear mapping function, and the application method of the standardization is not described here.

[0041] The average information gain value is subtracted from the high adjustment threshold value to obtain a positive adjustment ratio, and the positive adjustment ratio is multiplied by the high adjustment threshold value to obtain a high adjustment threshold weight adjustment result, which is input into the dynamic weight matrix; Specifically, the difference normalization calculation formula is: ; wherein, is the positive adjustment ratio, is the average information gain value, is the high adjustment threshold value, is the adjustment sensitivity factor; The average mutual information value is calculated by the low adjustment mechanism to obtain a negative adjustment ratio, and the negative adjustment ratio is multiplied by the low adjustment threshold value to obtain a low adjustment threshold weight adjustment result, which is input into the weighted feature matrix; Specifically, the low adjustment mechanism calculation formula is: ; wherein, is the negative adjustment ratio, is the average mutual information value, is the low adjustment threshold value, is the adjustment sensitivity factor; S107, using a support vector machine to perform classification training on the weighted feature matrix to generate an image classification result of the target power equipment.

[0042] In the embodiment, the weighted feature matrix is classified and trained by using a support vector machine, specifically, in the training process, the weighted feature matrix is mapped from a low-dimensional space to a high-dimensional space by using a kernel function mapping method, a radial basis function is selected as the kernel function, the parameters of the kernel function are determined by a cross-validation method, a gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, and the training of the support vector machine is completed.

[0043] Specifically, the weighted feature matrix is mapped from a low-dimensional space to a high-dimensional space by using a kernel function mapping method to enhance the linear separability of the feature matrix, a radial basis function is selected as the kernel function, and the parameters of the radial basis function are determined by a cross-validation method. In the training process, a gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, and finally, a classification result is generated and output.

[0044] The above embodiment details the principle of the small sample image classification method based on multi-modal data fusion, that is, by integrating the information of multi-modal data such as images, texts, and audios, the limitations of single modal data in a small sample scene are solved, and at the same time, the utilization rate of multi-modal features is optimized by a dynamic weight distribution algorithm, and the classification performance and calculation efficiency are improved.

[0045] In addition, the application also introduces a confidence evaluation mechanism to analyze the reliability of the classification result, including: The confidence of each classification category is calculated by using the probability distribution of the image classification result. If the confidence of the classification category is lower than a preset confidence threshold, the hyperplane parameters of the support vector machine are adjusted, and the weighted feature matrix is classified and trained again until the confidence of the image classification result reaches the confidence threshold.

[0046] Specifically, first, the confidence of each category is calculated by using the probability distribution of the classification result, and the confidence formula is P_max / (P_sum-P_max); wherein P_max is the maximum value in the probability distribution, and P_sum is the sum of the probabilities of all categories. Second, the confidence threshold is set according to the distribution characteristics of the training data, and the reclassification mechanism is triggered when the confidence of the classification result is lower than the threshold. Finally, the feature matrix corresponding to the classification result with a confidence lower than the threshold is input into the support vector machine again, the hyperplane parameters of the support vector machine are adjusted, and the classification training is performed again until the confidence of the classification result reaches the threshold requirement.

[0047] In the above process, if the confidence of the classification result does not meet the requirements, a reclassification mechanism is triggered. At this time, the feature matrix corresponding to the classification result with a confidence lower than the threshold is re-input, and the support vector machine hyperplane parameters are adjusted for classification training again until the confidence of the classification result reaches the threshold requirement. This mechanism ensures the reliability of the classification result, while avoiding the misclassification problem caused by insufficient small sample data.

[0048] In practical applications, it can be deployed in intelligent monitoring scenarios, such as real-time classification of personnel behavior in a specific area. A high-resolution camera captures image data of personnel in the area, a microphone array collects sound information in the environment, and a text input device records the identity tags or behavior descriptions of relevant personnel. After preprocessing these multi-modal data, they are transmitted to the subsequent module. The behavior features of the target personnel are extracted from the image data, combined with the identity tags in the text data and the environmental sound features in the audio data, and a weighted feature matrix is generated through a dynamic weight distribution algorithm. The support vector machine is used for classification training of the weighted feature matrix, and finally the classification result of the personnel behavior is output and its confidence is evaluated. If the confidence does not meet the requirements, a reclassification mechanism is triggered until the conditions are met.

[0049] In some embodiments, the dynamic weight distribution algorithm uses a genetic algorithm, defines a fitness function to evaluate the effect of weight distribution, and uses the classification accuracy as the core indicator, combined with the computational complexity for comprehensive evaluation. The crossover and mutation operations of the genetic algorithm are used to continuously optimize the weight distribution parameters until the fitness function reaches the optimal value.

[0050] Specifically, to optimize the performance of the dynamic weight distribution algorithm, a genetic algorithm is used to optimize the algorithm parameters. A fitness function is defined to use the classification accuracy as the core indicator, combined with the computational complexity for comprehensive evaluation. The crossover and mutation operations of the genetic algorithm are used to continuously optimize the weight distribution parameters until the fitness function reaches the optimal value. At the same time, after the classification training is completed, the performance of the classification result is evaluated by introducing a confusion matrix, which contains four indicators: true positive, false positive, true negative and false negative. The precision, recall and F1 score of the classification model are calculated using these indicators to further analyze the classification performance of the model.

[0051] In summary of the above described embodiments, the present application can effectively integrate the information of multi-modal data such as image, text, audio, etc., and solve the limitations of single modal data in small sample scenarios. At the same time, the dynamic weight distribution algorithm assigns weights to the feature vectors in the multi-modal feature matrix, significantly improving the classification performance and computational efficiency. For example, in the intelligent monitoring scenario, the system can classify the behavior of personnel in a specific area in real time, and generate reliable classification results combined with identity tags and environmental sound features, thereby providing technical support for practical applications.

[0052] Figure 2 A module schematic diagram of a small sample image classification system based on multi-modal data fusion provided by an embodiment of the present application is shown in Figure 2 As shown, the system comprises: A data acquisition module 210 is configured to acquire multi-modal data of a target power equipment, and extract image data, text data and audio data from the multi-modal data; An image feature extraction module 220 is configured to extract local features and global features of the image data by using a convolutional neural network; A text feature extraction module 230 is configured to extract semantic features of the text data by using a word embedding model; An audio feature extraction module 240 is configured to extract spectral features of the audio data by using a short-time Fourier transform; A feature matrix construction module 250 is configured to construct a multi-modal feature matrix according to the fusion features, the semantic features and the spectral features; A weight distribution module 260 is configured to assign weights to feature vectors of each modality in the multi-modal feature matrix by using a dynamic weight distribution algorithm, to obtain a weighted feature matrix; An image classification module 270 is configured to perform classification training on the weighted feature matrix by using a support vector machine, to generate an image classification result of the target power equipment.

[0053] In the small sample image classification system based on multi-modal data fusion provided by the embodiment of the present application, by integrating various modal data such as images, texts and audios, features of each modality are extracted respectively, multi-level local and global features of the image data are extracted by using a convolutional neural network, semantic information of the text data is extracted by using a word embedding model, spectral features of the audio data are extracted by using a short-time Fourier transform, various modal features are fused to construct a unified feature matrix, the importance of each modality feature is adaptively evaluated and weighted processed based on a dynamic weight distribution algorithm, in the classification stage, the support vector machine is used to train and classify the fused features, and the confidence score mechanism and the confusion matrix analysis are introduced to evaluate and optimize the classification result, the discriminative information of each modality data can be fully mined in the small sample scene, the expression defects of a single modality under the condition of insufficient samples are made up, the overall accuracy, stability and generalization ability of the classification system are effectively improved, and the application effect of the system in the complex environment is significantly enhanced.

[0054] The above detailed description is further detailed for the purpose, technical solutions and advantages of the present application, and it should be understood that the above detailed description is only a specific embodiment of the present application, and is not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A few-sample image classification method based on multimodal data fusion, characterized in that, The methods include: Acquire multimodal data of the target power equipment, and extract image data, text data, and audio data from the multimodal data; Convolutional neural networks are used to extract local and global features from image data, and the local and global features are fused to obtain the fused features of the image data. Extracting semantic features from text data using word embedding models; Extracting spectral features from audio data using short-time Fourier transform; A multimodal feature matrix is ​​constructed based on fusion features, semantic features, and spectral features; A dynamic weight allocation algorithm is used to assign weights to the feature vectors of each mode in the multimodal feature matrix to obtain a weighted feature matrix; Support vector machines are used to train the weighted feature matrix for classification, generating image classification results for the target power equipment.

2. The few-sample image classification method based on multimodal data fusion according to claim 1, characterized in that, Convolutional neural networks are used to extract local and global features from image data, including: The shallow structure of a convolutional neural network is used to extract local features from image data. The shallow structure includes three convolutional layers, each followed by a max pooling layer, which is used to extract local details of the edges and textures of the target electrical equipment. The deep structure of a convolutional neural network is used to extract global features from image data; the deep structure includes five convolutional layers and two fully connected layers to capture the shape distribution and spatial distribution information of the target power equipment.

3. The few-sample image classification method based on multimodal data fusion according to claim 1, characterized in that, A dynamic weighting algorithm is used to assign weights to the eigenvectors of each mode in the multimodal feature matrix, resulting in a weighted feature matrix, including: An initial weight matrix is ​​constructed based on the dimension and distribution characteristics of the feature vectors of each mode in the multimodal feature matrix; where the initial weight value of the initial weight matrix is ​​1 / N, and N is the number of modes; The correlation between the feature vectors of each mode in the multimodal feature matrix is ​​calculated using the cosine similarity formula, and a correlation matrix is ​​generated. The initial weight values ​​of the initial weight matrix are adjusted based on the correlation matrix. The adjustment process is as follows: feature vectors with high correlation are assigned the first weight, and feature vectors with low correlation are assigned the second weight. The first weight is greater than the second weight. The initial weight matrix after adjusting the initial weight values ​​is normalized so that the sum of the weight values ​​in each column of the initial weight matrix is ​​1, thus obtaining the dynamic weight matrix. Multiply the dynamic weight matrix by the multimodal feature matrix to obtain the weighted feature matrix.

4. The few-sample image classification method based on multimodal data fusion according to claim 3, characterized in that, The method further includes: setting a dynamic adjustment strategy for each weight value in the dynamic weight matrix, and performing dynamic adjustment of each weight value in the dynamic weight matrix according to the dynamic adjustment strategy.

5. A few-sample image classification method based on multimodal data fusion according to claim 4, characterized in that, The dynamic adjustment strategy includes: Each weight value in the dynamic weight matrix is ​​compared with the preset high and low adjustment thresholds; If the weight value exceeds the high adjustment threshold, the high adjustment mechanism is activated; if the weight value is below the low adjustment threshold, the low adjustment mechanism is activated; if the weight value is between the high and low adjustment thresholds, the default adjustment mechanism is activated. The high adjustment mechanism works by collecting the average information gain value, calculating the positive adjustment ratio, inputting the feature vectors of each modality into the current classification model, evaluating the classification information of the feature vectors extracted from each modality, calculating the information gain of the feature dimensions of each modality as the average information gain value, multiplying the positive adjustment ratio by the weight value exceeding the high adjustment threshold, and inputting the calculation result into the dynamic weight matrix to achieve dynamic adjustment of the dynamic weight matrix. The low adjustment mechanism collects the average mutual information value through mutual information analysis and calculates the negative adjustment ratio. It extracts the output feature vector of each mode and performs mutual information analysis with the real label of the corresponding sample to obtain the average mutual information value. The negative adjustment ratio is multiplied by the weight value below the low adjustment threshold, and the calculation result is input into the dynamic weight matrix to realize the dynamic adjustment of the dynamic weight matrix.

6. The few-sample image classification method based on multimodal data fusion according to claim 1, characterized in that, The support vector machine (SVM) is used to train the weighted feature matrix for classification. Specifically, the kernel function mapping method is used to map the weighted feature matrix from the low-dimensional space to the high-dimensional space. The radial basis function is selected as the kernel function, and the parameters of the kernel function are determined by cross-validation. The gradient descent algorithm is used to optimize the hyperplane parameters of the support vector machine until the loss function converges, thus completing the training of the support vector machine.

7. A few-sample image classification method based on multimodal data fusion according to claim 1, characterized in that, The word embedding model employs a bidirectional long short-term memory neural network.

8. A few-sample image classification method based on multimodal data fusion according to claim 1, characterized in that, The dynamic weight allocation algorithm uses a genetic algorithm, defines a fitness function to evaluate the effect of weight allocation, and uses classification accuracy as the core indicator. It also combines computational complexity for comprehensive evaluation. The weight allocation parameters are continuously optimized by crossover and mutation operations of the genetic algorithm until the fitness function reaches the optimal value.

9. A few-sample image classification method based on multimodal data fusion according to claim 1, characterized in that, The method further includes: The confidence level of each category is calculated using the probability distribution of the image classification results; If the confidence level of the classification category is lower than the preset confidence threshold, the hyperplane parameters of the support vector machine are adjusted and the weighted feature matrix is ​​trained again until the confidence level of the image classification result reaches the confidence threshold.

10. A few-sample image classification system based on multimodal data fusion, characterized in that, The system includes: The data acquisition module is used to acquire multimodal data of the target power equipment and extract image data, text data and audio data from the multimodal data; The image feature extraction module is used to extract local and global features of image data using a convolutional neural network, and then fuse the local and global features to obtain the fused features of the image data. The text feature extraction module is used to extract semantic features from text data using a word embedding model; The audio feature extraction module is used to extract the spectral features of audio data using short-time Fourier transform; The feature matrix construction module is used to construct a multimodal feature matrix based on fused features, semantic features, and spectral features; The weight allocation module is used to assign weights to the feature vectors of each mode in the multimodal feature matrix using a dynamic weight allocation algorithm, so as to obtain a weighted feature matrix. The image classification module is used to train a weighted feature matrix using a support vector machine to generate image classification results for the target power equipment.

Images