Intelligent individual soldier system based on AI multi-mode

Abstract

The invention relates to the field of intelligent individual soldiers, in particular to an intelligent individual soldier system based on AI multiple modes. The method comprises the following steps: acquiring and preprocessing multi-modal original data, extracting initial features from the multi-modal preprocessed data, and carrying out standardization processing to obtain multi-modal standard features; the multi-modal standard features are optimized, and a comprehensive feature matrix is obtained; analyzing the comprehensive feature matrix, and extracting spatial features; inputting the spatial features into a gating circulation unit to obtain a time sequence state, and predicting the time sequence state to generate a prediction result; and the prediction result is transmitted to inspection personnel. The problem that although a traditional convolutional neural network and a long-short-term memory network can effectively process spatio-temporal data, long-time stable operation in a resource-limited single-soldier system is difficult due to high calculation complexity and resource consumption is solved. The problems that in the feature fusion process, fixed weights cannot dynamically adapt to environment changes, and a traditional feature processing method is low in robustness to noise and abnormal values are solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent individual soldiers, and in particular to an intelligent individual soldier system based on AI multimodality. Background Art

[0002] In the development of individual soldier systems, traditional single-modal data collection and processing methods have gradually been replaced by multimodal data fusion technology. The combination of multiple sensors such as high-resolution cameras, sensitive microphones, infrared sensors, temperature sensors, and gas sensors enables individual soldier systems to fully obtain on-site information from multiple dimensions such as vision, audio, thermal imaging, temperature, and gas. In addition, the development of deep learning algorithms and graph signal processing technology has made it possible to extract and fuse features from multimodal data, greatly improving the environmental perception capability and decision support level of individual soldier systems. Through the continuous advancement of these technologies, the performance of individual soldier systems in situational awareness, real-time monitoring, and mission execution has been significantly improved.

[0003] Although multimodal data fusion technology has significantly improved the environmental perception capabilities of individual soldier systems, there are still some key issues that need to be addressed in existing technologies. First, although traditional convolutional neural networks (CNNs) and long short-term memory networks (LSTMs) can effectively process spatiotemporal data, their high computational complexity and resource consumption make it difficult for them to operate stably for a long time in resource-constrained individual soldier systems. Secondly, the use of fixed weights in the feature fusion process cannot dynamically adapt to environmental changes, resulting in insufficient accuracy and stability of the fused features. In addition, traditional feature processing methods have low robustness to noise and outliers and cannot effectively cope with data changes in complex environments. These problems limit the efficiency and reliability of existing technologies in practical applications, and they urgently need to be improved through innovative algorithms and processing techniques. Summary of the invention

[0004] The present invention provides an intelligent individual soldier system based on AI multimodality to solve the problems that although traditional convolutional neural networks (CNN) and long short-term memory networks (LSTM) can effectively process spatiotemporal data, their high computational complexity and resource consumption make it difficult to operate stably for a long time in resource-constrained individual soldier systems; the use of fixed weights in the feature fusion process cannot dynamically adapt to environmental changes, resulting in insufficient accuracy and stability of fused features; traditional feature processing methods have low robustness to noise and outliers, and cannot effectively cope with data changes in complex environments.

[0005] The present invention provides an AI multimodal intelligent individual soldier system, which specifically includes the following technical solutions:

[0006] An intelligent individual soldier method based on AI multimodality comprises the following steps:

[0007] S1: Acquire and preprocess the multimodal raw data to obtain multimodal preprocessed data; extract initial features from the multimodal preprocessed data and perform standardization to obtain multimodal standard features; optimize the multimodal standard features to obtain a comprehensive feature matrix;

[0008] S2: Analyze the comprehensive feature matrix through lightweight graph signal processing technology to extract spatial features; input the spatial features into the gated recurrent unit to obtain the timing state, then predict the timing state to generate the prediction result; and convey the prediction result to the inspection personnel.

[0009] Preferably, the S1 specifically includes:

[0010] In the process of optimizing multimodal standard features, a dynamic weight optimization algorithm is introduced to obtain a comprehensive feature matrix.

[0011] Preferably, the S1 specifically includes:

[0012] The process of obtaining the comprehensive feature matrix through the dynamic weight optimization algorithm is as follows:

[0013] Calculate the information gain value of each mode in the multimodal standard features; calculate the dynamic weight of the mode based on the information gain value of each mode; perform feature fusion based on the dynamic weight to obtain a comprehensive feature matrix.

[0014] Preferably, the S1 specifically includes:

[0015] The information gain calculation formula is as follows:

[0016]

[0017] Among them, IG x It is a standard feature The information gain value of; Y represents the target variable; H(Y) is the entropy of the target variable Y; It is the standard feature F x Entropy under conditions; It is the sum of the conditional entropy of all standard features except the target on the target variable.

[0018] Preferably, the S1 specifically includes:

[0019] When performing feature fusion, the weight of feature fusion is adjusted based on the dynamic weight of each modality.

[0020] Preferably, the S2 specifically includes:

[0021] The process of analyzing the comprehensive feature matrix using lightweight graph signal processing technology is as follows:

[0022] The comprehensive feature matrix is ​​represented as a graph signal. Based on the graph signal, the square root of the Laplace matrix and logarithmic smoothing are introduced to perform graph convolution operations and extract spatial features.

[0023] Preferably, the S2 specifically includes:

[0024] The graph convolution operation formula is as follows:

[0025]

[0026] Among them, S is the spatial feature obtained after the graph convolution operation; σ is the sigmoid activation function; L is the graph Laplacian matrix; I is the identity matrix; is the comprehensive feature matrix.

[0027] Preferably, the S2 specifically includes:

[0028] The gated recurrent unit is improved by multiplying the spatial features with the temporal state of the gated recurrent unit at the previous moment, and then processing it through the sigmoid activation function to obtain the updated gate value; the calculation of the reset gate uses the multiplication operation of the spatial features and the temporal state of the gated recurrent unit at the previous moment.

[0029] Preferably, the S2 specifically includes:

[0030] The fully connected layer is used to predict the time series state and generate prediction results. The prediction results are conveyed to the inspectors through display screens or voice prompts. The prediction results and actual conditions are recorded at the same time. By analyzing the differences between erroneous predictions and actual conditions, the intelligent individual system based on AI multimodality is adjusted and optimized.

[0031] An intelligent individual soldier system based on AI multimodality includes the following parts:

[0032] Multimodal data acquisition module, data preprocessing module, data fusion module, intelligent analysis module, communication module;

[0033] The multimodal data acquisition module is used to obtain environmental information of the power plant inspection and output multimodal raw data. The multimodal data acquisition module is connected to the data preprocessing module through data transmission;

[0034] The data preprocessing module obtains the multimodal raw data transmitted from the multimodal data acquisition module, and preprocesses the multimodal raw data to obtain multimodal preprocessed data. The data preprocessing module is connected to the data fusion module by means of data transmission;

[0035] The data fusion module obtains the multimodal preprocessed data transmitted from the data preprocessing module, extracts the initial features from the multimodal preprocessed data, and standardizes the multimodal initial features to obtain the multimodal standard features; the dynamic weight optimization algorithm is used to perform weighted fusion on the multimodal standard features, and the comprehensive feature matrix is ​​output. The data fusion module is connected to the intelligent analysis module through data transmission;

[0036] The intelligent analysis module obtains the comprehensive feature matrix transmitted from the data fusion module, processes the comprehensive feature matrix using lightweight graph signal processing technology, and outputs the prediction results. The intelligent analysis module is connected to the communication module through data transmission;

[0037] The communication module receives the prediction results transmitted from the intelligent analysis module, conveys the prediction results to the inspection personnel, and records the prediction results and actual conditions.

[0038] The beneficial effects of the technical solution of the present invention are:

[0039] 1. Through multimodal data collection and fusion technology, the environmental perception capability of the intelligent individual soldier system is improved. Through high-resolution cameras, sensitive microphones, infrared sensors, temperature sensors and gas sensors, the intelligent individual soldier system can fully obtain on-site information from multiple dimensions, ensuring efficient identification and real-time monitoring of complex environments, and improving the situational awareness level and decision-making support capabilities of the intelligent individual soldier system.

[0040] 2. Through the dynamic weight optimization algorithm, the accuracy of feature analysis can be significantly improved while ensuring computational efficiency. The dynamic weight optimization algorithm calculates the modal weight in real time based on information gain, so that the fusion features can more accurately reflect the importance of each modal feature, which not only improves the prediction accuracy, but also ensures efficient operation in resource-constrained environments.

[0041] 3. Through lightweight graph signal processing technology, the stability and robustness of feature analysis are further improved. Deep spatial features are extracted through graph convolution operations, and gated recurrent units are used to capture timing features, reducing the impact of noise and outliers and enhancing adaptability in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a structural diagram of an intelligent individual soldier system based on AI multimodality according to the present invention;

[0043] Figure 2 This is a flow chart of an intelligent individual soldier method based on AI multimodality described in the present invention. DETAILED DESCRIPTION

[0044] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0046] The following is a detailed description of a specific scheme of a real-time image recognition scratch detection system based on a deformation network provided by the present invention in conjunction with the accompanying drawings.

[0047] See attached Figure 1 , which shows a structure diagram of an AI multimodal intelligent individual soldier system provided by an embodiment of the present invention, the system includes the following parts:

[0048] Multimodal data acquisition module, data preprocessing module, data fusion module, intelligent analysis module, communication module;

[0049] The multimodal data acquisition module is used to obtain environmental information of power plant inspection, including vision, audio, thermal imaging, temperature and gas concentration, and output multimodal raw data. The multimodal data acquisition module is connected to the data preprocessing module by means of data transmission;

[0050] The data preprocessing module obtains the multimodal raw data transmitted from the multimodal data acquisition module, preprocesses the multimodal raw data according to the characteristics of different modal data, and obtains multimodal preprocessed data. The data preprocessing module is connected to the data fusion module by means of data transmission;

[0051] The data fusion module obtains the multimodal preprocessed data transmitted from the data preprocessing module, extracts initial features from the multimodal preprocessed data, and performs standardization on the multimodal initial features to obtain multimodal standard features; a dynamic weight optimization algorithm is used to perform weighted fusion on the multimodal standard features, and a comprehensive feature matrix is ​​output. The data fusion module is connected to the intelligent analysis module by means of data transmission;

[0052] The intelligent analysis module obtains the comprehensive feature matrix transmitted from the data fusion module, processes the comprehensive feature matrix using lightweight graph signal processing technology, and outputs the prediction result. The intelligent analysis module is connected to the communication module by means of data transmission;

[0053] The communication module receives the prediction results transmitted from the intelligent analysis module, conveys the prediction results to the inspection personnel through a display screen or voice prompts, and records the prediction results and actual conditions for subsequent analysis.

[0054] See attached Figure 2 , which shows a flow chart of an AI multimodal intelligent single soldier method provided by an embodiment of the present invention, the method comprising the following steps:

[0055] S1: Acquire and preprocess the multimodal raw data to obtain multimodal preprocessed data; extract initial features from the multimodal preprocessed data and perform standardization to obtain multimodal standard features; optimize the multimodal standard features to obtain a comprehensive feature matrix;

[0056] At the power plant inspection site, multimodal raw data is obtained through the deployed sensor equipment. The sensor equipment includes high-resolution cameras, sensitive microphones, infrared sensors, temperature sensors and gas sensors. The sensor equipment collects visual, audio, infrared (such as thermal imaging), temperature and gas (such as gas concentration) data to form multimodal raw data, which is transmitted using wireless communication technology (such as Wi-Fi, 4G / 5G, dedicated radio frequency bands).

[0057] The multimodal raw data is preprocessed to obtain multimodal preprocessed data. Specifically, the visual data is subjected to denoising and filtering to eliminate environmental noise and image blur; the audio data is subjected to spectral analysis and denoising to extract clear audio signals; for infrared data, correction and standardization are performed to ensure the consistency and accuracy of thermal imaging data; for temperature data, the measured values ​​are converted to standard temperature units (such as Celsius or Fahrenheit) and the measurement range is adjusted to meet the analysis requirements; for gas data, the gas concentration data is converted to standard concentration units (such as one part per million or one part per billion) and corrected to eliminate the influence of sensor drift or environmental factors.

[0058] Extract multimodal initial features from multimodal preprocessed data. Specifically, convolutional neural network (CNN) is used to extract initial features for visual data; Mel frequency cepstral coefficient (MFCC) is used to extract initial features for audio data; thermal image processing algorithm is used to extract initial features for infrared data; time series analysis is used to extract initial features for temperature data; and gas sensor array combined with principal component analysis (PCA) is used to extract initial features for gas data.

[0059] The preprocessing technology and feature extraction technology are both existing technologies and will not be described in detail here.

[0060] The multimodal initial features are standardized to obtain the multimodal standard features. The processing formula is as follows:

[0061]

[0062] in, is the initial feature F x The standardized features are called standard features; x is the identifier of the modality category, which is used to distinguish different types of data. Corresponding to visual, audio, infrared, temperature and gas data respectively; F x is the initial eigenvalue of mode x; F v 、F a 、F i , F g Represent the initial eigenvalues ​​of visual data, audio data, infrared data, temperature data and gas data respectively; μ x and σ x are the mean and standard deviation of mode x respectively; β x is the smoothing parameter of mode x, which is adjusted according to the characteristics of the data to prevent data value jumps. For different types of data, β x The values ​​of β are different to ensure that the standardized data fluctuates within a reasonable range. Specifically, visual data has a large dynamic range. To ensure that the eigenvalues ​​are within a reasonable range, β v =0.1; audio data contains high-frequency and low-frequency components, and the smoothing parameter needs to be moderate, taking β a =0.5; infrared data has a smaller dynamic range, so a larger smoothing parameter can be selected, taking β i =1.0; the temperature data usually changes slightly, so the smoothing parameter is Gas data has large volatility, so the smoothing parameter needs to be moderate, taking β g =0.5.

[0063] The dynamic weight optimization algorithm is used to optimize the multimodal standard features to obtain the comprehensive feature matrix. The dynamic weight optimization algorithm calculates the modal weight according to the information gain (IG) of the standard features of different modalities, so that the modal weight can reflect the importance of the standard features of each modality in real time, enhancing the adaptability and accuracy of feature fusion; and, the modal weight is used to perform weighted fusion operations on the standard features, avoiding the problem of numerical instability of the comprehensive feature matrix, improving the robustness and numerical stability of feature fusion, so that the comprehensive feature matrix can more accurately represent the multimodal information and improve the accuracy of graph signal processing and task prediction.

[0064] The process of obtaining the comprehensive feature matrix through the dynamic weight optimization algorithm is as follows:

[0065] First, the information gain formula is used to calculate the information gain value of each mode in the multimodal standard features.

[0066] The larger the information gain value, the greater the contribution of the standard feature in reducing the uncertainty of the target variable and the higher its importance. Using the information gain formula to process multimodal standard features helps to select the features that contribute most to the prediction of the target variable. By removing standard features with low information gain values, the dimension of the feature space can be reduced, the computational complexity can be reduced, and the importance of each standard feature to the target variable can be clarified.

[0067] The information gain calculation formula introduces the variable "logarithm of the sum of conditional entropies of all standard features except the target on the target variable" in the denominator, so that when calculating the information gain value of the target standard feature, the influence of all standard features on the target variable is taken into account. Compared with the traditional formula that considers the influence of a single standard feature on the target variable, the correlation factor between standard features is added, avoiding the influence of a single standard feature on the information gain calculation result, and the importance of standard features can be evaluated more comprehensively, especially in the context of multimodal data fusion, which helps to select the features with the greatest importance for predicting the target variable, while reducing the dimension of the feature space and the computational complexity.

[0068] The information gain calculation formula is as follows:

[0069]

[0070] Among them, IG x It is a standard feature The information gain value; Y represents the target variable. In an AI multimodal intelligent individual system, Y represents the category to which each sample belongs. It is a vector containing category labels and its length is the number of samples. The specific number of samples and the category to which the samples belong can be set according to the specific implementation scenario; H(Y) is the entropy of the target variable Y, which measures the difficulty of predicting the target variable Y in the absence of any standard feature information and is calculated using the probability distribution of the target variable Y; Given the standard feature F x The entropy under the conditions is a measure of the known In the case of , the difficulty of predicting the target variable Y is calculated using the conditional probability distribution; H(Y) and The calculation process of is a common technical means in information theory, which will not be elaborated here; It is the sum of the conditional entropy of all standard features except the target on the target variable. It is a numerical scalar that indicates the impact of the correlation between standard features on the overall information gain.

[0071] Next, the dynamic weight of the modality is calculated based on the information gain value of each modality. The dynamic weight of the modality is calculated based on the information gain value, which can better adapt to changes in data and tasks. The dynamic weight calculation formula is as follows:

[0072]

[0073] Among them, w x represents the dynamic weight of mode x.

[0074] Finally, feature fusion is performed in combination with dynamic weights to obtain a comprehensive feature matrix.

[0075] The dynamic weights of modal features and the normalization of eigenvalues ​​are introduced during feature fusion. The normalization process avoids the numerical instability of the comprehensive feature matrix by calculating the bi-norm of the eigenvalue and adding a constant to prevent the denominator from being zero. It also takes into account the importance of each modal feature, adjusts the weight of feature fusion through dynamic weights, further standardizes the range of eigenvalues, improves the robustness and stability of feature fusion, and enables the fused features to more accurately represent multimodal information.

[0076] The feature fusion formula is as follows:

[0077]

[0078] in, is the comprehensive feature matrix; ∈ is a constant to prevent the denominator from being zero; ||·|| 2 represents the two-norm.

[0079] S2: Analyze the comprehensive feature matrix through lightweight graph signal processing technology to extract spatial features; input the spatial features into the gated recurrent unit to obtain the timing state, then predict the timing state to generate the prediction result; and convey the prediction result to the inspection personnel.

[0080] In individual soldier systems, traditional technologies such as convolutional neural networks (CNN) and long short-term memory networks (LSTM) have problems with high computational complexity and high resource consumption. As a result, it is difficult for resource-constrained individual soldier systems to achieve long-term stable operation. Lightweight graph signal processing technology constructs graph signals to represent the intrinsic structure and relationship of multimodal data, and uses graph filters such as graph convolution to extract deep spatial features, which can improve the accuracy of feature analysis while ensuring computational efficiency. Because lightweight graph signal processing technology has lower computational complexity and resource consumption, it is more suitable for real-time analysis needs.

[0081] Lightweight graph signal processing technology for comprehensive feature matrix The process of conducting the analysis is as follows:

[0082] First, the comprehensive feature matrix Represented as a graph signal.

[0083] Graph signals represent the mutual relationships and structural information between different modal features, where the nodes of the graph signals represent the elements in the comprehensive feature matrix, including visual, audio, infrared, temperature and gas features, and the edges of the graph signals represent the relationships between different features. The relationships can be statistical associations, similarities, or correlations calculated by some measurement criteria. They can be obtained by measuring the correlation between features through statistical methods such as mutual information, Pearson correlation coefficient, calculating based on the similarity of feature values, or establishing relationships based on expert knowledge or predefined rules. The specific technical means are set according to the specific implementation scenario.

[0084] The graph signal expression is as follows:

[0085] G=(V,E)

[0086] Among them, G is the graph signal; V is the node set, which represents the comprehensive feature matrix, E is an edge set, which represents the relationship between different modal features.

[0087] Then, a graph convolution operation is performed to extract spatial features.

[0088] The square root and logarithmic smoothing of the Laplace matrix are introduced into the graph convolution operation formula. The square root and logarithmic smoothing of the Laplace matrix are used to smooth and normalize the comprehensive feature matrix, which enhances the smoothness and stability of the comprehensive features, so that the comprehensive features can better reflect the correlation between features in the graph convolution operation and reduce the influence of noise and outliers. In addition, the use of logarithmic operations can compress the dynamic range of the comprehensive feature values, making the features more stable and easier to handle in subsequent processing steps. Compared with the traditional formula, the improved method better retains the structural information of the graph signal during the spatial feature extraction process, improves the expressiveness of the comprehensive features, and helps to achieve more accurate task prediction and decision support in complex environments.

[0089] The graph convolution operation formula is as follows:

[0090]

[0091] Among them, S is the spatial feature obtained after the graph convolution operation; σ is the sigmoid activation function, which is used to introduce nonlinearity; L is the graph Laplacian matrix; I is the identity matrix; It is a comprehensive feature matrix, which is introduced into the graph convolution operation in the form of an input signal matrix to extract spatial features.

[0092] Next, a gated recurrent unit (GRU) is used to obtain the temporal state.

[0093] Improvements have been made to the gated recurrent unit. When calculating the GRU update gate, the spatial features are multiplied by the GRU temporal state of the previous moment, and then processed through the sigmoid activation function to obtain a more accurate update gate value; the calculation of the reset gate also uses the multiplication operation of the spatial features and the GRU temporal state of the previous moment. This operation can better capture the interactive relationship between the spatial features and the GRU temporal state of the previous moment, and achieve more refined state updates and resets in time series data processing. Through element-level multiplication, the GRU model can effectively enhance the sensitivity to input spatial features, avoid ambiguity in information transmission, and retain richer contextual information. It improves the prediction performance and stability in complex environments, can more accurately process real-time data, and adapt to dynamically changing task requirements.

[0094] The calculation process of GRU is as follows:

[0095] Z t =σ(W z S⊙U z H t -1+b z )

[0096] r t =σ(W r S⊙U r H t-1 +b r )

[0097]

[0098] Among them, z t is the update gate of time step t, which controls the update degree of the timing state; the subscript z indicates that it is related to the update gate; σ is the sigmoid activation function; W z is the weight matrix of the update gate; U z is the weight matrix of the update gate; H t-1 It is the time series state of the t-1 time step, which contains the comprehensive representation of the input information from the initial time step 0 to the time step t-1. H0 is the time series state of the initial time step, which can be initialized to a full zero vector or a small random value vector. The specific selection method is set according to the specific implementation scenario. z is the bias vector of the update gate; ⊙ is element-wise multiplication; r t is the reset gate of time step t, which controls the degree of reset of the timing state at the previous moment; the subscript r indicates that it is related to the reset gate; W r is the weight matrix of the reset gate; U r is the weight matrix of the reset gate; b r is the bias vector for the reset gate; is the candidate time series state at time step t; the subscript h indicates the correlation with the candidate time series state; tanh is the hyperbolic tangent activation function; W h is the weight matrix of the candidate time series state; U h is the weight matrix of the candidate time series state; b h is the bias vector of the candidate timing state; H t is the time series state at time step t; the bias vector b z 、b r 、b h and the weight matrix W z , W r , W h , U z , U r , U h It is calculated and optimized by the back propagation algorithm during the training of GRU. The specific back propagation algorithm is an existing technology and will not be described in detail here.

[0099] The fully connected layer (FC) is used to predict the time series state. The formula is as follows:

[0100]

[0101] in, is the prediction result of Ht, which is a numerical value or a numerical vector. Depending on the specific task, it can be a category identifier or the probability distribution of each category; W y is the weight matrix of the fully connected layer; b y is the bias vector of the fully connected layer; the subscript y represents W y and b y And output prediction results Related; W y 、b y It is calculated by back propagation algorithm.

[0102] The intelligent individual soldier system conveys the prediction results to the inspectors through a display screen or voice prompts. The display screen can be a display on the helmet worn by the inspectors or a screen on a handheld device, and the voice prompts can be conveyed through the headphones worn by the inspectors.

[0103] The intelligent individual soldier system records the prediction results and actual conditions for subsequent analysis. The records include information such as the system's prediction results, actual conditions, and feedback from patrol personnel. By analyzing the differences between incorrect predictions and actual conditions, the intelligent individual soldier system based on AI multimodality is adjusted and optimized to improve the accuracy and reliability of the intelligent individual soldier system based on AI multimodality.

[0104] In summary, an intelligent individual soldier system based on AI multimodality has been completed.

[0105] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0106] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0107] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An intelligent individual soldier method based on AI multimodality, characterized in that: The following steps are involved: S1: Acquire and preprocess the multimodal raw data to obtain multimodal preprocessed data; extract initial features from the multimodal preprocessed data and perform standardization to obtain multimodal standard features; Optimize multimodal standard features and obtain a comprehensive feature matrix; S2: Analyze the comprehensive feature matrix through lightweight graph signal processing technology to extract spatial features; input the spatial features into the gated recurrent unit to obtain the timing state, then predict the timing state to generate the prediction result; and convey the prediction result to the inspection personnel.

2. The AI ​​multimodal intelligent individual soldier method according to claim 1 is characterized in that: The S1 specifically includes: In the process of optimizing multimodal standard features, a dynamic weight optimization algorithm is introduced to obtain a comprehensive feature matrix.

3. The AI ​​multimodal intelligent individual soldier method according to claim 2 is characterized in that: The S1 specifically includes: The process of obtaining the comprehensive feature matrix through the dynamic weight optimization algorithm is as follows: Calculate the information gain value of each mode in the multimodal standard features; calculate the dynamic weight of the mode based on the information gain value of each mode; perform feature fusion based on the dynamic weight to obtain a comprehensive feature matrix.

4. The AI ​​multimodal intelligent individual soldier method according to claim 3 is characterized in that: The S1 specifically includes: The information gain calculation formula is as follows: Among them, IG x It is a standard feature The information gain value of; Y represents the target variable; H(Y) is the entropy of the target variable Y; It is the standard feature F x Entropy under conditions; It is the sum of the conditional entropy of all standard features except the target on the target variable.

5. The AI ​​multimodal intelligent individual soldier method according to claim 3 is characterized in that: The S1 specifically includes: When performing feature fusion, the weight of feature fusion is adjusted based on the dynamic weight of each modality.

6. The AI ​​multimodal intelligent individual soldier method according to claim 1, characterized in that: The S2 specifically includes: The process of analyzing the comprehensive feature matrix using lightweight graph signal processing technology is as follows: The comprehensive feature matrix is ​​represented as a graph signal. Based on the graph signal, the square root of the Laplace matrix and logarithmic smoothing are introduced to perform graph convolution operations and extract spatial features.

7. The AI ​​multimodal intelligent individual soldier method according to claim 6 is characterized in that: The S2 specifically includes: The graph convolution operation formula is as follows: Among them, S is the spatial feature obtained after the graph convolution operation; σ is the sigmoid activation function; L is the graph Laplacian matrix; I is the identity matrix; is the comprehensive feature matrix.

8. The AI ​​multimodal intelligent individual soldier method according to claim 6 is characterized in that: The S2 specifically includes: The gated recurrent unit is improved by multiplying the spatial features with the temporal state of the gated recurrent unit at the previous moment, and then processing it through the sigmoid activation function to obtain the updated gate value; the calculation of the reset gate uses the multiplication operation of the spatial features and the temporal state of the gated recurrent unit at the previous moment.

9. The AI ​​multimodal intelligent individual soldier method according to claim 1, characterized in that: The S2 specifically includes: The fully connected layer is used to predict the time series state and generate prediction results. The prediction results are conveyed to the inspectors through display screens or voice prompts. The prediction results and actual conditions are recorded at the same time. By analyzing the differences between erroneous predictions and actual conditions, the intelligent individual system based on AI multimodality is adjusted and optimized.

10. An intelligent individual soldier system based on AI multimodality, characterized in that: Includes the following sections: Multimodal data acquisition module, data preprocessing module, data fusion module, intelligent analysis module, communication module; The multimodal data acquisition module is used to obtain environmental information of the power plant inspection and output multimodal raw data. The multimodal data acquisition module is connected to the data preprocessing module through data transmission; The data preprocessing module obtains the multimodal raw data transmitted from the multimodal data acquisition module, and preprocesses the multimodal raw data to obtain multimodal preprocessed data. The data preprocessing module is connected to the data fusion module by means of data transmission; The data fusion module obtains the multimodal preprocessed data transmitted from the data preprocessing module, extracts initial features from the multimodal preprocessed data, and performs standardization on the multimodal initial features to obtain multimodal standard features; The dynamic weight optimization algorithm is used to perform weighted fusion on multi-modal standard features and output a comprehensive feature matrix. The data fusion module is connected to the intelligent analysis module through data transmission; The intelligent analysis module obtains the comprehensive feature matrix transmitted from the data fusion module, processes the comprehensive feature matrix using lightweight graph signal processing technology, and outputs the prediction results. The intelligent analysis module is connected to the communication module through data transmission; The communication module receives the prediction results transmitted from the intelligent analysis module, conveys the prediction results to the inspection personnel, and records the prediction results and actual conditions.

Images